AI Evals & Observability

Teams shipping AI features in production usually reach a specific realisation around month six: they’re iterating on prompts, swapping models, adjusting retrieval — and they have no reliable way to tell whether the changes are helping. The demo still demos. The edge cases still break. The team ships changes on vibes, rolls them back when customers complain, and slowly loses confidence in its own ability to improve the feature. AI Evals & Observability is the engagement we designed for that moment.

Eval harnesses are the first half of the work. A good eval harness has three layers: a gold-standard dataset the team agrees represents the real workload, a set of graders — automated, LLM-judged, human-reviewed as warranted — that produce scores the team can trust, and a CI integration that runs the whole thing on every prompt or model change. We build these in Braintrust when the team wants a hosted platform, custom-built when the client’s privacy posture or workflow needs something specific.

Observability is the second half. What we build here looks like observability for any distributed system — traces, spans, logs — but tuned to the shape of AI workloads: prompt versioning, per-request latency and cost, tool-call trees for agentic flows, and the ability to drill from an aggregate metric into the specific conversation that caused it. Langfuse is our default when a self-hostable option is preferred; LangSmith and Braintrust when the client is already in those ecosystems. We pipe into the client’s existing observability (Sentry, Datadog, Grafana) rather than replacing it.

The dashboards are the third thing. They’re where the work becomes actionable for non-engineering stakeholders: product managers who need to know whether quality is up or down this week, ops teams who need to spot regressions before customers do, and leadership that wants to know what’s being spent on inference and what it’s returning. We design these dashboards around the questions the client’s team actually asks, not the questions the platform templates suggest they should.