Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
Agent observability is based on similar primitives as regular observability (OTEL), but is wildly different. Trace payloads are huge, and errors are much more subtle, they are not increased latency or elevated 500 response rate. We recommend tracing everything in detail, and reading a few traces manually at the beginning while you iterate. At scale, the answer is almost always use another LLM look at your traces, which is what this post is all about.
Evals are critical. I really recommend adopting the Anthropic's capability / regression eval model [1]. We also deeply believe that agent evals should be end-to-end, and as high-level as possible. This again often involves a quality LLM-as-a-judge, tuned on a set of manually labelled data.
With this idea in mind (to answer your question), we think that, for eval datasets, the most crucial bit is to be able to store input of what your agent was asked about in production. Collect a certain distribution of inputs for happy and unhappy cases, and then run evals on those inputs. That's why Laminar went great lengths to help our users automatically get actual inputs their users send to their agents [2].
[1] https://www.anthropic.com/engineering/demystifying-evals-for...
[2] https://laminar.sh/blog/extracting-tasks-from-agent-runs
I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
I'd say in general, with respect to errors/low quality responses etc you have to be extremely clear about what you are doing because once you get into the weeds, what is and is not an error is very grey in many cases.
Also, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
Yes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.
Workflows are constantly changing but our design process happens across Figma, old-fashioned paper, and code.