What Session-Level Tracing Taught Me About LLM Applications

An AI phone agent handling insurance calls might walk a customer through canceling a policy when they actually wanted to purchase one. This error occurs without any exception, error code, or health check failure; latency remains normal. Conventional dashboards would flag this call as a complete success, illustrating that uptime and answer quality are unrelated concepts.

My initial impulse was to simply tack on a quality metric to existing dashboards, but this approach proved insufficient given how these applications function. A single user request flowing through a RAG pipeline or multi-agent system actually branches out. It involves retrieval, several model calls (sometimes sequential, sometimes parallel), tool invocations, and a final synthesis step.

If I had consolidated this entire process into one opaque 'request' span, as a standard APM trace would, I would have discarded the precise data needed for later debugging. Therefore, the pipeline I developed instruments at the span level, treating 'prompt,' 'LLM call,' 'tool call,' and 'parser' as distinct span types, adhering to OpenTelemetry’s GenAI semantic conventions.

Redaction must occur during collection, not later, because prompts and completions contain sensitive customer data. Capturing everything at every span before filtering creates a PII pipeline regardless of intent. What was critical for catching the scenario described above was implementing a scoring stage that runs hallucination detection and answer-quality scoring in parallel with standard metric extraction.

A deeper issue surfaced when I realized that conversational systems often degrade in quality as a session progresses. This problem remains invisible if one only examines individual requests. It only becomes apparent when comparing turn 10 of a conversation against turn 1 of that same interaction.

To address this, I layered tracing on top of request-level data, aggregating by session ID rather than by single trace. This included total tokens and duration at the session level, a breakdown of every trace within that session, and, importantly for multi-agent setups, a round-by-round view of which agent consumed time and tokens.

This granular view allowed me to pinpoint precisely where the issue lay, stating, “The second agent in this chain is where both the latency and the spend are concentrated,” instead of merely knowing a three-hour session was excessively costly. Tagging token spending by model and by session within a daily dashboard proved to be a high-value addition because it made a previously hidden cost visible to actionable personnel.

A significant unanswered question remains: reviewing a single capability, such as one querying a monitoring API, reveals little about what happens when an agent chains it with other capabilities at runtime in an untested sequence. While runtime filtering and maintaining a human confirmation step are partial mitigations, no one in the field seems to possess a complete solution.

For those building or evaluating observability for LLM-backed applications, I would prioritize two things: first, establishing the quality-scoring pipeline as a primary parallel signal alongside performance metrics from the start; and second, incorporating session-level aggregation before it feels necessary, since the data needed to diagnose a quiet failure is often gone by the time the symptom appears.