· ai-devtools

Datadog Brings Production Experimentation to AI Agent Observability

The capability combines agent evaluation scores with operational measurements such as latency, token consumption, execution cost, and errors, helping teams establish whether changes to models, prompts, retrieval strategies, or tools actually improve an agent.


The announcement addresses a growing challenge in agent development: a change that performs better in a controlled test may behave very differently when exposed to unpredictable user requests. Datadog's approach brings controlled experimentation, production telemetry, and execution traces together, giving teams evidence of both the outcome and the behaviour behind it. Offline evaluations remain an important part of agent development.

Teams can run different versions against predefined datasets, assess responses and identify known regressions before deployment. However, these evaluations are only as representative as the datasets and scoring criteria behind them. New user workflows, unexpected requests, and changing tool responses can expose weaknesses that existing test cases never anticipated.

Online experiments address this limitation by evaluating agent behaviour against real user interactions. Teams can compare a control version with a modified variant by randomly distributing production traffic between them, reducing the influence of differences in user requests. This provides a more realistic view of how an agent performs under actual workloads, while also helping teams identify new scenarios that should eventually become part of their offline regression datasets.

Earlier coverage

Related coverage

Also worth reading

  1. Angular 22.2.7 released, fixes hydration warnings, improves forms date formatting
  2. next.js 16.5.0-canary.54 released
  3. AI toys are on the naughty list again this year
  4. Program-as-Weights, a paradigm for fuzzy functions, compiles natural language to neural artifacts
  5. No AI release details remain unknown