· ai-devtools
Datadog Scout AI model investigates production problems in Bits Investigation
Scout, a specialized artificial intelligence model, is now available from Datadog, designed specifically to aid in the investigation of live production issues.
This capability resides within Bits Investigation, Datadog's AI SRE offering, allowing it to ingest an alert, examine telemetry data, and subsequently propose potential causes for the problem. Currently, Datadog engineers and a select group of clients are utilizing this feature in a private preview setting.
A significant portion of the details presented in the announcement centers around cost considerations. Because Bits must cycle through numerous model calls—querying, analyzing the outcome, and determining the subsequent check—the operational expense can be substantial. While general-purpose models possess the capacity to perform this sequence, Datadog notes that the associated billing makes running them against every incoming alert economically challenging. The creators of the announcement aim for Bits to begin its work while a human engineer is still actively engaged with the page.
In performance testing, Scout medium achieved a score on Datadog's AlertEval benchmark comparable to GPT-6.1 Sol high and Claude Opus 5.5 high, though its estimated inference cost was between three and eight times lower. Furthermore, at the xhigh setting, Datadog reported that Scout matched Claude Fable 5.1 xhigh while incurring only a quarter of the expense. These figures are relative to the cost of a Scout medium investigation, as no specific pricing has been published. AlertEval itself employs genuine alerts, recorded telemetry, and explanations authored by Datadog personnel, with an LLM judge comparing Scout's findings against an expert's analysis, awarding points for concordance at both the immediate and underlying cause levels; conversely, marks are deducted for unsupported claims or contradictions. The resulting score ranges from zero to 100, although Datadog cautions that this metric does not equate to a percentage of incidents successfully resolved.
Scout is architecturally built upon GLM-5.3 Flash. Datadog enhanced this model through post-training utilizing reinforcement learning, which was guided by incident simulations derived from patterns observed in past incidents. These simulations incorporate an alert, queryable telemetry, and a known outcome, yet the company confirms they contain no proprietary customer data. Scout is trained to interact with tools modeled after those found in Bits, such as searching across metrics, logs, traces, and inspecting code, deployment information, and feature flags.
The technical report offers insight into the challenges encountered during the training phase. Initially, a reward structure based solely on the final answer encouraged Scout to make fewer assertions and cease investigation prematurely, which reduced the risk of penalties but did not improve its investigative quality. The development team subsequently altered the reward system to account for correctness, the scope of exploration, and the supporting evidence provided. Additionally, to prevent the model from defaulting to a single, predictable path through an incident, they either omitted certain tools from some training runs or limited the time allotted.
In a separate evaluation conducted on SREGym, a public dataset comprising 90 faults, Scout successfully resolved 71 cases at a medium effort level, 79 at a high level, and 80 at an xhigh level. For comparison, GPT-6.1 Sol xhigh managed to resolve 82 cases. Since Datadog conducted these evaluations internally, including the runs against newer comparative models, external validation of these results would be beneficial.
Scout emerges as observability vendors grapple with defining the scope of incident handling for an automated agent. AWS DevOps Agent integrates data from various monitoring tools alongside source code, runbooks, and deployment logs, effectively consolidating information across different platforms. In contrast, Datadog is focused on developing a cost-effective model for recurrent use within its own product suite. Honeycomb's Canvas Agent mirrors a more manual investigative process favored by many engineers; when an SLO breaches or an alert triggers, it opens a shared Canvas, allowing for querying, idea testing, and maintaining visible evidence for subsequent human review. Teams can formalize established debugging routines as Canvas Skills, and Honeycomb also supplies an MCP server enabling external agents to query its telemetry. Honeycomb has yet to introduce a dedicated SRE model, prioritizing instead its query engine and the documentation of the investigation's progression.
Splunk and New Relic have advanced further into the remediation phase. Splunk's AI SRE capability groups alerts and generates a probable cause accompanied by an evidence trail, while a linked Claude agent is capable of drafting a code modification within a sandbox and opening a pull request. New Relic's SRE Agent incorporates prior incident reviews, runbooks, and resolution histories into its analysis, and its Autopilot feature can suggest corrective measures, though both systems mandate an approval step before any changes reach production.
An InfoQ study involving ClickHouse concluded that while general-purpose models can produce summaries, they require human oversight to consistently pinpoint root causes. Instacart employs a similar methodology with Blueberry, wherein agents gather necessary context before an engineer evaluates the suggested cause within Slack. The evidence supporting Scout remains primarily derived from Datadog's internal testing.
In response to the launch, the independent AI publication AGTP highlighted that AlertEval has not undergone independent scrutiny. Furthermore, AGTP requested an audit log detailing precisely which evidence Scout considered and which it disregarded. The requirement for transparency in evidence, as seen in Honeycomb's visible Canvas and Splunk's evidence chain, suggests this is a common expectation across vendors. While a swift answer can be helpful during an active incident, engineers also require visibility into the model's reasoning pathway. Datadog has not provided a timeline for when this feature will be generally available. The company intends to expand the types of failures included in its simulations and eventually enable Bits to execute repairs; however, the current preview stage stops before that point, meaning Scout diagnoses and suggests recovery actions, leaving the final change execution to the engineer.
Why it matters
This release signals a trend toward specialized, cost-effective AI tooling for Site Reliability Engineering tasks. Developers should watch how this specialized approach compares to broader agentic workflows from competitors like AWS and Honeycomb. The focus on cost efficiency is a key operational consideration for production monitoring tools.
