Making Failure Diagnosis Reliable with Pensive AI

On peak days, approximately 100,000 workloads fail across Netflix's compute engines and orchestration systems like Spark, Trino, and Druid.

When a failure occurs, the initial event is just the start; someone must sift through numerous logs to find meaningful signals, determine if the issue resides in the code or the platform, and decide on escalation, retry, or reconfiguration. Even a small proportion of difficult failures can consume significant engineering time, waste compute resources, and disrupt vital data pipelines.

Netflix initially addressed this challenge with Pensive, an auto-diagnosis and remediation system within its data platform. Pensive gathered logs from failed jobs, employing hand-authored Java rules and regular expressions to classify and extract errors, then directing retry or remediation advice to products such as Maestro. This system managed to make about 50% of failures self-service, and in some instances, fully automatic.

However, rules only cover known patterns, meaning coverage expands only when an engineer recognizes a new failure mode, understands the logs, writes a rule, and maintains it against evolving engines and log formats. Over three months, Pensive processed around seven million job-failure classifications, yet nearly half of those fell into an unknown category.

To expand coverage without abandoning Pensive's reliable, deterministic functions, the team invested in Pensive AI. This introduced AI-assisted error extraction and focused diagnosis and remediation guidance for data and ML practitioners. The design prioritized bounded extraction, predictable evaluation inputs, responsive product behavior, and quality checks spanning from local development through production.

Pensive AI integrates AI around these existing deterministic processes, offering broader coverage, lower latency, manageable AI costs, and simplified maintenance across the software lifecycle. Early results are promising: AI error extraction now accounts for 100% of failures, including those legacy Pensive marked as unknown. Furthermore, observational data showed that failed step attempts without Pensive AI averaged 321% more manual debug-log exploration actions than those where the AI was used.

The system features two complementary paths: always-on error extraction and on-demand debugging, both sharing log sources, the LLM gateway, tracing, and the cache. Error extraction provides a quick first look at every failure with low latency and cost, helping in many cases. On-demand debugging offers a deeper analysis by incorporating the LLM's reasoning across logs, though this incurs slightly higher latency and cost.

For automated workflows, "Pensive AI Error Extraction" runs automatically upon step failure in a Maestro workflow. This AI call executes within the same request as the existing regex classification, constrained by Maestro’s strict 90-second post-processing timeout.

Beyond simple extraction, Pensive AI Debugging offers root-cause analysis and suggested fixes via APIs, MCP, and the Data Portal UI. Since generating a full debugging response can take tens of seconds, this capability is run on demand to control costs, treating the failed step attempt as the unit of work to ensure consistent diagnosis across all investigators.