· ai-devtools
Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines
In the complex world of finding deep software flaws, researchers pitted two automated detection methods against each other: Fuzzilli, a generative JavaScript engine fuzzer, and BigSleep, an LLM-based bug finding system.
Project Zero conducted this comparison by examining how these tools locate vulnerabilities within Chrome’s JavaScript engine, V8, over a six-month period. Initially, BigSleep uncovered more issues than Fuzzilli, reporting 45 vulnerabilities compared to Fuzzilli's 37, with a 25% overlap in the specific bugs identified.
Despite this initial finding, dedicated experiments designed to estimate theoretical overlap since 2023 suggested a much higher overlap, around 60%. This indicated that a considerable number of vulnerabilities were discoverable by one method alone or the other.
Regardless of the precise overlap figure, the research underscored that fuzzing and LLMs are not replacements for one another; rather, they are complementary approaches. The recommendation for vendors is to employ both fuzzing and agentic analysis to maximize vulnerability detection before users are impacted.
The investigation focused on three core questions: the real-world bug count for each approach, the degree of overlap between them, and the unique types of bugs each system uncovers. The analysis narrowed its scope to V8, a high-performance JavaScript and WebAssembly engine with a mature testing infrastructure.
When analyzing actionable vulnerabilities—those not marked as duplicates or WAI—the results showed a near tie, with BigSleep having a slight edge (45 vs. 37). However, comparing the cost-per-bug proved challenging because neither system was optimized for that metric.
To gauge overlap, the team first examined bug tracker entries, finding roughly a 25% overlap (20 out of 81 actionable reports), with Fuzzilli often reporting the bug first. A secondary experiment involved having BigSleep attempt to rediscover older Fuzzilli bugs.
This second test revealed that the majority of historical BigSleep bugs were likely missed by fuzzers like Fuzzilli because the fuzzers had already been testing vulnerable builds for months or years beforehand.
The setup for this re-discovery trial involved filtering historic Fuzzilli vulnerabilities down to 112 bugs that could reproduce under a standard set of flags when presented as "code review challenges" to the agent.
