So Many AI Security Benchmarks, But None Tell the Whole Story

Dream Group, a national sovereign AI and cyber-defense firm, trained an 8 billion parameter model, the Mid-trained Security Transformer (MiST), utilizing 220,000 vulnerability records, threat intelligence reports, and detection rules. This model, among the smallest modern ones, showed improved average performance across 13 cybersecurity tasks, which mostly involved true-false and multiple-choice questions regarding vulnerability analysis and cybersecurity domains.

Elad Ben Zaken, an AI researcher at Dream Group, stated that stronger security capabilities do not necessarily require a more complex model; specialized training and better surrounding software also boost outcomes. He remarked, "I expect efficiency to keep improving in coding and security tasks through specialization and better workflows," adding, "The model supplies capability; the harness determines how effectively it uses context and tools. Adding more of either can increase cost without improving results."

Anthropic's release of Opus 5.5 in September demonstrated performance gains while costing 40% less than Opus 5 on "typical workloads." However, for cybersecurity tasks, such typical scenarios are rare. A model’s performance in one area may hold little meaning for an enterprise seeking an expert system to generate secure code or discover weaknesses in its attack surface.

Offensive-security firm XBOW focuses its metrics on a model's capacity to find and exploit vulnerabilities. Albert Ziegler, XBOW's head of AI, explains that the company develops its own harness to optimize results regardless of which model is used, because different clients require different outcomes. Ziegler noted, "In its most atomic form, that's the ability of the model to find an exploit — that is our bread and butter in terms of seeing what's the value that the model produces," but conceded, "For other companies, this bread and butter could obviously be something else."

The SEC-Bench Pro, created by a University of Illinois Urbana-Champaign research team, assesses an AI agent's power to independently locate and trigger a known memory-safety bug in a browser engine or Linux kernel, provided the general bug location and type are specified. This metric currently ranks Codex and GPT-5.5 as favorites, achieving an overall solve rate of 58%, while open-weight models struggle; GLM-5 succeeded in only 13 of the 344 test cases.

Booz Allen, a government contractor, publishes the Cyber Weapons Index (CWI), which combines a model's skill in finding and exploiting vulnerabilities with its ability to execute attacks against a target. The latest version places OpenAI's GPT-6 Astra at the top with a CWI of 80.5, narrowly ahead of Claude Mythos with a score of 80. Eric Syphard, senior vice president and head of AI at Booz Allen, stated, "The CWI offers one empirical approach to offensive cyber capability — test models against a common enterprise network and score demonstrated actions using network evidence, rather than relying on what models say they can do."

Joe Tyler, general manager and AI researcher at Sonar, cautions that a newer model does not equate to greater security. Sonar posts scores on its LLM Leaderboard for Code Quality & Security after running LLMs through a testing pipeline covering cognitive complexity and vulnerability occurrence. Tyler advises that public benchmarks should only serve as a starting point, urging organizations to customize their test pipelines to reflect their specific code and workflows, saying, "Start with the task and the failures you can't accept, rather than assuming the newest model is the right choice."

Nico Waisman, XBOW's CISO, points out that attackers adopting AI will increase the volume of vulnerabilities and hasten exploitation times. He advises organizations to invest in tools and processes that accelerate security team operations, automate routine tasks, and decrease remediation time. Waisman concluded, "Ultimately, I wouldn't evaluate AI security tools based on which model they use or how impressive their demos look," instead focusing on measurable outcomes like reducing exposure or speeding up the team.