
I was working on investigating scaling test-time compute for vulnerability detection. We found that running a cheap model multiple times for vulnerability detection can find more vulnerabilities than Mythos at less than a fifth of the cost. Anthropic's post, on discovering vulnerabilties with claude, also found that test-time scaling significantly helps with finding vulnerabilities. This means attackers may actually be able to get a lot more "juice" from open weight models for hacking that we might expect, matching performance of frontier ones. So, it's important that defenders are also adopting these techniques, maximizing test-time compute on codebases to find more vulnerabilities before attackers can.
Running a lower-cost model multiple times can match the vulnerability detection rate of frontier models at a small fraction of the cost. On a 30-task subset of dfbench, five runs of DeepSeek v4.1 Flash found about 71% of known vulnerabilities, up from about 52% with a single run, at $4.24 per task. Running it 15 times increased recall further to 80%, roughly 13% of Mythos 5’s cost, which found approximately 69% of the vulnerabilities. Because LLMs are stochastic, they can find different vulnerabilities each time they analyze the same codebase. Combining findings across runs increases coverage, but can also introduce more false positives. To control that noise, we used our harness to review and deduplicate findings across runs, then filter them by confidence. In separate validation experiments, this approach generally improves both recall and precision. These results suggest that repeated sampling can improve the tradeoff between coverage, precision, and cost. The gains depend on the model, the number of runs, and how findings are filtered. Read the research: depthfirst.com/research/bon-…


















