The Ethereum Foundation’s protocol security team ran coordinated artificial intelligence (AI) agents against the code Ethereum depends on, surfacing at least one remotely exploitable bug along with a flood of convincing false positives that humans had to untangle.
Key Takeaways
One agent produced about 1,000 candidate findings, with 86% of top-tier picks surviving expert review.The foundation said July 9 that triage, not bug-finding, is the bottleneck; human validation stays essential.The surprise, the foundation wrote, was not that AI agents could find bugs but “how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real.”
The team catalogued the recurring shapes of those imposters, à la crashes that only occur in debug builds and never in production, reproducers that rely on unreachable internal values no attacker could actually supply, and formal-verification proofs that are technically true but so unconstrained they demonstrate nothing.
The foundation’s answer was a hard evidentiary standard it summarized as “reproducible or it didn’t happen.” To elaborate, every candidate finding is henceforth required to ship with a self-contained artifact that reproduces the failure against the actual code, independent of how confident the reporting agent claims to be.
Agents, in this context, can be viewed as hypothesis generators (search tools, not decision-makers) organized into recon, hunting, gap-filling, and validation stages, with humans making the final call.
The post also offered a rare benchmark for how well the current generation of tools performs. A property-based testing agent generated roughly 1,000 candidate findings, and after expert review, about 86% of its top-tier recommendations survived scrutiny (strong for a machine, but a rate that still demands a human filter before anything touches production code).
The tools are clearly finding real vulnerabilities in critical infrastructure, thereby undermining the dismissal that AI-generated bug reports are pure noise. Yet the workload has not disappeared but simply moved downstream to triage, where experienced engineers separate signal from simulation. For a network securing hundreds of billions of dollars in value, that filter is important.
The foundation is now pushing the work forward rather than treating it as a one-off. Its Ecosystem Support Program, for instance, is funding a dedicated grant round for AI-powered protocol security, covering research, auditing, and vulnerability detection.
