QuillShield found 98 of 117 known vulnerabilities on EVMbench

EVMbench is a benchmark from OpenAI and Paradigm that measures how well AI agents detect, patch and exploit vulnerabilities from real smart contract audits. In our own run on 8 July 2026, the QuillShield alpha found 98 of the 117 known vulnerabilities across 40 audited projects (83.8%, self-reported). The best single model in the original EVMbench paper found 45.6%.

Published . 6 minute read. By the QuillShield team.

We ran QuillShield on this benchmark ourselves, so read the number as self-reported. Other teams publish their own EVMbench runs, each on its own setup, and scores from different setups are not directly comparable. We do not claim a rank. The comparison in this post is between a multi-agent system and one model working alone, as the paper measured it.

What is EVMbench?

EVMbench is a benchmark for AI agents on smart contract security. OpenAI and Paradigm released it in February 2026. OpenAI's page says it draws on "117 curated vulnerabilities from 40 audits". Most are high-severity findings from public audit competitions, kept only if the bug could lose user or platform funds. Several more come from the security review of the Tempo blockchain.

The paper sets three tasks.

Mode What the agent does How it is graded Vulnerabilities
Detect Audits a repository and writes a report A model-based judge checks the report against each known vulnerability 117
Patch Edits the code to fix the vulnerability The project's tests must still pass and the known exploit must fail 44
Exploit Attacks contracts deployed on a local test chain Its transactions are replayed and the resulting chain state is checked 23

Detect is the mode closest to an audit. The agent gets the repository a human auditor in the competition received and is scored on every high-severity bug in it, so an agent that stops after the first one scores low.

What did QuillShield score on EVMbench?

The QuillShield alpha finished a run across all 40 audited projects on 8 July 2026. It found 98 of the 117 known vulnerabilities, which is 83.8%, and missed 19. In 28 of the 40 projects it found every known vulnerability.

System Known vulnerabilities found Who measured it, and when
QuillShield alpha 98 of 117 (83.8%) Our own run, 8 July 2026, self-reported
Best single model in the original EVMbench paper 45.6% The paper's authors, February 2026

The second row depends on which version of the paper you open. The original paper, linked from Paradigm's launch post of 18 February 2026, counts 120 vulnerabilities and reports 45.6% for the best single model in detect mode. The arXiv version of 5 March 2026 counts 117 and reports 45.9%. Our run is scored against 117, so 45.9% is the like-for-like baseline. We quote 45.6% because it is the figure in the paper released with the benchmark, and it is 0.3 points more favourable to us than the revised one. The benchmark section of our home page shows both.

The two rows are different kinds of system. The paper's agents are single models, each working alone in one autonomous run with internet access switched off. The QuillShield alpha was a multi-agent system built for this one job. The paper itself found that the tooling around a model moves the score, so a purpose-built system should do better than a general one.

How did we run the benchmark?

The alpha ran on each of the 40 projects, and the figure counts how many of the 117 known vulnerabilities it found. That makes 83.8% a detection measure. This post makes no claim about patching or exploiting.

The alpha was a multi-agent pipeline. It mapped the in-scope contracts and their call graph, then listed the invariants and the assumptions made about external dependencies. It generated attack hypotheses. A triage step removed duplicates and weak candidates and tagged each remaining hypothesis with the invariant it would break. A validation stage then stress-tested each one.

We use the benchmark to check our own work, and we never fit findings to it.

Two details of the method are missing from this post. The paper grades detection with a model-based judge and reports each agent as an average of three runs, with a confidence interval. We have not yet published how our findings were matched to the known vulnerabilities, or how many runs sit behind 83.8%. Until we do, read the figure as a self-reported count.

What about the 19 vulnerabilities QuillShield missed?

Nineteen of the 117 were missed, about one in six. The alpha found every known vulnerability in 28 projects, so all 19 sit in the other 12.

We have not published the list of the 19. The paper makes a related point about single models. A high score on a project did not stop them from missing a critical bug in that same project. Finding every bug is harder than finding one.

A system that finds five in six known high-severity bugs leaves one behind, and on your own code nobody can say in advance which one. That is why QuillShield does not replace a manual audit, and why any finding can be sent to a QuillAudits auditor.

What should you check beyond an EVMbench score?

  • How many false alarms came with the hits. Detect mode scores a report against a list of known bugs. The paper says it cannot check the other findings in a report, so wrong ones cost nothing. It also gives no credit for real bugs the original auditors did not report.
  • Whether the result holds on code no model has seen. The bugs come from published audit reports, so the models underneath any AI auditor, ours included, may have seen them. A follow-up paper raises this and tests agents on newer incidents.
  • How a tool does on your codebase. Half of the benchmark's projects have about 1,300 lines in scope or fewer, and the largest has about 10,000. The bugs are high-severity, loss-of-funds issues in Solidity. The benchmark says nothing about lower severities, other languages or anything off-chain.
  • How tools compare. Each team that publishes a score chose its own setup. Unless the tooling, grader, time limit and number of runs match, two scores cannot be put in order.
  • What the product does today. The run was on the alpha in July 2026. QuillShield has changed since, and we have not published a newer run. Today it supports Solidity with Foundry and nothing else. The docs describe how a full audit runs.

For false alarms we use a different measure. During the private beta, 90% of the findings that QuillAudits auditors reviewed on client Solidity code were valid. That is a separate measurement on different code. Can AI audit smart contracts covers it, and how to evaluate an AI smart contract auditor lists the questions to ask of any benchmark claim, ours included.

What to do next

A benchmark cannot tell you what QuillShield finds in your code. A full audit of your own repo can. Start an audit. New accounts get $50 in credit, good for up to half of an order.

Common questions

Does QuillShield claim the top score on EVMbench?

No. The 83.8% figure (98 of 117 known vulnerabilities across 40 audited projects) comes from the QuillShield alpha's own run on 8 July 2026 and is self-reported. We compare it only with the best single model in the original EVMbench paper, which found 45.6%. Other teams publish their own runs, and scores from different setups are not directly comparable.

Does a high benchmark score mean my contracts are safe?

No. EVMbench counts known, high-severity bugs in code from past audits. It does not penalise false positives and it says nothing about your codebase. In our own run on 8 July 2026 the QuillShield alpha still missed 19 of the 117 known vulnerabilities. Treat a benchmark score as one input next to a manual audit.

Why do some sources count 120 vulnerabilities and others 117?

The original EVMbench paper, released in February 2026, counts 120 vulnerabilities. The version posted to arXiv on 5 March 2026 and OpenAI's EVMbench page count 117. We have not found a public note that explains the change. The QuillShield run is scored against 117.

Sources

QuillShield is building the security layer between AI and money, so you can keep shipping between audits.

Sign-up is free, and new accounts get $50 in credit, good for up to half of an order.

  1. 1Sign up and import a repo
  2. 2See the exact price before anything is charged
  3. 3Approve it, and the audit starts