How to evaluate an AI smart contract auditor
To evaluate an AI smart contract auditor, ask for evidence in five areas. They are how many known vulnerabilities it finds on a public benchmark, how many of its findings human auditors confirm as valid, whether each finding comes with a proof of concept, what happens to your code and findings, and how it reviews your next pull request.
QuillAudits auditors check AI findings as part of their work. These are the questions we would put to any AI auditor before trusting it with unreleased code. Under each one is our own answer for QuillShield, with a number and a date where we have them and a plain note where our evidence is thin, and we name no other product.
What does an AI auditor find on a public benchmark?
Ask for a result on a benchmark you can read yourself, with the count, the date, who ran it and how it was graded. EVMbench is one of these, and OpenAI and Paradigm built it from 117 known vulnerabilities in 40 real audits of Solidity code.
In our own run on 8 July 2026, the QuillShield alpha found 98 of those 117 known vulnerabilities (83.8%, self-reported) across 40 audited projects. It found every known vulnerability in 28 of the 40 projects and missed 19. The best single model in the original EVMbench paper found 45.6%. That compares a multi-agent system with one model working alone.
Three cautions apply to any benchmark claim, ours included.
- Most scores are self-run, ours included, because each team publishes its own run. Setups differ, so the scores are not directly comparable and none of them is a ranking.
- Detect mode does not penalise wrong findings. The paper says it cannot check findings outside its known list, so a tool can score well and still bury you in noise.
- Ask what was missed and how the run was graded. We have published the number of misses. We have not yet published the list, the grader or the number of runs. Our EVMbench results says what is known.
How many findings hold up when a human checks?
A false positive is a reported finding that is not a real vulnerability, and a benchmark cannot count them. Ask what share of the tool's findings were valid when people reviewed them, who the reviewers were, what code it was and how large the sample was.
During QuillShield's private beta, 90% of the findings QuillAudits auditors reviewed on client Solidity code were valid. The reviewers work for the firm that makes the product and we do not publish the count, so ask us about both. An independent researcher's public test is written up on our blog, and can AI audit smart contracts goes through the evidence in more detail.
Does each finding come with a proof of concept?
A proof of concept is a test that carries out the attack. If it passes, the attack worked under the test's conditions, which is stronger evidence than a written argument. Ask whether the tool runs the test itself, what share pass, and what it does with findings that have no passing test.
On a QuillShield full audit, a finding can show a proof of concept status. The docs list four: none, pending, passed and failed. In the demo audit on the home page, QuillShield reported 5 findings, 3 of them Critical, and 5 of 5 proofs of concept passed, in 2h 7m. That was a demo vault, and on real code proofs of concept do not pass for every finding. A finding without one has not been proven wrong. A passing one shows that the attack path works, and it does not tell you how large the loss would be in production. Run your first audit lists what a finished audit includes, such as the proof of concept status and the PoC bundle.
What happens to your code and your findings?
You are handing over unreleased code and, after the audit, a list of its weaknesses. Ask whether source is kept, whether anything is used for training, who else can see the findings and what access the integration needs.
QuillShield's terms of service and docs cover these points.
- Findings are disclosed to you alone. They never go to a bug bounty platform or an audit contest.
- Source code is deleted after each audit. Reports, findings and the security graph, which the terms call Memory, stay in your workspace until you delete them.
- Your code and findings are never used to train a model, ours or a third party's.
- The GitHub App reads your code and writes only checks, statuses and pull request comments.
QuillAudits is certified to ISO/IEC 27001:2022 as a company. Ask any vendor for the same things in writing.
What happens on the next pull request?
Code keeps changing after an audit. Ask whether the tool reviews a pull request alone or against what it learned from the whole protocol, what it does with a large change, and whether a failing result can stop a merge.
QuillShield keeps a security graph from the full audit and checks each pull request against it, so a full audit has to come first. The check fails on a High or Critical finding by default, and you can set another threshold. On GitHub you can make the check required, which blocks the merge, on plans that include branch protection. Checks also run on GitLab.com. A large change is still checked, and the result recommends a new full audit. A project uploaded as a zip gets a full audit and no pull request checks.
A short scorecard you can reuse
| Question to ask | What good evidence looks like | QuillShield's answer |
|---|---|---|
| What does it find on a public benchmark? | A count, a date, who ran it, how it was graded and what was missed | 98 of 117 known vulnerabilities on EVMbench (83.8%), alpha, own run, 8 July 2026, self-reported. The best single model in the original EVMbench paper found 45.6%. Grader and number of runs not yet published |
| How many findings hold up? | The share valid under human review, with the reviewers and sample named | 90% of the findings QuillAudits auditors reviewed on client Solidity code were valid, private beta. Count not published |
| Is there a proof of concept? | A test the tool runs itself, with passes and failures shown | A proof of concept status on findings from full audits. Not every finding has one, and not every one passes |
| What happens to code and findings? | Written terms on deletion, training and disclosure | Source deleted after each audit, no training use, findings disclosed to you alone |
| What happens on the next pull request? | Each change checked against the full audit, with a merge gate | Checked against the security graph. Fails on High or Critical by default |
| Has anyone outside tested it? | A public write-up by someone outside the company | An independent researcher's public test, summarised on the home page |
| What does it support, and how long does it take? | A stated list and real timings | Solidity with Foundry only. Hours for a full audit, 2h 7m for the demo |
What to do next
Run these questions against a repo that has already been audited. Start an audit and compare what comes back with the report you have. New accounts get $50 in credit, good for up to half of an order.
Common questions
What is a false positive in an AI audit?
A false positive is a reported finding that is not a real vulnerability. Each one costs your team review time. During QuillShield's private beta, 90% of the findings QuillAudits auditors reviewed on client Solidity code were valid, so expect to triage some findings that do not hold up.
Should an AI auditor write a proof of concept?
Yes, where it can. A proof of concept is a test that carries out the attack, so a passing one is stronger evidence than a written claim. On a QuillShield full audit, a finding can show a proof of concept status. Proofs of concept do not pass for every finding.
Do AI auditors train on my code?
It depends on the vendor, so read the terms. QuillShield does not. Its terms of service say your code and findings are never used to train a model, ours or a third party's.