How to add smart contract security checks to CI on GitHub
To add a smart contract security check to every pull request on GitHub, run a full QuillShield audit on your base branch, install the GitHub App, turn on Scan Pull Requests and choose the severity that fails the check. Then require the check in GitHub's branch protection, and a pull request with a failing check cannot merge.
When you finish, every pull request in your GitHub repository gets a QuillShield check that compares the change with your last full audit and fails at the severity you chose. Once you make that check required in GitHub, a failing check blocks the merge. The slow part is the full audit that becomes the baseline. The demo vault on the QuillShield home page took 2h 7m, and larger codebases take longer.
What should a pull request security check do?
A pull request security check should do three things: judge the change against the whole protocol and not the diff alone, fail at a severity the team has chosen, and be able to stop the merge. QuillShield does the first by checking each pull request against the security graph from your last full audit. The steps below set up the other two.
What do you need before you add the check?
- A GitHub repository with a Solidity project built with Foundry. A project uploaded as a zip, a folder or a single file cannot use checks.
- The admin or owner role in your QuillShield organization.
- Rights on GitHub to install an App on the repository or organization, and to change its branch rules. Required checks need branch protection or rulesets, which GitHub offers on public repositories and, for private repositories, on its Pro, Team and Enterprise plans.
- A completed full audit, which step 2 covers. Each one comes with five pull request checks, and more can be added.
The docs call these checks light scans, and the dashboard page is CI/CD Checks.
How do you add the check to a GitHub repository?
- 1 Create a git project. In QuillShield, open Projects, click New project, choose the GitHub tab and pick the repository. If GitHub is not connected yet, connect it under Integrations in Settings.
- 2 Run the full audit that becomes the baseline. Audit the CI base branch, which starts out as the repository's default branch. If your workspace shows a Workflow step, keep the default QuillShield Audit Agent, because an audit run with another workflow does not become a baseline. Run your first audit has the steps.
- 3 Install the GitHub App. Open CI/CD Checks at shield.quillaudits.com/ci, go to Setup & automation and install the QuillAudits GitHub App on the repository or the organization. Authorize it when GitHub asks, because an install without that step is refused. The App asks to read metadata and contents, and to read and write pull requests, checks and commit statuses.
- 4 Turn on pull request checks. In Setup & automation, switch on Scan Pull Requests. Both switches are off until you turn them on. Scan Every Commit checks pushes to branches that have no open pull request.
- 5 Choose the fail threshold. Set Fail check on to the lowest severity that should fail the check: Critical, High or above (the default), Medium or above or Low or above. A change affects later checks only. Existing reports keep the threshold they ran with.
- 6 Run one check. A check starts when a pull request is opened, updated, reopened or marked ready. To start one by hand, open Pull Requests, choose the pull request and click Run CI Scan.
- 7 Make the check required in GitHub. The QuillShield docs do not cover this step, because it is GitHub's own setting. In the repository on GitHub, open Settings, then Branches or Rulesets, and add or edit the rule for your base branch. Turn on the option that requires status checks to pass before merging, and select the QuillShield check. Its exact name is the one shown on your pull request from step 6.
What does the pull request check cover?
- On each pull request, a GitHub Check, a commit status and one comment that is updated in place.
- A report for every check in QuillShield. It can mark findings from the baseline audit that are still present, and it gives a verdict for the pull request.
- A check on large changes too. When a change adds new
.solfiles, more than 150 added Solidity lines or more than 5 changed.solfiles, the report recommends a new full audit. It does not block.
The switches only run checks and never start a full audit. A .solidity-review.yml file, @quillshield commands and merge queue orchestration are not supported.
If you cannot install the App, copy the workflow from Use GitHub Actions and add the secrets QUILLAUDITS_API_KEY and QUILLAUDITS_ORG_ID. It calls POST /v1/ci/scans/trigger and does not check out or run your code. Pull requests from forks do not receive secrets, so they need the App.
On GitLab.com, connect GitLab and turn on Scan Merge Requests, and QuillShield posts a commit status named QuillShield Check and one note on the merge request. On the GitLab Free plan you cannot make that status required the way you can on GitHub, and self-managed GitLab is not covered.
Common problems with pull request checks
| What you see | What to do |
|---|---|
| HTTP 409, or no baseline | The commit being checked does not descend from a completed full audit on the CI branch. Run a full audit on that history, or open Setup & automation, find Audited baseline and click Change base branch |
| The check or status did not post | Check that the App still has access to checks, statuses and pull requests, and that Scan Pull Requests is on |
| A check failed or was cancelled | Open the pull request in QuillShield and click Retry CI Scan. Run with new threshold is on the same page |
| Wallet error, or HTTP 402 from GitHub Actions | The checks that came with your full audit are used up. Add balance under Billing. GitHub Actions spends from the organization owner's wallet |
Full reference in the QuillShield docs
CI/CD Checks in the QuillShield docs is the full page. Connect GitHub lists what the App can read and write.
Common questions
Can a failing check stop a merge?
Yes, on GitHub, where your plan includes branch protection or rulesets. Make the QuillShield check required and a pull request cannot merge while it has a High or Critical finding, or one at the threshold you set.
Why do I need a full audit before pull request checks?
Each check compares the change with the stored security graph of your last full audit. If no completed full audit is in the history of the branch, the check has no baseline and does not run.
What is the QuillShield check called in GitHub's branch protection settings?
The QuillShield docs do not state the name. Use the name shown in the checks list on your own pull request.
Do pull request checks work on GitLab?
On GitLab.com, yes. QuillShield posts a commit status named QuillShield Check and a note on the merge request. On the GitLab Free plan you cannot make that status required the way you can on GitHub, and self-managed GitLab is not covered.
Sources
- CI/CD Checks, QuillShield docs
- Connect GitHub, QuillShield docs
- CI/CD Checks for developers, QuillShield docs
- Connect GitLab, QuillShield docs
- Workflows, QuillShield docs
- Core concepts, QuillShield docs
- Review findings, QuillShield docs
- Create a project, QuillShield docs
- Why isn't my integration working?, QuillShield docs