How precise is Codecop?

The numbers, how we measure them, the one false alarm we produced in the wild, and what these numbers do not tell you.

Measured on engine 0.1.0. Corpus results from the harness that runs on every push. Wild sweep from 2026-09-01.

Why precision is the number we publish

Most AI code review tools compete on how much they find. We think that is the wrong contest. A reviewer that is wrong one time in five gets muted within a month, and a muted reviewer catches nothing.

So Codecop runs three deterministic checks and stays silent when it is not sure. Every finding carries evidence you can verify yourself: the registry URL that returned 404, the redacted match and the rule that fired, the test block that has no assertion in it. If we cannot show the evidence, we do not post the finding.

That only works if we hold ourselves to it. This page is the measurement.

What the two numbers mean

• Precision: of all the findings the engine produced across the corpus, the share that were correct. 100% means zero wrong findings.
• Seeded recall: of the failures we planted on purpose, the share the engine caught.
• Both are counted per finding, not per pull request. A PR with three planted failures counts three times.
• The harness calls the same runEngine function production calls. Nothing in the checks is mocked. Only the package registry is replaced with an offline stub, so the numbers do not depend on npm being up at test time.

The corpus: 68 cases in three groups

Group	Cases	What it is	Drives
Seeded failures	30	Failures planted on purpose. 8 hallucinated dependencies (fake package in package.json, typosquat of a popular package, fake bare import, subpath import, CommonJS require, dynamic import, fake line in requirements.txt, fake entry in pyproject). 8 leaked secrets (GitHub PAT, AWS-style key, Stripe live key, Slack bot token, GitLab PAT, private key block, high-entropy generic key, a token buried in a big diff). 14 test-theater cases across JavaScript and Python (no assertions, empty body, self-comparison, constant assertion, skips added next to a source change, xdescribe, assert True, pass-only, docstring-only).	Recall
Clean diffs	16	Honest changes the engine must stay silent on. 12 written by hand: a normal feature diff with real dependencies, honest JS and Python tests, a refactor moving code between files, a docs-only change, a PR that removes a leaked secret, a requirements bump of real packages, a config file full of URLs and IDs, mocked tests, a large mixed diff. Plus 4 real merged pull requests, stored as their exact patches: axios #11172, fastify #7003, flask #6133, requests #7609.	Precision
Traps	22	Diffs engineered to look like failures. Silence required. Listed in full below, because the traps are where the work is.	Precision

The 22 traps

• Scoped private package import (@acmecorp/http-client)
• Scoped private package in the manifest
• Requirements file pointing at a private index
• Package allowed via codecop.yml
• scripts and engines keys in package.json
• Workspace and file: dependencies
• Python import whose name differs from its PyPI name (import cv2)
• Fixture key under an allowed secrets path
• Lockfile integrity hashes
• A secret that appears on a deleted line only
• Custom assertion helper, no config
• Unrecognizable helper, allowed via config
• Snapshot-only test
• Pre-existing bad test the PR did not touch
• Negated comparison (expect(a).not.toBe(a))
• JS determinism test comparing two calls
• Python determinism test comparing two calls
• Python skipif platform gate
• Pytest fixture named like a test
• Python docstring that mentions assert
• Fake import inside an ignored vendor path
• Checks disabled by config stay silent

Results per check

Check	Planted	Caught	Wrong findings	Precision	Recall
deps-exist	8	8	0	100%	100%
secrets	8	8	0	100%	100%
test-theater	14	14	0	100%	100%
Overall	30	30	0	100%	100%

The wrong-findings column counts every finding on the clean diffs and the traps as well. Zero means the engine said nothing on all 38 of them.

The gate is in CI, and we proved it fires

The build fails when corpus precision drops under 90% or seeded recall under 80%. A regression in a check cannot ship without turning the build red.

We did not take that on faith. We deliberately broke the tautology rule and pushed: seeded recall fell to 53.3%, the build failed, and we restored the rule. The gate works.

57 real pull requests

A corpus we wrote ourselves can only say so much, so we also ran the engine, read-only, against public pull requests we did not write. Live registries, real gitleaks, same runEngine.

• 33 agent-authored PRs found through GitHub search: the Copilot coding agent, Devin, and PRs whose body says "Generated with Claude Code".
• 24 recent PRs from busy open-source repositories: express, fastify, flask, axios, next.js.
• 36 repositories, 301 changed files. 4 further PRs were skipped as too large for the sweep caps (50 files or 10,000 added lines).

Codecop's authorship detector tagged 43 of the 57 as AI-authored, including Claude Code PRs inside next.js and a Codex PR in flask:

Agent detected	PRs
Claude Code	13
Devin	12
GitHub Copilot	10
Dependabot	6
Codex	1
GitHub Actions	1
None detected	14

Verdicts: 56 LOW, 1 MEDIUM. The engine produced exactly one finding across all 57, and it was wrong.

The false alarm, in full

Pull request 583 on michaelayoade/dotmac_starter_mt, in tests/unit/test_deployment_foundation_execution_plan.py:

def test_the_digest_is_stable_across_renders() -> None:
    """Nothing here reads a clock, an environment variable or a filesystem."""
    assert _rendered().digest() == _rendered().digest()

The tautology rule saw two textually identical sides and flagged it. But each side is a fresh call, and comparing two invocations is exactly how determinism and idempotency tests are written. x == x is a tautology. f() == f() is a test.

What changed the same day:

• Both analyzers now exempt identical-source comparisons that contain a call. JavaScript walks the AST for a call expression; Python checks the compared expression for a call.
• Two regression unit tests, one per language.
• Two new traps in the corpus quoting the wild pattern, so the corpus count went from 66 to 68.
• The pull request was re-checked with the fixed engine: verdict LOW, zero findings.

What these numbers do not tell you

• 68 cases is small, and we wrote most of them. Seeded recall says the checks catch the failure shapes we designed them for. It says nothing about shapes we have not imagined yet.
• The wild sweep measures silence, not catching. We have not yet watched Codecop catch a real hallucinated dependency in someone else's pull request. When it happens, it goes on this page.
• Language coverage is uneven. Secrets scanning runs on every added line in every language. Dependency and test-theater checks cover JavaScript, TypeScript, and Python only.
• Unknown never flags. A registry timeout resolves to "unknown", which is treated as silence. That protects precision and costs recall in the rare minute npm or PyPI is slow.
• Secrets detection is only as wide as gitleaks' rules. A credential shape gitleaks does not know is a credential we do not catch.
• Every number here belongs to a specific engine version. Re-run the harness on the current commit rather than trusting this page.

Reproduce it

The corpus, the harness, and the sweep script ship in the Codecop repository. You need Node 22 and gitleaks on your PATH.

git clone https://github.com/ShreyanshVaibhaw/Codecop
cd Codecop
npm ci
npm run eval        # replays the corpus, prints the per-check table, writes eval/results.json

The wild sweep needs a GitHub token and a Postgres database for the registry cache:

npm run build
GITHUB_TOKEN=$(gh auth token) DATABASE_URL=postgres://... node dist/sweep.js
# writes eval/sweep-results.json: one row per PR with verdict, findings, and the agent detected

See it catch something

The demo pull request is a scripted bad AI change: a hallucinated npm package, a GitHub token in a settings file, and four tests that test nothing. Codecop returned HIGH with 6 findings in 5 seconds, each one an inline annotation with evidence.

Or install Codecop on a repository and open a pull request. The default mode is advisory: it never blocks a merge until you ask it to.