Measure discovery and verification separately

DoorDash built a benchmark because its production acceptance metric could see reactions to comments while missing every issue the reviewer never raised. That leaves an operator judging only the findings that made it through. Trust starts with measuring discovery and verification separately, because acceptance cannot show what the whole system missed.

Acceptance observes only the comments that exist

DoorDash built DashBench because production acceptance can record reactions to comments but cannot record false negatives, the issues a reviewer never raised. That blind spot is fatal to a trust metric. A reviewer can look agreeable because it comments cautiously while missing the changes that deserved attention.

The report also treats human acceptance as product telemetry rather than ground truth. Authors may accept or reject a comment for reasons tied to timing, urgency, ownership context, or the cost of the fix. Those reactions still matter. They just measure a different thing from whether a finding was real.

The operator’s job is to keep those meanings apart.

• Discovery asks how much relevant ground the system covers.
• Verification asks how well it rejects suspicions that do not survive scrutiny.
• Adoption asks whether people use the output.

Mixing the three produces a pleasant dashboard and a weak control surface.

Staging makes the trade visible

DoorDash’s production reviewer assigns a lead scout to flag suspicious areas, then sends the strongest leads to deep reviewers that confirm or discard each concern. The architecture creates two places to inspect failure. A missed lead belongs to discovery. A weak claim that reaches an author belongs to verification.

DoorDash’s 105-case report makes the coverage, precision, cost, and latency trade visible.

Trustworthy evaluation leaves that bill on the table. Operators can then decide whether added coverage is worth the spend and delay for their own risk profile.

A human proof gate changes what confirmed means

Mandiant’s AVDH report keeps four review stages separate:

1. Discovery: parallel agents scan every in-scope file and extract entry points and input sources.
2. Specialist analysis: Access Control and Data Flow Analysis agents turn enriched context into vulnerability hypotheses.
3. Agent challenge: multiple Validation agents test each hypothesis before a synthesis agent classifies it.
4. Expert proof: a consultant replicates the exploit, runs proof-of-concept code, and discards findings that fail human review.

Mandiant reports ten months of AVDH use. In one incident response investigation involving stolen corporate repositories, Mandiant reports over 100 true-positive critical vulnerabilities found in two days. Across the wider program, Mandiant also reports 12 assigned CVEs. A CVE is a public identifier for a disclosed vulnerability.

An agent-confirmed finding has not crossed Mandiant’s proof gate. Human exploit proof remains a separate result that operators can count and inspect. Mandiant and Google neither use nor endorse muniment.

A benchmark is a local instrument

DoorDash held the case set, context track, prompt and skill-pack selection, output contract, and grading pipeline fixed within a comparison. That discipline makes changes inside the report interpretable. It does not turn one company’s historical pull requests, labels, severity weights, or operating costs into a universal model ranking.

There is another staged result in the same report, and it should stay separate. A Kimi K2.6 scout paired with a Claude Fable 5 reviewer reached 65.2% weighted recall and 89.2% weighted precision. Those are model-mix experiment values, not the 53.6% recall and 87.0% precision reported for DoorDash’s production staged reviewer.

DoorDash is neither a Muniment customer nor an endorser. Its report is useful public evidence because it exposes the seams that an operator can measure. If production acceptance is the trust metric, who is counting the issues that never earned a comment?