When the machine finds more
than people can check

What happens when a testing tool produces candidates faster than a practice can check them? In my independent run of Capital One's open-source VulnHunter against OWASP's deliberately vulnerable crAPI app, the tool left me with 46 candidate packages, and its generated tests reproduced the predicted behavior for 39. Human review was not completed, so none became an accepted finding (field study).

Before they reached me, the candidates went through layered checks inside the tool: hunt gates, a pass that tries to disprove each one, a generated exploit test, and a sweep for other instances of the same root cause (VulnHunter). That layering is real, and I think it makes the queue better than raw scanner output. What it can't supply is independence: the same tool wrote the rebuttals and the tests. In this workflow the independent check is a person, and that's the check volume presses on.

Text description of Figure 1

Four layers of checks are inside VulnHunter. A person is the independent check, which this run did not complete.

There's good reason to expect more of this. In a university-network study, the authors' own agent, ARTEMIS, outperformed nine of ten cybersecurity professionals, while AI agents also produced more false positives. XBOW reports nearly 1,060 bug-bounty submissions overall; in one 90-day period, 54 of the 885 rated by program owners were critical and 242 high. Elastic reports more submissions in the first half of 2026 than in 2024 and 2025 combined, at 30 to 60 minutes of a senior engineer's time each. In source-code review of 11 Python web apps, Semgrep found false-positive rates of 86% and 82% for two AI coding agents in its 2025 test. These are different settings and denominators, not a pooled result, but together they suggest review is becoming the constraint. For anyone holding a testing budget, the cost shows up in review hours, not tool hours. Writing about bug-bounty platforms generally, Bugcrowd's David Brumley put it plainly: the bottleneck is "no longer report generation—it's validation."

Software can send back incomplete packages or replay behavior, but a qualified person has to own what becomes a delivered finding. This piece follows the queue through a practice, including a small review aid of mine that showed how a label can sound like a verdict before anyone has earned one.

• Candidate: something a tool reports that might be a vulnerability, with the tool's own severity. A reason to look, not a conclusion. A candidate package bundles the claim, its evidence and its rating.
• Finding: a candidate a qualified person has reproduced, judged and accepted for the report.
• Validation: the work that turns a candidate into a finding, or rules it out.
• Attack chain: several weaknesses used together, each making the next possible. A chain can be more severe than any part, and every link has to hold.

That run shows how much work reached the review boundary, not a detection or false-positive rate. Open the Practice framed this as a staffing problem more than a model problem, and The Leverage Between Penetration Tests ended at the same review step.

The practice framework I published names the stages each candidate passes through, including candidate, human validation, review, delivery and correction. The sketch below groups them by who acts. None of that is new; OWASP's testing guide already says to "weed out any false positives" before a report goes out.

Text description of Figure 2

A candidate starts as a tool claim. Software can prepare the review. A person decides what is delivered, and a later correction can withdraw a finding.

Without a better route, volume lands on people. Validation and review take qualified hours. More accepted findings mean more tickets for a client's remediation team, and any correction has to reach those tickets too. Chains cost the most: proving one in a live environment can mean running the exploitation again, inside the rules of engagement and a change window. A route under that pressure is the one most likely to let the tool's own words stand in for a decision.

Working with AI coding agents, I built a review card: a small screen that gathers a candidate's evidence so a reviewer doesn't have to hunt. It decided nothing, sent nothing anywhere, and used fictional cases. To let new cases arrive without code changes, I let each case bring its own observation codes: short labels describing what was seen.

To test that, I gave one case the code VULNERABILITY_CONFIRMED. Its integrity checks, hashes of its files, passed because the files matched the fingerprints the case came with, and the card displayed the label as written. A reviewer would now see a field that read as though confirmation had happened, when the only one who'd said so was the case's author.

Text description of Figure 3

Three fields look alike. The observed bytes were hashed; the case author asserted VULNERABILITY_CONFIRMED; no reviewer had decided.

In a real pipeline, that word could be the model's own phrasing, or text from the target itself (a code comment reading "confirmed safe") carried through to the reviewer. It's a cousin of indirect prompt injection, aimed at the person instead of the model. Reserving verdict words for the decision step helps, but new wording ("exploit succeeded", "validated by test") can imply the same thing.

The hashes were right about the bytes and silent about their meaning. I think the interface has to show which fields were observed, asserted or decided, so that formatting can't lend one source another's authority.

I treat each lever as a hypothesis until it beats the raw workflow on a named task. Six I'd test first:

• Completeness checks that bounce a candidate missing its request, response or conditions back to the tool, not the tester. Plain software.
• Ready-to-run checks. VulnHunter already writes an exploit test for each candidate; the lever is shaping that into something a reviewer can run and read. Each candidate arrives with the exact request to replay, the owner's legitimate request as a baseline, a control that shows what a refusal looks like (the same call with no credentials, or for an object that doesn't exist), and the difference that would show the claim holds. The same script becomes the retest after a fix, and a retest should look for the expected refusal, not just a different response. Because the script comes from the tool, I'd want a person to read it before running it, and in a live environment replay is testing under the same rules of engagement. Software prepares; a person runs it and reads the result.
• Retrieval of the exact source lines a reviewer needs, including the ones that cut against the candidate. Anything that summarizes can hide the line that argues against the summary (see Notes). Software, maybe an agent.
• Grouping candidates that are really the same issue, without merging away the link that makes a chain work. An agent, checked by a person.
• Structured abstention, so "not enough evidence" is a recorded outcome, not a silent drop. A person's judgment, easy to record.
• Recorded capacity, so "not reviewed: no capacity" is a visible state with a reason and an age, and unreviewed candidates don't quietly vanish. A person's call, made visible.

I'd start with completeness checks (deterministic, and they decide nothing), then ready-to-run checks, which turn "go reproduce this" into "run this and look."

My framework's AI extension, Section 90, is my attempt to go past "keep a human in the loop" and say how a practice could measure validation when a tool adds volume. Two of its rules carry the weight here. A qualified person decides every delivered finding, with full validation for critical results (90.1). And a missing answer key is reported as "unknown", never as zero misses (90.3). OWASP's Autonomous Penetration Testing Standard covers the testing platforms themselves; Section 90 covers the practice that receives their output.

When a tool promises to cut validation work in half, these are the questions I'd want to work through with whoever built it, me included.

Text description of Figure 4

A proposed claim of half the validation work can refer to different denominators: candidate volume, reviewer hours, accepted findings, linked issues and correction time.

The denominator. Half of what, and does a five-step chain count once or five times?

The severity gap. How often does the tool's rating survive a person's? If most tool-rated criticals come back as mediums, the queue was less urgent than it looked. And any sampling should start from a person's rating, not the tool's, because a "low" can be one link in a chain.

The misses, in both directions. A workflow can appear to eliminate false positives by rejecting everything. One way to count misses is to plant known-real issues and see how many come out the other end.

The baseline. Compare the assisted workflow with the same reviewer doing the same task from the raw material, not with "nothing".

Who decided. For every field a reviewer sees, can you tell whether a tool observed it, an author asserted it or a person decided it, and when?

The correction path. Withdrawing a delivered finding is a workflow, not a notification. Tickets may be open, exceptions requested and remediation clocks running. How long until every one is unwound?

Take one result from that crAPI run: the tool reported that one user could read another user's vehicle location. crAPI's published challenges include that flaw, so whether it's real may feel settled. The harder question is what you'd need before signing it: the request, the response, the affected user, the conditions, and someone willing to say "I checked."

Here's a test I'd suggest. Take a review aid you own or are building. Change one field to a confident, verdict-like word nobody earned, and follow it through every summary, heading, export and status a reviewer sees. Is its origin obvious? Can the reviewer still reach the evidence against it?

If you'd like to try the review step yourself, the field study's synthetic first-use exercise walks through a review packet up to the point where a person has to decide.

I expect machines to keep producing more candidates, including real weaknesses. The job is keeping three things visibly apart: what was observed, what was asserted, and what a person checked and decided.