What the research says about fan-out-then-verify, adversarial review, and judge panels

In my post on dynamic workflows I wrote that workflows let you bake quality patterns like fan-out-then-verify, adversarial review, and judge panels into a script. I then went and read the research behind those three patterns, because "more agents checking each other" sounds like it should work and I wanted to know if it does.

Mostly it does, but not for the reasons the names suggest. The gains come from two places. The first is sampling several independent answers and voting. The second is a checker that knows something the generator doesn't, such as a test suite, a different model, or a reviewer with a fresh context. Agents arguing with each other and panels of judges that share a base model mostly cost tokens.

The short version:

• Generating more attempts is cheap and reliably finds more right answers. Picking the right one is the hard part, and an imperfect checker caps how good the result can get.
• Majority voting explains most of what's been credited to multi-agent debate. At equal compute, plain voting usually wins.
• Asking a model to critique its own answer, with no outside signal, often makes the answer worse.
• Judge panels beat a single judge mainly when the judges come from different model families. Even then, nine judges behaved like about two independent votes in a 2026 study.
• In Claude Code, you can get all of this with a one-line prompt, a saved workflow, a reviewer subagent on a different model, or a Stop hook that runs your tests.

All papers below were checked against their full text in October 2026, and I looked for peer reviews or replications where they exist.

Generating is easy, picking is hard

The cleanest evidence comes from inference-time scaling, which means spending more compute when you run a model rather than when you train it.

Large Language Monkeys (Brown et al., 2024) measured "coverage," the share of problems where at least one of many attempts is correct. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct solved 15.9% of problems with one attempt and 56% with 250 attempts. The best single-attempt result at the time was 43%. So the right answer was usually somewhere in the pile.

Getting it out of the pile was the problem. Without an automatic checker like unit tests, the paper found that majority voting and reward models "plateau beyond approximately 100 samples" while coverage keeps climbing. (The abstract says "several hundred"; the body says about 100.) The paper was rejected at ICLR 2025, with reviewers arguing the coverage curves are what you'd expect mathematically. That's fair, and it doesn't change the gap between finding and picking.

Stroebl, Kapoor, and Narayanan put a hard number on why. If your checker ever accepts a wrong answer, resampling can't push that error rate down, so there's a ceiling "regardless of compute budget." On coding benchmarks with thin unit tests, they found "optimal sampling attempts are often fewer than 10." It was accepted at ICLR 2026 as a borderline poster; reviewers worried it generalizes too much from coding.

Checkers also get fooled in embarrassing ways. One Token to Fool LLM-as-a-Judge (Zhao et al., 2025) sent judges answers that contained no reasoning at all, just a colon or the words "Thought process:", and counted how often the judge marked them correct. Averaged across their ten "master key" responses and five benchmarks:

A lone ":" got Claude-4 to accept 53.6% of answers on MATH. The paper was rejected at ICLR 2026, partly because reviewers saw it as another form of reward hacking and the judges weren't the newest. I still find it a useful warning. A judge that only reads the answer can be fooled by its shape.

Fan-out-then-verify

This is the pattern with the strongest evidence. Fan out several attempts, check them, and keep what survives.

The simplest version is voting. Self-consistency (Wang et al., ICLR 2023) samples 40 reasoning paths and takes the most common answer. On the GSM8K math benchmark it lifted PaLM-540B from 56.5% to 74.4%, a gain of 17.9 points. More Agents Is All You Need (Li et al., TMLR 2024) found the same effect across model sizes. With 15 samples and a vote, Llama2-13B matched a single Llama2-70B on GSM8K. Voting has limits, though. Chen et al. showed that past a point, more calls can make voting worse, because extra samples help on easy questions and hurt on hard ones.

A trained checker does better than a vote. OpenAI's GSM8K paper (Cobbe et al., 2021) sampled 100 solutions per problem and let a trained verifier pick one. They said this gave "approximately the same performance boost as a 30x model size increase." The same paper also found the verifier got worse past about 400 samples, because more samples meant more wrong answers good enough to fool it. Let's Verify Step by Step (Lightman et al., ICLR 2024) compared checking each step against checking only the final answer, picking the best of 1,860 attempts on MATH problems:

The authors say the two checkers were trained on different data and "are not directly comparable," and ICLR reviewers flagged reproducibility because the model sizes weren't disclosed. A 2025 Qwen paper also argues that best-of-N tests flatter step checkers, so treat the exact gap loosely.

This is how Anthropic's own tools are built. The Claude Code Review docs say "a verification step checks candidates against actual code behavior to filter out false positives." The public code-review plugin shows the shape. Four reviewers run in parallel (two on Sonnet for CLAUDE.md rules, two on Opus for bugs), then a separate subagent validates each issue, and anything not validated is dropped. The built-in /deep-research workflow "votes on each claim," and when a verifier can't check a claim, say after a rate limit, it lists the claim "as unverified instead of counting it as refuted." I think that last rule matters more than it looks. A checker that fails is not a checker that disagrees.

Adversarial review and debate

Adversarial review means a second agent tries to break the first agent's answer. Debate is the multi-round version, where agents read each other's answers and revise.

Debate had a strong start. Du et al. (ICML 2024) ran three GPT-3.5 agents for two rounds and got 85.0% on GSM8K, against 81.0% for a majority vote and 77.0% for a single agent. But debate used more model calls than the vote it beat, and each benchmark had only about 100 questions.

When others matched the compute, the advantage went away.

• Huang et al. (ICLR 2024) re-ran debate on the full GSM8K set. With 9 responses, debate scored 83.0% and self-consistency scored 88.2%.
• Should we be going MAD? (Smit et al., ICML 2024) found debate didn't reliably beat self-consistency or ensembling. On MMLU with GPT-3.5, self-consistency scored 0.78 and a debate setup scored 0.73.
• Stop Overvaluing Multi-Agent Debate (Zhang et al., 2025) tested 5 debate methods on 9 benchmarks with 4 models, and found no method beat plain chain-of-thought in more than 20% of the 36 settings. It's a position paper, not peer-reviewed.
• Debate or Vote (Choi, Zhu, and Li, a NeurIPS 2025 spotlight) concluded that majority voting "alone accounts for most of the performance gains typically attributed to" debate. They also proved that debate on its own is a martingale, meaning the expected accuracy doesn't improve from round to round.

More rounds made it worse in that study. A big part of the reason is conformity. Talk Isn't Always Cheap (Wynn, Satija, and Hadfield, an ICML 2025 workshop paper) found agents flip from right to wrong more often than wrong to right, and a correct agent is most likely to cave when it's the only one holding that answer. Three Mistral agents on MMLU went from 33.6% before debate to 24.4% after.

Self-critique without an outside signal has the same problem. Huang et al. asked models to review and fix their own answers with no new information. GPT-4 on GSM8K dropped from 95.5% to 89.0%, and GPT-3.5 on CommonSenseQA fell from 75.8% to 41.8%. The model mostly talked itself out of right answers.

So where does adversarial review work? When the reviewer brings something new. Cemri et al.'s study of 1,642 multi-agent traces (NeurIPS 2025) found that verification problems, such as checking too little, checking wrongly, or stopping early, caused 23.5% of failures. Two 2026 security preprints show what a good adversarial checker looks like in practice.

• Refute-or-Promote (Agarwal) used "kill" agents to reject about 79% of 171 vulnerability candidates, leading to 4 CVEs. A critic from a different model family found problems in 3 of 19 fixes that same-family reviewers had approved. Its two failure stories are the most useful part. Ten agents unanimously endorsed a false finding that one empirical test killed, and one real vulnerability was unanimously killed and later became CVE-2026-41254.
• OpenAnt (Korda and Evron) had agents simulate an attacker against each finding, which cut 376 findings to 190. Of those 190, 144 (75.8%) produced working exploits.

Both are single-team preprints with no ablation, so they show the pattern can work, not how much it adds. I looked for a controlled study comparing a "try to refute this" prompt against a neutral "check this" prompt at equal compute and didn't find one, in more than ten targeted searches. The closest is Qiu and Gill, where making disagreement explicit raised F1 from 0.457 to 0.533 on 100 examples. If you know of a better study, I'd like to read it.

Judge panels

A judge panel has several models score the same output and combines their votes. It exists because single LLM judges have well-documented biases. The original MT-Bench paper (Zheng et al., NeurIPS 2023) found GPT-4 agreed with human raters 85% of the time, slightly more than humans agreed with each other (81%). But it also found:

• Position bias. When the same answer pair was shown in swapped order, GPT-4 gave a consistent verdict only 65.0% of the time, and Claude-v1 23.8%.
• Verbosity bias. Padding an answer with a repetitive list fooled GPT-3.5 and Claude-v1 91.3% of the time (GPT-4, 8.7%), though only 23 answers were tested.
• A possible preference for its own answers, which the authors said they couldn't confirm with their data. Panickssery, Bowman, and Feng (NeurIPS 2024) later showed GPT-4 could pick out its own summaries 73.5% of the time, and that self-preference rose with self-recognition. A 2025 follow-up found much of that preference is earned, because the stronger model's output really is better.

Pairwise or Pointwise? (Tripathi et al., COLM 2025) adds a practical note. When distracting features were added to answers, pairwise "which is better" verdicts flipped in about 35% of cases, against about 9% for scoring each answer on its own.

The case for panels comes mostly from Replacing Judges with Juries (Verga et al., Cohere, 2024). A panel of three smaller models from three families (Command R, Claude 3 Haiku, and GPT-3.5) agreed with human annotators more than GPT-4 alone on all three QA datasets tested and was "seven to eight times less expensive." It has two caveats. Claude 3 Haiku alone beat the panel on one dataset. And when the authors gave GPT-4 a prompt that told it not to overthink, GPT-4 scored 0.725 on Natural Questions against the panel's 0.699. I couldn't find an independent replication or released code.

The case against is correlated errors. Kim et al. (ICML 2025) compared hundreds of models and found that when two models both get a HELM question wrong, they pick the same wrong answer 60% of the time, against 33% by chance. Models from the same provider agree more, and more capable models agree more. Goel et al. (ICML 2025) found judges score models higher the more similar those models are to themselves, with an average correlation of 0.84.

Nine Judges, Two Effective Votes (Kohli, 2026) measured this directly. Nine frontier judges from seven families behaved like 2.0 to 2.5 independent votes. The panel scored 8 to 22 points below what truly independent votes would have reached, and the best single judge matched or beat the panel on every dataset.

It's a single-author preprint on classification tasks that were picked for being ambiguous, and you'd need the answer key to know which judge is "best" in advance. But a follow-up with 32 judges reproduced the roughly two-vote result under the same measure, while showing other measures give between 2 and 6.5.

I read the two sides as consistent. A panel helps when you can't pick a strong single judge, when its members come from different families, and when the task has a reference answer. It helps much less when every judge shares the same blind spot. One note on a source people cite against panels. Anthropic's multi-agent research post says "a single LLM call with a single prompt" was more consistent than "multiple judges," but its multiple judges were one per scoring criterion, not a panel of different models.

How to build these in Claude Code

There are four levels, from least to most setup.

Ask in the prompt. Any prompt with the word ultracode, or one that says "use a workflow," makes Claude write a workflow script for the task. The docs' own example of fan-out-then-verify is:

use a workflow to audit every route handler under src/routes/ for missing authentication checks, and adversarially verify each finding before reporting it

For loop-until-dry, the docs suggest "stop once two rounds in a row find nothing new." Since v2.1.210 the keyword only works in prompts you type, not in -p runs or relayed PR comments. /effort ultracode turns on workflow planning for every task in the session, and /deep-research <question> runs the built-in research workflow.

Save the script. When a run does what you want, open /workflows, select it, and press s. It saves to .claude/workflows/ (shared with the repo) or ~/.claude/workflows/ (just you) and runs as /<name> afterward. Arguments reach the script as a global named args. The adversarial check from the /workflow-authoring reference looks like this:

const votes = await parallel(Array.from({length: 3}, () => () =>
  agent(`Try to refute: ${claim}. Default to refuted=true if uncertain.`, {schema: VERDICT})))
const survives = votes.filter(Boolean).filter(v => !v.refuted).length >= 2

Given the research above, I'd change two things. Give each skeptic a different job instead of three copies of the same prompt, and give at least one of them a way to run something. Here's my sketch of a judge panel that does both, with each judge on a different model and scoring answers one at a time instead of in pairs:

const JUDGES = [
  {model: 'opus',   lens: 'correctness: run the tests and check edge cases'},
  {model: 'sonnet', lens: 'does it meet every requirement in the task?'},
  {model: 'haiku',  lens: 'what would a maintainer object to?'},
]
const SCORE = {type: 'object', properties: {score: {type: 'number'}, reason: {type: 'string'}},
               required: ['score', 'reason']}

const scored = await parallel(candidates.map((c, i) => () =>
  parallel(JUDGES.map(j => () =>
    agent(`Score this candidate from 0 to 10 on ${j.lens}. Do not compare it to others.\n\n${c}`,
          {model: j.model, schema: SCORE})))
    .then(vs => ({i, total: vs.filter(Boolean).reduce((s, v) => s + v.score, 0)}))))

The per-agent model option counts as the per-invocation model in the subagent model order. All three are still Claude models, though. Claude Code doesn't route to non-Claude models, so a judge from another family has to come in through an MCP tool or an HTTP hook.

Use a reviewer subagent on a different model. A file in .claude/agents/ with a model field gives you a reviewer that doesn't share the main session's context or, if you choose, its model:

---
name: skeptic
description: Tries to refute a claimed fix or finding. Use before reporting a bug or closing a task.
tools: Read, Grep, Glob, Bash
model: opus
---

Assume the claim is wrong. Reproduce it by running code or tests where you can.
Reply REFUTED, CONFIRMED, or UNVERIFIED with the evidence. If you couldn't check it, say UNVERIFIED.

Make your tests the judge. A Stop hook can block Claude from finishing until a check passes. This agent hook, straight from the docs, runs the test suite first (agent hooks are marked experimental):

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "agent",
            "prompt": "Verify that all unit tests pass. Run the test suite and check the results. $ARGUMENTS",
            "timeout": 120
          }
        ]
      }
    ]
  }
}

For code review specifically, /code-review takes an effort level from low (fewer, more confident findings) to max, and the ultra mode says "every reported finding is independently reproduced and verified."

What it costs

Verification is where the tokens go. Anthropic's research-system post says multi-agent systems use about 15 times the tokens of a chat. The best real-world number I found is in a public workflow script, adversarial-review.js in rosenfieldben/nyc-transit-live. Its header records a review that ran 6 finders with 3 verifiers per finding. It used 123 agents and about 8.3M tokens, and 117 of those agents were verifiers. Re-run with 29 agents and about 1.6M tokens, it still caught both critical bugs. The same script marks a finding UNVERIFIED rather than refuted when a verifier read the wrong commit, after that caused real bugs to be thrown out.

Claude Code shows a Large workflow warning when a run schedules more than 25 agents or is projected past 1.5 million tokens, and the default size guideline asks Claude to stay under 10 agents (under 5 on Pro). I'd start with one verifier per finding and only add more when the first round lets bad findings through.

Open-source options

If you don't use Claude Code, or want to see how others built these, here's what I found on GitHub. Stars and licenses were checked on October 11, 2026.

Anthropic's code-review plugin is worth reading as a reference, but it isn't open source; the repo's license is "All rights reserved." Its README also still describes an older 0 to 100 confidence filter that the current command file replaced with validation subagents. Licenses matter here because three of the most-starred projects in this list have none, which means you can read the code but not legally reuse it.

What I'd do

My own results with these patterns have been mixed. Verification has caught real problems for me, and it has also burned tokens confirming things that were already right. The research above explains both.

My rules, after reading all of this, are short. Sample and vote before you reach for debate. Make the checker as independent as you can, preferably by running something, otherwise by using a different model or at least a fresh context. Treat "couldn't check" as its own result, not as a rejection. And spend verification tokens on the findings you're about to act on, not on every candidate.

If you only set up one thing, save a review workflow that adversarially verifies its findings, and run it on every branch. Save it once, and the checking happens every time without you having to ask for it.

Sources

Papers (checked against full text, October 2026):

• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023
• Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021
• Lightman et al., Let's Verify Step by Step, ICLR 2024 (reviews)
• Brown et al., Large Language Monkeys, 2024 (reviews)
• Li et al., More Agents Is All You Need, TMLR 2024
• Chen et al., Are More LLM Calls All You Need?, NeurIPS 2024
• Stroebl, Kapoor, and Narayanan, The Limits of Inference Scaling Through Resampling, ICLR 2026 (reviews)
• Zhao et al., One Token to Fool LLM-as-a-Judge, 2025 (reviews)
• Du et al., Improving Factuality and Reasoning in Language Models through Multiagent Debate, ICML 2024
• Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024
• Smit et al., Should we be going MAD?, ICML 2024
• Zhang et al., Stop Overvaluing Multi-Agent Debate, 2025
• Choi, Zhu, and Li, Debate or Vote, NeurIPS 2025
• Wynn, Satija, and Hadfield, Talk Isn't Always Cheap, ICML 2025 workshop
• Cemri et al., Why Do Multi-Agent LLM Systems Fail?, NeurIPS 2025
• Agarwal, Refute-or-Promote, 2026 preprint
• Korda and Evron, OpenAnt, 2026 preprint
• Qiu and Gill, Adversarial Review, 2026
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023
• Panickssery, Bowman, and Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024
• Qwen team, The Lessons of Developing Process Reward Models in Mathematical Reasoning, 2025
• Chen et al., Do LLM Evaluators Prefer Themselves for a Reason?, 2025
• Tripathi et al., Pairwise or Pointwise?, COLM 2025
• Verga et al., Replacing Judges with Juries, 2024
• Kim et al., Correlated Errors in Large Language Models, ICML 2025
• Goel et al., Great Models Think Alike and this Undermines AI Oversight, ICML 2025
• Kohli, Nine Judges, Two Effective Votes, 2026 preprint
• Li, Yu, and Li, How Many Humans Are 32 LLM Judges Worth?, 2026 preprint

Docs and posts:

• Claude Code docs: dynamic workflows, subagents, hooks guide, code review, ultrareview, LLM gateway
• Anthropic, How we built our multi-agent research system, June 2025
• Anthropic, code-review plugin command
• rosenfieldben, adversarial-review.js