AI for Literature Review: A Verification Loop Caught My 80% Claim

The first version of the literature synthesis stated that 80% of studies in an aggression-paradigm corpus, built around the voodoo doll task, misapplied OLS-style methods to zero-inflated count data. It sounded right. It read as a competent, citable finding, and it slotted smoothly into the methodological argument of the paper I was working on. The trouble was that the verification pass immediately contradicted it. When I spot-checked the per-paper extraction records, the real figures were different: 71% of the voodoo-doll-task papers used appropriate count models (negative binomial or Poisson), the rest used Gaussian methods on clearly non-Gaussian outcomes, and 0% of the corpus used the hurdle specification that methodological sources in the same literature recommend. The methodological critique was real. The draft had stated a different one.

This post is about why that happened, what the verification pass actually does, and what I think it implies for anyone using AI to produce content that’s supposed to survive a second reader. It’s a narrow finding, but I think the generalization is broader than it looks.

Table of Contents

• What the AI Literature Review Skill Does
• Why Aggregate Claims Are Where Single-Pass Extraction Breaks
• How the Verification Pass Checks AI Extraction
• The Vault Quality Gate: Never Cite Stubs
• What It Costs and What It Gains
• The Wider Pattern

What the AI Literature Review Skill Does

The literature review skill is a goal-driven pipeline inside a paper-writing workflow. It takes a research question as input and runs four stages: discovery (scan a local source vault and the web for candidate papers), curation (rank which candidates are relevant to this paper’s angle), deep-scan (read each relevant paper through the lens of the research question and extract structured notes), and verification (spot-check the extracted claims against the sources before anything gets written into the wiki).

The reason this exists as a structured skill is that literature reviews are where AI collaboration gets most seductive and most dangerous. Seductive, because a model can read 50 papers in the time it takes a human to read five. Dangerous, because a model can also summarize 50 papers into one plausible-sounding paragraph that doesn’t survive spot-checking. Version one of the skill had the first three stages. Version two added the verification pass after the incident in the opening paragraph. The incident is the only reason this post exists.

Why Aggregate Claims Are Where Single-Pass Extraction Breaks

A structured extraction across a corpus produces two kinds of outputs. The first is per-paper: this paper used method X, reported result Y, collected data of type Z. Those are local claims, each anchored to a single source, and you can verify them by going back to that source. The second is aggregate: of the 50 papers, N used method X, the mean sample size was Y, the dominant framing was Z. Those are global claims that sit above the per-paper records and describe the corpus as a whole.

The failure mode is that aggregate claims are where plausibility wins over accuracy. When a model synthesizes a corpus, it produces a narrative-level summary that integrates what it has read. That summary sounds computed. Often it was generated. A plausible “80% of studies misapplied this method” reads as a citable statistic, but unless something in the pipeline actually counts per-paper method choices and compares them to a criterion, the 80% is a vibe. Nobody tallied it.

On the VDT corpus, the real story was that roughly three-quarters of the voodoo-doll-task papers had already moved to appropriate count models, a long tail still used Gaussian methods on visibly non-Gaussian data, and a specific advanced approach (hurdle models for zero-inflation) wasn’t used in any paper in the corpus despite being recommended in several of the methodological papers in the same corpus. The corrected finding is more specific, more defensible, and more useful for positioning a new paper’s contribution. But it would never have emerged from a single-pass narrative summary, because narrative summaries are built to sound coherent, and nothing in them forces the counts to add up.

I’ve come to treat this as a general pattern. Individual paper extractions are usually right. Aggregate claims are usually plausible, sometimes right, and never trustworthy by default. The interesting methodological work in an AI-driven literature review is the tally step that converts per-paper records into global counts, and that step has to be structurally separate from the narrative generation that uses them.

How the Verification Pass Checks AI Extraction

The verification pass runs after the deep-scan stage produces per-paper extraction records, and before any synthesis text is written. It does three concrete things.

First, it requires that every extraction record carry an explicit confidence marker and a source anchor. “Paper X used negative binomial regression” is acceptable only if the extraction record points at the specific passage in the source, and only if the extractor flags how confident it is about the identification. Methods explicitly stated in an abstract are high-confidence. Methods inferred from the results section are medium. Methods inferred only from the presence of a regression table are low. This sounds bureaucratic, and it is, but it’s the thing that makes the next step possible.

Second, it spot-checks numbers. Any claim of the form “N papers did X” or “M% of the corpus reported Y” is cross-checked against the per-paper records. This is mechanical: count the records, compare the count to the claim, flag any mismatch. In my VDT case, the verification pass produced a direct contradiction between the draft’s “80% misapplied” claim and the underlying records, because the records said 71% used appropriate methods and the arithmetic didn’t reconcile. There was no clever reasoning involved. Counting beat synthesis.

Third, it produces a structured flag for anything the extractor couldn’t verify from the source, along with a “what this would take to resolve” note. A claim that a paper reached a particular conclusion, when the extractor only saw the abstract, gets flagged as unresolved. A claim that a sample was of a specific demographic, when the paper’s methods section wasn’t clear, gets flagged. The flags are explicit inputs to the next stage, which either resolves them by reading the source more carefully or leaves them out of the final wiki.

The visual above is the shape of what changes. Single-pass extraction produces a synthesis-level claim with no anchor. Verification-gated extraction produces a stack of per-paper records with anchors and confidence markers, plus a separately-computed aggregate whose arithmetic you can check. The information is the same. The shape changes, and the shape is what makes the verification possible.

The Vault Quality Gate: Never Cite Stubs

A related safeguard added at the same time is the vault quality gate. The paper-writing workflow uses a shared source vault, a directory of per-paper summaries and notes, that grows over time as papers get ingested. Before the verification pass, any summary in the vault was fair game for the literature review to pull from. After the verification pass, summaries are classified into three buckets:

• FULL. The summary is complete, including methods, results, sample, and theoretical framing, based on a full read of the paper.
• THIN. The summary covers the main findings but lacks methodological detail, or was produced from an abstract rather than a full-text read.
• STUB. A placeholder with little more than the citation and a few sentences of context, often from a paper I intended to read but haven’t.

The rule is that stubs never get cited. A thin summary can be referenced only for a substantive finding it actually covers, since it was never written to carry methodological detail. Stubs can’t be cited at all, no matter how convenient.

Without this gate, the path of least resistance during synthesis is to pull citations from wherever they’re available, and the vault always has more stubs than FULL entries. A cited stub is worse than a missing citation. A missing citation is visible. A cited stub looks authoritative until someone follows the reference and discovers the underlying note is a paragraph long. Conceptually a trivial rule; in practice, the part of the verification pass that had the largest effect on the draft’s content.

What It Costs and What It Gains

Adding the verification pass roughly doubles the time of the deep-scan stage. A literature review that used to take 30 minutes now takes about an hour. The per-paper extraction step is unchanged. The new cost is the cross-check at the end: counting, comparing, flagging, deciding.

The gain is less dramatic than I expected and more important than I expected. It’s less dramatic because the output doesn’t look different at first glance. The wiki reads about the same. The theme pages synthesize the same literature. The writing voice is the same. The output is more important because the claims in the output are claims I’d be willing to defend in front of a reviewer.

The verified records still support the substantive point. The methodological critique is real: these studies produce outcomes where a large share of observations are zeros, and on data like that OLS is the wrong tool. The useful finding is that the field is split, with a clear methodological majority using appropriate count models and a clear minority still using Gaussian methods on data that visibly doesn’t meet their assumptions. The hurdle specification, which several of the methodological papers in the corpus recommend for bimodal “no events vs many events” patterns, is absent. That’s the gap a new paper can fill, and that’s the gap the draft would have missed.

I keep the verification pass in the skill because the alternative is writing methodological critiques built on aggregates I can’t defend. The cost of the extra half hour is a fraction of the cost of a reviewer pointing out that your stated figure doesn’t match what the cited papers actually report.

The Wider Pattern

Three things I’ve come to believe from this, in increasing order of strength.

First, AI content’s most dangerous weak spot gets little attention. Hallucinated citations, made-up authors, factually wrong individual statements all show up, but they’re usually caught by anyone doing even a cursory read-through. The subtler failure mode is plausible aggregates: summary-level statements that integrate across a corpus in ways that sound computed but were generated. They don’t trigger the same “this doesn’t feel right” alarm that a hallucinated reference does, because they read as the kind of synthesis a competent reviewer would produce. They’re wrong in the same way a confident guess is wrong, and they’re almost impossible to spot by reading alone.

Second, verification needs to be structurally separate from generation. The same model that wrote “80% misapplied” is not in a good position to evaluate whether 80% misapplied, because the generation step and the evaluation step share the same priors. The verification pass has to do something mechanical and external: count per-paper records, compare numbers, check anchors against sources. It has to be a different kind of operation from the one that produced the claim. “Let me double-check that” prompts almost never catch the aggregate error.

Third, the verification pass is what makes AI-produced content usable for work that has to survive a second reader. If the output of your AI workflow is going to a peer reviewer, a client, a regulator, or any audience that reads closely, verification is the difference between “publishable” and “reads well, won’t hold up”. It’s also the cheapest intervention available. A mechanical check between the extraction step and the synthesis step works with the model and context window you already have, as long as the check is non-optional.

I’ve been treating this as a narrow finding about literature reviews, but I suspect the principle transfers. If your workflow produces content that will be read by someone who can check it, the only question that matters is whether the content survives going back to its sources. Everything else is style.