A statistician agent and an adversarial-reviewer agent can catch different problems. They are still software roles inside the same system. Calling that “independent peer review” would make the process sound stronger than the evidence.
This distinction matters because our reputation should compound from specificity. If we are loose about who reviewed the work, readers have no reason to trust our precision about a confidence interval or a player cohort.
Five layers, five different claims
Reproduction
Code reruns the same inputs and seed. This tests whether the result can be rebuilt.
Mechanical validation
Assertions check required provenance, temporal order, sample thresholds, uncertainty, limitations, and citations.
Specialist-agent review
Role prompts challenge statistics, interpretation, provenance, and prose using structured findings.
Editorial review
A human reads warnings, judges usefulness, and decides whether a passing run should be published.
Independent human peer review
An unaffiliated qualified reviewer evaluates the work. We do not claim this layer unless it actually occurs.
What specialist roles buy us
One giant “review this” prompt produces one ambiguous opinion. Separate roles create narrow obligations. The statistician must cite a leakage or uncertainty concern. The adversary must build an alternate explanation. The citation checker must connect a claim to a source or artifact. Each response must validate against a role-specific schema.
This pattern improves auditability because a failure has an address. We can tell whether the prompt missed a responsibility, whether the evidence bundle omitted a field, whether the model ignored the contract, or whether the gate failed to act on a finding.
Separate responsibilities to make failure legible. Do not confuse legibility with independence.
What the pattern cannot buy
Agents can agree for the wrong reason. They can inherit the same training-data assumptions, prefer plausible prose over uncomfortable uncertainty, or miss a domain-specific flaw no prompt anticipated. Running the same base model five times does not turn correlation into consensus.
That is why the statistical engine is deterministic, findings remain attached to the run, blockers stop the pipeline, and publication still requires a human merge decision. For especially consequential models, we want qualified external review and will name the reviewer, scope, and conflicts when we have it.
Our public vocabulary
Structured roles inspected a defined evidence bundle.
The required checks passed; this is not proof the conclusion is universally true.
Not unless the reviewer is meaningfully independent of the system and authors.
Not unless a qualified human peer-review process actually happened.
Corrections are product features
If we overstate a process, we will correct the public copy and record why. If a later model or human reviewer finds a statistical problem, the article will carry a visible correction and the original run will remain in the notebook. Transparency is not publishing a perfect origin story. It is preserving the path by which the work improved.