GPT-5.6-Credited Counterexample Narrows a Benjamini-Hochberg Guarantee

• A July 13 preprint gives a certified counterexample to false-discovery-rate control for a particular sequence of correlated, two-sided Gaussian tests.
• The proof is asymptotic; among three retained finite simulations, only the largest has a 95% Monte Carlo interval entirely above the nominal level.
• The author credits GPT-5.6 Pro with obtaining the proof, but the 90-minute result and GPT-5.5 comparison are one reported episode, not a controlled evaluation.

A statistics conjecture believed for roughly two decades now has a counterexample backed by a runnable certificate. The consequential result is narrower than the surrounding AI headline: it removes a universal guarantee in one dependence setting, without showing that the Benjamini-Hochberg procedure generally fails in practice or that GPT-5.6 reliably solves comparable open problems.

The theorem breaks one guarantee, not the method

The Benjamini-Hochberg procedure addresses the false positives that accumulate when researchers test many hypotheses at once. It controls the false discovery rate, or the expected share of false rejections among all rejections, rather than trying to prevent even a single false rejection.

The historical comparison matters because dependence was not an untouched problem. An abstract for a 2001 analysis says the original procedure was proved for independent test statistics and was more powerful than comparable familywise-error controls. The same analysis proved control under positive regression dependence on the true-null statistics, including multivariate normal tests with a positive correlation matrix, and described a conservative modification for other dependence structures.

Edgar Dobriban's July 13 preprint addresses a different and more specific claim. It constructs a Gaussian factor-model sequence in which correlated, two-sided Gaussian p-values defeat ordinary Benjamini-Hochberg control. At a nominal level of 0.01, the paper says interval arithmetic certifies a false discovery rate above 0.0104 for all sufficiently large numbers of hypotheses.

That is enough to disprove the conjectured guarantee. It is not evidence that every correlation structure causes inflation, that one-sided tests have the same problem, or that studies using Benjamini-Hochberg should be retracted. The paper is also a preprint by the researcher who checked the result; the retained materials do not document peer review or an independent replication.

The certificate and the simulations answer different questions

Dobriban's reproducibility bundle explicitly separates a rigorous certificate from a finite-dimensional Monte Carlo experiment. The outward-rounded Arb computation proves liminf FDR > 0.010416829070473713... for the stated factor-model sequence. The theorem does not rely on the simulations.

The retained simulation results are more qualified:

The progression is consistent with the theorem, but only the largest run cleanly separates its estimate from 0.01 under the reported interval. The certificate establishes that a violation exists asymptotically; it does not establish how often comparable violations occur in applied datasets or how large inflation can become across other correlation structures.

The bundle makes the numerical claim unusually inspectable. It fixes Python and library versions, seeds, batching and a machine-readable run plan. Its verifier checks seven experiment chunks, 300 macro-replications, pooled outputs, file hashes and a fresh certificate run. Complete regeneration is computationally more substantial: each macro-replication contains 1,000 strata and generates and sorts full Gaussian vectors. The repository deliberately omits elapsed time because timing is not reproducible.

Those controls reduce dependence on a screenshot or an AI transcript, but they do not amount to independent validation by themselves. They provide the materials with which another researcher can attempt that validation.

The AI capability claim has a weaker comparison base

The preprint says GPT-5.6 Pro obtained the proof and Dobriban carefully checked it. Two accounts of Dobriban's public description add a more specific product name and chronology: they say he used GPT-5.6 Sol Pro for about 90 minutes, after several GPT-5.5 agents failed to find a valid answer over roughly 20 hours. One account also links to a shared chat, while a second describes the same contrast.

The attribution chain is important. The timing, exact Sol Pro label and predecessor comparison do not appear in the retained preprint abstract; they are reports of what Dobriban said. No controlled protocol in the retained evidence fixes the number of attempts, prompts, inference budgets or stopping rules across the two model generations. The episode shows that the newer system contributed to this successful workflow. It cannot isolate whether the outcome came from model capability, greater inference-time compute, interaction choices or variation in a difficult search.

Reproducing the discovery is also economically different from checking the certificate. OpenAI's release page says Sol Pro can be selected in Chat by Pro and Enterprise users. It separately lists ordinary Sol API pricing at $5 per million input tokens and $30 per million output tokens, but it does not price the reported Sol Pro session or present Sol Pro as one of the API's three model tiers. The public bundle therefore makes the mathematical output more accessible than the precise AI search process that produced it.

The practical choice is still about dependence and power

The counterexample removes a blanket assurance for correlated, two-sided Gaussian tests. It does not remove the guarantees already proved under specified positive dependence, nor does it eliminate the conservative modification available for other dependence structures.

That leaves researchers choosing between ordinary Benjamini-Hochberg where its assumptions are defensible and the conservative modification for broader dependency structures. The retained abstract establishes that the broader protection exists, but it does not quantify the detection-power cost for Dobriban's model. Measuring that cost alongside the possible inflation is part of the follow-up.

A retained account of Dobriban's assessment describes the gap in his construction as relatively small and mainly theoretical for now. The certificate should not be stretched beyond that statement: it proves a lower bound above the target, not a universal upper bound on possible inflation. The practical risk remains an empirical and theoretical question.

What evidence would change the conclusion

The next statistical decision requires a map of the failure boundary. Independent researchers need to check the analytical argument and rerun the certificate, then determine which correlation structures generate inflation, how large it can become, and how quickly the asymptotic behavior appears at realistic numbers of tests. Those results would show when ordinary Benjamini-Hochberg remains defensible and when a dependence-safe alternative is warranted.

The AI claim needs a separate test. A public record of one successful session can establish provenance, but repeated attempts under disclosed prompts, budgets and stopping rules—run on GPT-5.6, GPT-5.5 and competitive alternatives—would be needed to estimate reliability. Until then, the strongest supported conclusion is specific: a GPT-5.6-credited, human-checked process produced an inspectable counterexample to a narrow statistics conjecture. Neither broad practical failure nor a general model breakthrough follows from this case alone.