🔍 Read the full analysis: Making With AI Is Cheap; Trusting The Results Takes Work on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI published 722 mathematical manuscripts this week after its model was given about 4,000 problems, while checking results still requires scarce human expertise. Data cited on software reviews and contract work points to the same tension: AI increases output, but does not remove the work of deciding whether it is correct, useful and safe to rely on.
OpenAI published 722 mathematical manuscripts this week, generated after its model was given about 4,000 problems, highlighting a widening gap between the low cost of producing AI work and the human effort needed to verify it. The release, alongside evidence from software and contract workflows, points to review capacity as a potential constraint on how much AI-generated work organisations can responsibly use.
The manuscripts cover 372 families of problems, according to the source account. Some results have been formally checked using Lean, a proof-assistant system; OpenAI cautioned that some results without formal verification could have issues. The account contrasts the volume with an earlier result from the same programme: a proposed counterexample to an old Erdős conjecture that was carefully checked by five leading mathematicians. Producing many candidate results, in other words, does not settle which ones are sound or significant.
Software data cited in the account suggests a similar strain on review. Faros AI reported that teams merged 98% more pull requests during high-AI-adoption periods, while review time rose 91%. LinearB, analysing 8.1 million pull requests across 4,800 organisations, reported that AI-generated changes waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A 2026 peer-reviewed study found that 61% of AI-agent pull requests received no human review before being merged or closed. Several cited firms sell code-review products, a commercial interest readers should weigh when interpreting their findings.
In contract work, OpenAI’s partnership with contract-software company Ironclad involved evaluating GPT-6 Astra on real contracting workflows. The source says the model met 55% of evaluation criteria on average across 11 tasks, describing this as an improvement on its predecessor. The remaining criteria still matter: a missed approval requirement or incorrect jurisdiction clause may make an otherwise polished draft unsuitable for use.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Review Capacity Limits AI Output
The figures describe more than a quality-control problem. If production accelerates while qualified review remains scarce, organisations may have to choose between slower approval, accepting more risk, or limiting how much machine-generated work they use. That affects delivery timelines in software and professional services, as well as the reliability of work that reaches clients, users or the public.
The pressure can also reshape the value of professional work. Senior engineers, lawyers, auditors and research reviewers may become more important as the people able to judge and take responsibility for output. But treating their time as a final checkpoint misses a longer-term concern: expertise is built through practice, and junior workers need opportunities to develop the judgment that review requires.
The source account argues for a potential “referee premium”: greater demand for people whose approval carries professional or institutional accountability. That is an interpretation, not a measured labour-market finding. The cited examples nevertheless show why organisations may need to plan for verification alongside generation, rather than assuming faster drafting automatically means faster usable work.
Products Worth Considering
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Amazon Kindle Paperwhite Signature Edition, 32GB (2024), Metallic Black
As an affiliate, we earn on qualifying purchases.
From Math Proofs to Contracts
The mathematical example distinguishes between formal validity and the broader judgment researchers apply. A proof assistant can check whether a proof follows from a stated theorem, but that check does not by itself establish that the theorem is the right question, that the result matters, or that the work is communicated well. OpenAI’s warning about some unformalized manuscripts makes the difference between generated work and checked work explicit.
Software tests have a comparable limit: they can show that code passes the tests written for it, but not that those tests cover the intended need. The source account also argues that reviewers often lack the clues they would get from a human colleague’s reasoning and working habits. Even when AI tools can assist with checking, someone still has to decide what should be checked and whether the outcome is adequate.
Responsibility is another part of the problem. Engineers, lawyers and researchers may be expected to stand behind decisions under professional or institutional rules. The source’s examples suggest that automated checking cannot alone settle who is accountable for a mistaken result or a decision to rely on it.
How Much Work Has Been Checked
The available figures do not establish how often AI-generated work contains errors across all fields or how much extra review it requires compared with human work. The cited software studies use different methods and measures, and some come from companies that sell code-review tools; their results should not be treated as a single, universal estimate. The source account does not provide enough methodological detail to independently compare every figure.
For the mathematics release, it is not clear from the material how many of the 722 manuscripts have undergone independent expert review, beyond the note that some have Lean checks and some do not. Formal verification also cannot answer every question about significance or whether a result addresses the intended problem. The cited contracting evaluation covers 11 tasks, but the account does not give task-by-task results or explain how the criteria were weighted.
It also remains uncertain whether organisations will add review staff, change workflows, or accept more unreviewed output as AI use grows. The “referee premium” is a forecast in the source’s analysis; no wage or hiring data is provided to show whether that shift is already occurring.
More Evidence on Review Practices
The immediate test is how the published manuscripts fare under further scrutiny: which results receive formal checks, which draw independent mathematical review, and whether any require correction. OpenAI’s caution about unformalized work makes follow-up verification especially relevant to interpreting the collection.
In software and contracting, the next useful evidence will include longer-term, independently assessed measures of review coverage, error rates and the time required to approve AI-generated work. Organisations will also need to clarify who is responsible when AI-assisted work is accepted. Until those results are available, the evidence supports a narrower conclusion: AI can increase production, but the cost and capacity of trust remain unresolved.
Key Questions
What did OpenAI publish this week?
OpenAI published 722 mathematical manuscripts spanning 372 problem families, after its model was given about 4,000 problems, according to the source account.
Have all of the manuscripts been verified?
No. The account says some results have been formally checked in Lean and quotes OpenAI warning that some unformalized results could have issues. It does not specify how many have received independent expert review.
What do the software figures suggest?
The cited studies report rising pull-request volume alongside longer or missing review in some settings. Their methods differ, and several sources sell code-review tools, so the figures are evidence of a possible capacity problem rather than a universal measure of AI code quality.
Why can’t AI simply verify other AI work?
Automated tools can check defined properties, such as whether a proof follows from a stated theorem or code passes specified tests. They cannot by themselves decide whether the theorem, tests or contract terms capture the real need, or who should take responsibility for approval.
What remains unknown?
The material does not establish how error rates compare across all AI and human work, how much review time is needed in different fields, or whether organisations are adding staff to meet demand. The mathematical manuscripts’ independent review status is also not fully detailed.
Source: ThorstenMeyerAI.com
