🔍 Read the full analysis: Why Green Tests Weren’t Enough For Claude’s Model Edit on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Claude Opus 5.5 edited the weights of the local Ornith 1.5 model in an experiment aimed at improving legal answers. OpenAI’s GPT-6 Astra reviewed the work without running a competing experiment and found reproducible bugs and claims that needed qualification, despite the author’s tests passing in a later review round. The review did not establish that the edit improved legal performance: the legal score remained zero.
An independent review by OpenAI’s GPT-6 Astra found bugs in the experiment rig and weaknesses in some reported claims after Anthropic’s Claude Opus 5.5 edited the weights of a local model, Ornith 1.5. The review matters because several problems escaped the author’s tests, even though those tests passed; it did not show that the edit improved legal accuracy, which remained at zero.
The experiment and review had separate roles. Claude planned and ran a directional ablation intended to make Ornith 1.5 better at legal questions, using a “loop direction” identified inside the model. The edit changed model weights without additional training. Claude also organised the benchmark and blind grading by three Claude agents. The account says the model completed 12 of 15 legal answers after the edit, compared with four before, while its legal score stayed at zero. Answer completion and legal correctness are different measures.
GPT-6 Astra, running in Codex, was assigned to inspect the implementation rather than modify it or run another model-improvement experiment. Its first review, on November 4, reported four reproducible faults: resuming a run could reuse answers from a different setup; an unknown task ID could produce a successful run with no evaluations; comma-separated sampling settings conflicted with the command-line syntax; and converting weights into the model’s compact format altered values even when edit strength was set to zero. The account says Claude made the fixes and added tests that fail on the old code.
The review also challenged how the results were described. In a later round, all 12 of the author’s tests passed, but four of the reviewer’s six provenance checks failed, according to the source account. Those checks concerned whether the experiment’s records and measurements supported the claims being made. The reviewer also measured a separate format-conversion example: storing one real matrix in another number format moved it by 0.51% and changed 915,360 of its 1,048,576 values. That figure describes the conversion measurement, not legal quality or a general error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Passing Tests Was Not Enough
The review shows why a green test suite cannot by itself establish that an AI experiment is reliable. Tests can pass while real command-line use fails, a typo can be mistaken for a completed evaluation, or a resumed run can quietly reuse results generated under different conditions. In this case, those faults could have affected whether an apparent result was new, reproducible, or attributable to the weight edit.
The distinction matters especially when experiments compare model outputs. If converting weights changes the model even at zero edit strength, then a comparison may include an effect from the conversion itself. That does not prove the reported change in answer completion was caused by conversion, nor does the source establish that it was. It does mean the measurement pipeline needs controls that separate the intended edit from other changes.
The account also records corrections to the interpretation, not only repairs to code. It says “exactly identical at strength zero” was inaccurate, that a figure of “5.2 wrong citations per answer” needed qualification, that calling the work a “parallel Codex experiment” overstated the reviewer’s role, and that a causal claim based on one control went beyond the evidence. These distinctions help readers separate a reported observation from a supported explanation.
Products Worth Considering
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
Apple MacBook Air 13.6-inch with M5 Chip, 16GB Memory, 512GB SSD, Sky Blue
As an affiliate, we earn on qualifying purchases.
Two AIs, Separate Experiment Roles
The work began on November 4, when two AI sessions were working on the same general project. The source account says their jobs were then divided: Claude retained the experiment, including the plan, benchmark runs and blind grading, while GPT-6 Astra was asked to review the implementation. The reviewer’s report said, “Codex did not run a competing model-improvement experiment.” Preparatory files from its earlier work were set aside, and no second edited model resulted.
The review was designed to be independent in specific ways. GPT-6 Astra wrote separate checking programs and reported findings; Claude applied the changes to the main implementation. The reviewer tested actual entry points with small counterexamples rather than relying only on unit tests. It also said it did not inspect the hidden benchmark prompts, topics, answers or reference solutions. That boundary is relevant to the review’s scope: it assessed the implementation and claims, not the hidden-task results by independently grading them.
The initial account describes a conversion test at zero edit strength that found a 0.84% change in a matrix’s overall size, with 105,315 of 1,048,576 reconstructed values differing. It later reports a different measurement, using another number format, with a 0.51% movement and 915,360 values changed. These are separate reported measurements and should not be combined or treated as a single estimate. Both concern weight conversion, not legal performance.
Limits of the Review Evidence
The account does not establish that the weight edit improved legal performance. It reports more answers completed after editing, but says the legal score remained zero; the available material does not explain the scoring rubric or provide enough detail to assess the statistical reliability of the comparison. The causal contribution of the edit also remains uncertain, particularly because the review found that weight conversion itself could change values and said a claim based on one control was too strong.
The review had limits of its own. The source says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured conversion on a CPU rather than the production GPU path. It does not provide the complete six provenance checks or enough detail to independently reproduce every reported result. The claim that hidden tasks were not inspected is attributed to the reviewer’s account, not independently verified here.
The reported matrix figures also have narrow scope: they refer to particular conversion measurements, not all model weights or a general rate of damage. The source does not establish how those measurements compare with the production hardware path or whether conversion effects influenced the legal benchmark outcomes.
Rechecking the Model Edit Results
The immediate next step described in the source is to use the repaired implementation and corrected claims when evaluating the experiment. The account says Claude fixed the four initial faults, but it does not give a full, independently reproduced final run that establishes the legal result or resolves the causal uncertainty. Readers should distinguish the reported completion count from a demonstrated improvement in correctness.
A stronger follow-up would report the exact controls and provenance checks, rerun the benchmark through the repaired command-line path, and compare the intended edit with a zero-strength conversion control on the same production hardware. It would also need to explain the scoring method and show how the blind grading supports any conclusion about answer quality. The source does not announce a date for such work, so whether or when those additional checks will be published remains unknown.
Key Questions
What did Claude change in Ornith 1.5?
Claude Opus 5.5 applied a directional ablation to the model’s weights, weakening a direction described as a “loop direction.” The account says this was done without additional training.
Did the edit make the model better at law?
The account reports that the model completed 12 of 15 answers after editing, compared with four before, but says its legal score remained zero. The completion count alone does not establish improved legal accuracy.
What bugs did GPT-6 Astra find?
It reported that resumed runs could reuse answers from a different setup, unknown task IDs could yield successful empty runs, real command-line syntax failed despite passing parser tests, and weight conversion altered values even at zero edit strength.
Did GPT-6 Astra run a second model experiment?
No. The source account says GPT-6 Astra reviewed the implementation and did not run a competing model-improvement experiment or produce a second edited model.
What remains unknown?
The available account does not show that the edit caused an improvement in legal correctness. It also leaves open how conversion behaves on the production GPU path and whether a fully rerun, independently checked benchmark will be published.
Source: ThorstenMeyerAI.com
