🔍 Read the full analysis: GPT-6 Astra Checked Claude’s Model Edit After The Tests Passed on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Anthropic’s Claude Opus 5.5 led an experiment editing the weights of the local model Ornith 1.5, while OpenAI’s GPT-6 Astra independently reviewed the test rig rather than running a competing edit. The review found reproducible problems in the benchmark workflow and weight conversion; after fixes, all 12 author tests passed, but four of six reviewer provenance checks still failed in the second review round.
OpenAI’s GPT-6 Astra, running in Codex, reviewed an experiment in which Anthropic’s Claude Opus 5.5 edited the weights of the local model Ornith 1.5, and found reproducible faults in the evaluation process that the project’s tests had not caught. The reviewer did not conduct a rival model-improvement experiment; its findings put limits on how confidently readers can interpret the reported benchmark result.
The experiment’s reported outcome was that the edited Ornith 1.5 answered all 10 published Lexassay tasks, compared with four before editing, while its legal score remained at zero. The source says the figures were updated on November 9 to refer to 10 published tasks, with runs made public on Lexassay. Those results concern this benchmark and do not establish broader legal capability.
Claude’s experiment used a directional ablation inspired by a MIT CSAIL post on weight-space programming. It identified what the authors called a “loop direction” and weakened it in the model weights without training. Claude also organised blind grading by three Claude agents. GPT-6 Astra was assigned a separate role: review the implementation, reproduce problems, take measurements and scrutinise the claims, without changing the main test rig or inspecting the hidden benchmark tasks.
The reviewer found four issues in its first round. A resumed run could reuse an answer from a different model or sampling setup; an unknown task ID could finish as a successful run with no evaluations; commas in the sampling syntax conflicted with commas used to separate models in the command line; and converting weights altered them even when the edit strength was zero. The project author says the Claude session fixed these issues, including rejecting unknown IDs and storing edited tensors in higher-precision bf16 rather than re-quantising them.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Review Changed the Result’s Meaning
The review matters because evaluation machinery can shape a benchmark result. If a resumed run silently keeps answers from a different configuration, or a mistyped task produces an empty run that appears successful, readers cannot rely on a green test suite alone to show that the reported measurements correspond to the intended experiment. The command-line conflict also showed that passing parser unit tests did not prove the real entry point worked.
The weight-conversion finding bears directly on attribution. In the first review, a conversion at edit strength zero changed 0.84% of one matrix, with 105,315 of 1,048,576 reconstructed values differing. That raises the possibility that conversion, not only the intended edit, could affect model behavior. A later measurement reported a 0.51% change and 915,360 differing values after storing one real matrix in a different number format. These are measurements of conversion on a matrix, not legal-quality scores or error rates; the source describes measurements from different points in the review.
In the second round, all 12 of the author’s tests passed, but four of the reviewer’s six provenance checks failed. The figures show that passing the project’s tests did not settle every concern the reviewer raised. The article does not report that the benchmark result itself was invalidated, but its caveats make a careful reading of the result necessary.
Products Worth Considering
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Raspberry
Bose TV Speaker (2nd Gen) Compact Soundbar
Sony WH-1000XM5 Premium Noise Cancelling Wireless Headphones – Midnight Blue
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The work began on November 4, 2026, when two AI sessions started on the experiment. According to the source article, the sessions were assigned separate responsibilities after it became clear they were working on the same project. Claude retained control of the plan, implementation, model edit, benchmark runs and blind grading. GPT-6 Astra’s stated assignment was to “Review and validate the other session’s implementation.”
The review was designed to keep the roles distinct: GPT-6 Astra wrote separate check programs rather than modifying the main implementation, and the Claude session made the fixes. The reviewer tested real entry points with reproducible examples instead of relying only on unit tests. Its second review said it had not inspected hidden benchmark prompts, topics, answers or reference solutions. The source says preliminary files from GPT-6 Astra were set aside after the role decision, and no second edited model resulted from them.
The published material and runs are available through Lexassay, including the weight-edited model, an edit control and model-card sampling. The source article describes the work as an AI experiment, not legal advice. Its reported benchmark change should be understood as a result on the specified published tasks, not as a general finding about legal performance.
Limits of the Review Evidence
The review did not settle every question about the model edit. The source says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured weight conversion on a CPU rather than the production GPU path. The source does not establish whether those differences would change the reported benchmark outcome.
The article reports that four of six provenance checks failed in the second review round, but the supplied material does not identify all six checks or explain each failure. It also does not provide enough detail to independently assess the full 10-task benchmark, the legal score’s scoring method, or whether the result generalises beyond those tasks. The conversion measurements refer to a matrix and different reported stages; they should not be treated as a single universal conversion rate.
The causal claim remains limited as well. The source says the reviewer judged the claim drawn from one control to be stronger than the evidence supported. It does not provide enough detail here to resolve what portion of the change in answers came from the directional edit rather than other elements of the pipeline.
Public Runs and Further Checks
The source points readers to the public Lexassay runs for the weight-edited model, edit control and model-card sampling. The next step for readers assessing the result is to compare those materials with the stated task set and the described review limits, rather than treating the test-suite status as a complete validation.
No further test round, independent replication, or new benchmark result is announced in the supplied material. It remains unknown whether the authors or another reviewer will address the four failed provenance checks, test the conversion on the production GPU path, or publish additional evidence about the relationship between the edit and the 10-task outcome.
Key Questions
What did GPT-6 Astra do?
GPT-6 Astra reviewed the implementation and evaluation workflow used in Claude Opus 5.5’s Ornith 1.5 experiment. It wrote separate checks and reported problems; it did not run a competing model-editing experiment.
What did the experiment report on the published tasks?
The source article says the edited model answered 10 of 10 published Lexassay tasks, compared with four before the edit, while its legal score stayed at zero. The result applies to those benchmark tasks and does not establish general legal ability.
What faults did the review find?
The first review found that resuming a run could reuse results from another setup, an unknown task ID could yield a successful run with no evaluations, the real command line failed despite green parser tests, and weight conversion changed a matrix even at edit strength zero.
Were all review concerns resolved?
The source says the author’s 12 tests passed in the second round, while four of six reviewer provenance checks failed. It does not identify all those failures in the supplied material, and it says the reviewer did not rerun every test during final documentation.
Where can readers inspect the experiment?
The source says the runs are public on Lexassay, including the weight-edited model, an edit control and model-card sampling. It describes the work as an AI experiment, not legal advice.
Source: ThorstenMeyerAI.com
