🔍 Read the full analysis: How To Edit AI Weights At Home With Zero Legal Score on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment removed a detected looping direction from the weights of the local model Ornith 1.5 35B-A3B. It answered all 10 published Lexassay tasks after the edit, up from 4 on the same comparison setup, but scored zero on the legal benchmark and produced 55 wrong or invented citations. The result suggests the edit helped the model finish responses, not supply reliable legal knowledge.
A home experiment reported by ThorstenMeyerAI.com found that editing the weights of a local AI model helped it finish more responses, but did not improve its legal benchmark score: Ornith 1.5 35B-A3B answered 10 of 10 published tasks after the edit, while its score remained zero. The result highlights the difference between reducing a model’s looping and making its answers accurate enough to use.
The experiment targeted repeated, circular output in Ornith, a model its publisher developed for coding and agent work rather than legal questions. The researchers measured an internal direction associated with the loops and removed it at half strength from the output weights of 27 of the model’s 40 layers. They did not train the model, add legal data or have an AI inspect its weights as text.
On the same setup used for the edited comparison, the model completed 4 of 10 tasks before the change and all 10 afterward. But the edited answers met just 7 of 50 checked criteria, compared with 3 before, and included 55 incorrect or invented citations. The benchmark checks cited statutes and court decisions against primary sources; the reported legal score was zero both before and after the edit.
The authors also tested the unchanged model with sampling settings recommended in its model card. It then answered 9 of 10 tasks without any weight edit. That comparison cautions against attributing all improved completion to the edit. The source says the runs, answers, grading and citation checks are publicly available on Lexassay; its updated figures cover the 10 published tasks.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Not Better Law
The experiment matters because it separates a model’s ability to produce a complete response from its ability to produce a correct one. A system that stops repeating itself may appear more useful, but the reported legal results show that completion is not accuracy: the edited model still failed the benchmark’s legal scoring and generated numerous bad citations.
For people testing local AI systems, that distinction affects how improvements should be evaluated. Counting finished outputs alone can make a change look successful while missing errors in substance. The report’s figures point to the need to check answer quality and source citations alongside whether a model reaches the end of a task. The results are an experiment, not a validation of the model for legal work or a substitute for legal advice.
Products Worth Considering
EcoFlow Delta 3 Max Portable Power Station with 220W Solar Panel, 2048Wh
LG UltraGear 25G590B-B 25-Inch Full HD 1000Hz Gaming Monitor
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Raspberry
As an affiliate, we earn on qualifying purchases.
The MIT Weight-Editing Comparison
The experiment followed a report from four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they call “weight-space programming.” Their work described an AI reasoning about another model’s weights directly. In a Qwen example, the MIT team reported that an edit addressed looping and the model’s mathematics score rose from 7.9% to 48.1%.
The home test was related in aim but used a different method. A Claude Opus 5.5 session wrote code to run Ornith, measure the internal signals associated with looping and remove one direction from selected weights. The source calls this directional ablation, a technique that does not involve training. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers and claims and identified faults. It was an independent review of the same experiment, not a second experiment.
Lexassay presents models with questions on German, US and EU law without internet access or tools. Ten tasks are published, while five others are withheld, according to the source. The reported numbers in this article refer to the published set. The authors selected Ornith because they observed looping in its responses; they say other tested local models failed for different reasons, including insufficient knowledge.
“The edit repairs finishing, not legal knowledge.”
— ThorstenMeyerAI.com experiment report
Limits of the Ten-Task Test
The results are limited: the source describes one run per setting, a small development set and a loop detector that changed during the experiment. Three graders came from the same model family, and Claude agents both carried out the edit and performed blind grading. The GPT-6 Astra review adds a check from another provider, but it does not remove the limits of a small, single-experiment evaluation.
The source also warns that two local setups were not directly comparable. The unchanged model completed 8 of 10 tasks in a published run, but completed only 4 of 10 on the setup used for the edited comparison. The figures should therefore be read with their specific setups in mind, not as a clean estimate of a general improvement. It remains unclear whether the effect would persist across repeated runs, different tasks or other models; the experiment does not establish that the edit improves legal reasoning.
The source says the answers, grades and citation checks for the published tasks are available on Lexassay, along with a method page describing the experiment. Those records allow readers to inspect what the model produced and how it was evaluated. The five withheld benchmark tasks are intended to remain unavailable for tuning, according to the report.
Further testing would need repeated runs under matched settings and evaluation on tasks not used during development to determine whether the completion effect is reliable. Until such evidence is reported, the findings support a narrow conclusion: the tested edit coincided with more completed responses on the comparison setup, while the legal score stayed at zero. The source does not report a follow-up result or a timetable for additional experiments.
Key Questions
What did the weight edit change?
It removed an internal direction associated with looping at half strength from the output weights of 27 of Ornith’s 40 layers. The method used measured model behavior and code; it did not train the model or add legal data.
Did the edit improve the model’s legal score?
No. The report says the score remained zero. The edited answers met 7 of 50 checked criteria and contained 55 wrong or invented citations.
Did the edit make the model finish more tasks?
On the same comparison setup, the model completed 10 of 10 tasks after editing, compared with 4 of 10 before. The source also reports that recommended sampling settings let the unchanged model complete 9 of 10 on its original setup, so the different setups complicate comparisons.
Was this the same method used by the MIT researchers?
No. The source describes the home experiment as a related but simpler technique called directional ablation. The MIT report described an AI reasoning directly over another model’s weights; the home experiment used code to measure and remove a direction associated with looping.
Can the edited model be relied on for legal advice?
The reported results do not support that conclusion. Its legal score stayed at zero, and the source explicitly describes the article as an AI experiment, not legal advice.
Source: ThorstenMeyerAI.com
