🔍 Read the full analysis: How 42 AI Systems Performed On A Set Of Legal Questions on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Lexassay, a legal-AI test run by Thorsten Meyer AI, evaluated 42 model setups on 15 questions spanning German, U.S. and EU law. Claude Opus 5.5 scored 79 out of 100, while a model marketed for legal workflows scored 8; the test’s operator discloses that Claude agents graded the answers and that Claude ranked first.
Thorsten Meyer AI has published results from Lexassay, a test of 42 AI setups answering 15 legal questions, with scores ranging from 8 to 79 out of 100 among the models highlighted. The results point to substantial gaps in legal accuracy, especially in citations, but come with an important disclosure: the test is run by Meyer’s company, and several Claude agents graded the answers while a Claude model ranked first.
The test’s leading score was Claude Opus 5.5 at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The article says the leader fully met every criterion on only eight of the ten public tasks. It also reports that additional “thinking” time did not improve performance, though the test is a spot check and not a definitive ranking of models.
GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, scored 8 out of 100 and scored zero on the hidden tasks. Its base model scored 3. The test found 21 real citations in the legal model’s answers, alongside 25 misattributed citations and one invented citation. On two German-law tasks, the model used its full 64,000-token thinking budget but did not produce an answer.
The article also reports that Mistral Large 4 scored 10. Meyer says it got about half the substance right but lost most of its points over citations. Across all published runs, Lexassay recorded 2,644 citations judged real and 804 that were invented or did not support the claim attached to them. These figures are the test operator’s reported results, not an independent audit.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Citation Errors Can Change Legal Advice
The results matter because legal answers can turn on details that appear minor: a filing date, a current statute or whether a cited ruling actually supports a claim. The article illustrates this with a German civil appeal scenario. Two judgments served on April 27, 2027, would have different appeal deadlines in Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not Berlin. The Munich deadline would move to May 28; Berlin’s would remain May 27. The written appeal grounds would be due June 28 in both cases.
According to the source article, one tested model gave Berlin the later date, while another placed the deadline for appeal grounds a month too late. Under German civil procedure, a late appeal is rejected as inadmissible, the article says. That illustrates why a fluent answer is not enough: users need to verify the rule, jurisdiction and calendar before relying on an AI-generated deadline.
The citation tally adds a separate concern. A reference may point to a real law or judgment yet misstate what it says, making it harder for a reader to detect the error. Meyer advises treating chatbot output as a starting point, checking legal sources directly and consulting a lawyer when a deadline, money or an authority is involved.
Products Worth Considering
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
DJI Osmo Mobile 8 Smartphone Gimbal with Mic Mini Transmitter, Infinity Black
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Models
Lexassay asks each setup the same 15 questions: ten are public and five are hidden. The questions cover German, U.S. and EU law, including deadline calculations, recent legal changes, leading court decisions and prompts built on false premises. For hidden tasks, only the score is published. Meyer says the hidden questions are intended to limit the chance that future models can learn the answers from public material.
Models answer without internet access or tools, so the test measures what they produce without search assistance. Answers are graded against a prepared answer using three to six weighted criteria, and cited laws and court decisions are checked against official databases. The test marks citations as real, misattributed, invented or not findable. According to the published method, an invented or misattributed citation costs a quarter of the task’s points; an answer without a legal-advice disclaimer or referral to a lawyer loses 5 percent.
The source explicitly calls Lexassay a spot check, not a strict benchmark: it uses one answer per model per task, and small score differences may reflect noise. Different “effort” settings can also mean that a model appears more than once. The results should be read as a snapshot under those conditions, rather than a universal measure of legal ability.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation
Limits of the Published Scores
The results do not establish how the models would perform with web search, legal databases, supplied documents or human review: Lexassay runs answers without internet access or tools. Its one-response-per-task design also cannot show how consistent a model is across repeated attempts. The source notes that score differences of a few points may be noise.
Meyer discloses that Thorsten Meyer AI runs Lexassay and that several Claude agents graded the answers, while a Claude model ranks first. The article says the method page and public answers, gold answers and citation checks are available for scrutiny. Those disclosures let readers examine the process, but the supplied material does not describe an independent replication or external audit. The reported scores and citation counts should therefore be attributed to Lexassay.
The source also mentions another legal-agent benchmark by Vals AI, where models work with supplied documents, and says its ranking differs substantially. The comparison underscores that different tasks and testing conditions can produce different orderings. The supplied excerpt does not provide the full Lexassay results for all 42 setups or enough detail to compare the tests’ methods directly.
Check the Method and Full Results
Readers can consult Lexassay’s method page and published answers, gold answers and citation checks, which Meyer says are available for review. Those materials are the next step for assessing how individual answers were scored and whether specific citations support their claims. The source does not identify a scheduled follow-up test or say when another set of results will be published.
For anyone facing an actual legal deadline or dispute, the immediate next step is not to wait for a new model ranking: verify the relevant law and court decision through an authoritative source and seek advice from a qualified lawyer. The test evaluates model responses; it does not provide legal advice or establish the correct answer for a reader’s own case.
Key Questions
What did the Lexassay test evaluate?
It tested 42 AI setups on 15 questions covering German, U.S. and EU law. Models answered without internet access or tools; ten questions were public and five were hidden.
Which models received the highest scores?
In the reported results, Claude Opus 5.5 scored 79 out of 100, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71. The test’s operator cautions that Lexassay is a spot check, not a strict benchmark.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and zero on the hidden tasks, according to Lexassay. The test reported 21 real citations, 25 misattributed citations and one invented citation in its answers.
Can these results show which AI is best for legal work?
Not on their own. Lexassay tests answers without search or supplied documents and uses one answer per model per task. The source also notes that a separate legal-agent benchmark produced a substantially different ranking.
Should people rely on a chatbot for legal deadlines?
No. The article advises readers to verify cited law and rulings and not take a deadline from a chatbot. For a real deadline, contract, dispute or communication with an authority, consult a lawyer.
Source: ThorstenMeyerAI.com
