🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Results: 1 In 5 Tasks Passed, Near-Top Rank on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai says GLM-5.3-FlashX-Legal achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard lists it at 19.17%, seventh among 81 models, while reporting a 92.5% pass rate across individual criteria. The task score requires every criterion to pass, and the result does not establish that the model is safe to use without professional review.
Z.ai’s GLM-5.3-FlashX-Legal scored a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s archived leaderboard. The result means the model met every requirement on about one in five benchmark tasks; it does not mean it achieved only 19.2% accuracy across all individual criteria.
Z.ai describes the model as optimized for professional legal workflows, including legal research, contract review and drafting. Its developer documentation says it was built on GLM-5.3-Flash and underwent legal-domain mid-training and post-training, including reinforcement learning in what the company describes as tens of thousands of task environments modeled on legal work. Z.ai recommends using it as an agent connected to services providing laws, cases and company data.
The benchmark result appears in Z.ai’s documentation as a 19.2% all-criteria pass rate and an improvement of 12.5 percentage points over the base model. Vals AI’s archived Version 1 leaderboard, last updated October 18, records GLM Legal at 19.17%, seventh of 81, with a listed cost of $0.15 per test. The same table gives the base GLM 5.3 Flash a 6.67% task pass rate, making the difference 12.50 percentage points.
Vals also reports a 92.5% criteria pass rate for GLM Legal under that leaderboard version. Those figures measure different things: a task passes only when all its required criteria pass, while the criteria rate captures performance on individual rubric items. The benchmark uses two AI judges, GPT 5.5 and Claude Sonnet 4.6; each grades submissions, and the task score averages their pass rates.
Why Task Completion Matters
The gap between 92.5% of criteria passed and 19.17% of tasks completed highlights a practical issue for legal work: a high rate on individual checks can coexist with a much lower rate of fully acceptable work products. Under Harvey’s all-pass rule, one missed deadline, incorrect citation or other required item can cause an entire task to fail. That makes the score relevant to organizations evaluating whether an agent can complete a workflow, rather than only provide useful fragments.
The leaderboard also gives buyers a cost and performance point of comparison: Vals listed the model at $0.15 per test, below the listed costs of several higher-ranked models. But benchmark placement alone does not establish that a product is suitable for a particular firm, matter or jurisdiction. Legal professionals would still need to check its outputs and assess how client information is handled.
Products Worth Considering
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
DJI Mic 3 Wireless Microphone Kit (2 Transmitters, 1 Receiver, Charging Case)
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Work
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 practice areas, with more than 75,000 expert-written rubric criteria. Each task gives an agent an instruction and client-matter materials, then asks it to produce a work product for review. The criteria can cover facts, conclusions, citations, deadlines, recommendations, amounts and formatting.
Harvey’s all-pass rule requires every criterion for a task to pass before the task counts as complete. Vals AI’s public leaderboard uses a 120-task private holdout that mirrors the public benchmark mix; internet access is disabled during evaluation. Harvey has said that, in its own May 26 baseline, the best all-pass result among the frontier models it tested was 7.1%. That was a separate run, so it should not be treated as a direct comparison with the later Vals leaderboard.
Z.ai’s AutoClaw blog post announcing the model and an application-based beta was dated October 13, 2026; Vals lists the release date as October 12. Z.ai calls the result state of the art among open-source models, but the supplied announcement does not name the comparison set, and the source material says published weights for the legal model were not found.
“On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.”
— Z.ai, in its model documentation
Limits of the Published Result
The score documents performance on a defined benchmark and private holdout; it does not show how the model performs across every legal task, jurisdiction or real client matter. The source material does not provide enough detail to independently assess the legal model’s training data, its handling of confidential client information, or the safeguards available when it is connected to external legal and company-data services.
Z.ai’s description of the model as state of the art among open-source models is a company claim. The comparison set is not specified in the supplied material, and no published weights for this legal model were identified there. The source material also describes a four-task spot check in which the model cited rulings the reviewer could not find; that small check is not a benchmark and cannot establish an overall error rate. It does underscore why citations require verification.
What Firms Should Verify
Z.ai’s AutoClaw post announced an application-based beta, but the supplied material does not state when access will expand or whether the model, evaluation setup or leaderboard score will be updated. The next useful developments would include clearer documentation of the benchmark comparison set, the model’s data-handling terms and any independent testing beyond the reported leaderboard run.
For legal teams considering a trial, practical checks include confirming where client data is processed, which rules govern its use, how connected services handle information, and who reviews every citation and work product before it is used. The benchmark is evidence of a measurable result under one evaluation method; decisions about deployment require separate review of reliability, confidentiality and professional oversight.
Key Questions
What does the 19.2% score mean?
It is the share of benchmark tasks on which the model passed every required rubric criterion, averaged across the two judges used in the Vals evaluation. It is not a claim that the model passed only 19.2% of individual checks.
How can the task rate be 19.2% while the criteria rate is 92.5%?
The measures use different units. A task fails if even one required criterion fails, while the criteria rate counts individual rubric items. The model can pass many individual checks without completing many tasks in full.
Where did GLM-5.3-FlashX-Legal rank?
Vals AI’s archived Version 1 leaderboard lists GLM Legal at 19.17%, seventh of 81 models, at $0.15 per test. The leaderboard version and date matter when reporting a rank.
Does the result prove the model is ready to provide legal advice?
No. The benchmark measures performance on a specific set of tasks and does not establish suitability for independent legal advice. The source material describes the model’s outputs as drafts for legal professionals and notes that citations and work products need verification.
What is still unknown about the model?
The supplied information does not settle how client data is processed, what safeguards apply to connected services, or how the model performs across real-world matters and jurisdictions. It also does not specify the comparison set behind Z.ai’s open-source leadership claim.
Source: ThorstenMeyerAI.com
