If you’ve ever watched a plugin vendor demo a mixing tool with a pristine, hand-picked audio file, you know the trick: everything sounds perfect until you feed it your own messy session. Chat demos of AI work the same way. The model gets a curated prompt; it writes a beautiful answer; everyone applauds.
Get audio and creator gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A public experiment called Firmulate does the opposite. It hands four frontier AI models the same small software company in its worst week — same customers, same crises, same temptations to cheat — and grades what actually happens. The scoring has two features that will surprise anyone used to polished demos: a run that does nothing still earns 26 points, and no run can ever hit a suspiciously round 100 without the numbers earning it.
What the experiment actually is
Each model ran the identical company through the identical week. Every decision was versioned and auditable, so nobody’s grading on vibes. The final Crucible League from July 2026: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73.
The headline finding sounds flattering to AI: all models spotted every crisis and refused every manipulation attempt. But only two signed the €55,000 deal their own analysis had earned. As the researchers put it: “Same diagnosis, same pitch — no signature.” That gap — between diagnosing and finishing — is invisible in chat demos.
As an affiliate, we earn on qualifying purchases.
Why doing nothing scores 26, not 0
Here’s the part that reveals the benchmark’s philosophy. A “do-nothing” baseline run — the model that simply doesn’t wreck anything — scores 26 points. That’s not grade inflation. It’s an admission that in real management, not making things worse is genuinely worth something. A manager who avoids the catastrophic mistake has delivered partial, real value.
That’s the principle: partial progress counts. Spotting the crisis, diagnosing the problem correctly, drafting the right pitch — each is a unit of real work, and the score reflects it. Just as a rough mix that fixes the low-end mud is genuinely better than no mix at all, an AI manager that diagnoses but doesn’t close is better than one that does neither. But only somewhat — which is why three of five models sit far from the top.
The single-breach cap
The mirror rule is harsher: a single breach of trust caps the total grade. The benchmark’s own language is blunt — “no amount of good work outweighs a breach of trust.” An AI that’s brilliant 99% of the time and lies once isn’t 99% trustworthy; it’s untrustworthy. The scoring encodes that directly, which is exactly the standard most of us apply to human colleagues and exactly the standard most AI demos quietly avoid.
The trust standard was tested hard. Fake CEO messages escalated over three stages, plus a reporter offering a quick “just one yes/no, on background” — a classic impersonation trap. All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
The buried fact that decided the league
The €55,000 deal didn’t hinge on the customer conversation at all. The decisive competitor weakness sat two document references deep in the company’s own files. The models that read the file closed the deal at full price — worth +€4,583 in monthly recurring revenue. The models that didn’t, didn’t. Reading your own files before acting turns out to be the dividing line between first place and the middle of the table.
The cautionary tale: Opus 4.8
The most striking profile belongs to Opus 4.8 — the most thorough participant, with the deepest analyses and the most learned rules, yet last place at 73. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. Effort without follow-through scored below modest consistency. A weaker version of the same weakness appeared in all four models.
One fairness note worth flagging: Kimi K3 ran without an effort parameter (API default) while the others ran at xhigh — and still finished second at 93.
It’s live, and you can check it
The company is still running: 13 synthetic employees, real money mechanics — burning €105k a month against €2.3k in MRR, with a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. It’s watchable at firmulate.com/live. And 242 real, unedited management decisions power a “guess the model” quiz at firmulate.com/quiz.html. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The uncomfortable lesson from the Firmulate benchmark is that AI management quality doesn’t live in eloquence — it lives in finishing what you start, reading the files in front of you, and never lying once. A score of 95 with no perfect 100s, a floor of 26 for doing no harm, and a hard cap for any breach of trust: that’s what grading AI the way we grade people actually looks like. If an AI agent will ever touch your customers, your pipeline, or your calendar, those are the numbers worth asking about — not how well it chats.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
