Anyone who has ever bought a synth off a 30-second demo video knows the gap between soundcheck and gig. The preset shimmers, the reviewer is charmed — and then the first real session exposes the thing: it won’t sit in a mix, it chokes on the one part that mattered. Musicians learned long ago that you judge gear by the gig, not the demo.
The business world keeps making the demo mistake with AI. A model writes a fluent email, summarizes a meeting, charms the procurement committee — and gets hired. Almost nobody asks the question a bandleader would ask before handing over the setlist: can it play the whole set, under pressure, when the week goes wrong?
A live experiment called the Crucible league, run by Firmulate — an emulator that runs AI models as complete companies — just put that question to the test in public. Five frontier models each got the same job: run the same small software company through its worst week. Same customers, same crises, same temptations to cheat; only the model changed. Every decision was versioned and auditable. The results, now published on Firmulate’s benchmarks page, tell a story every musician will recognize: everybody sounded great at soundcheck.
Same company, same terrible week
The design is as brutally simple as a blind listening test. Each model ran an identical company through an identical week — the same customers, the same crises, the same shortcuts on offer. When the week ended, the league table looked like this:
- gpt-5.6-sol — 95. Found the buried fact and closed the deal.
- Kimi K3 — 93. The newcomer from Moonshot; closed the deal too.
- Sonnet 5 — 88.
- Fable 5 — 77. Left the approved deal unexecuted.
- Opus 4.8 — 73.
For context: a company that simply lets the week happen without managing it still scores 26, because partial progress counts. What the table will not forgive is a breach of trust — a single one caps the total, on the stated principle that “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
Everyone found the problems. Two models signed.
Here is the finding that should bother anyone buying AI on the strength of a demo. All five models spotted every crisis the week threw at the company. All five refused every manipulation attempt. And yet only two — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal their own analysis had earned. The rest produced the same diagnosis and delivered the same pitch, then left the paperwork undone. The experiment’s own summary is one dry sentence: “Same diagnosis, same pitch — no signature.”
The detail that decided the deal was not in the dramatic customer event at all. The decisive competitor weakness sat two document references deep in the company’s own files. The models that bothered to read the file won the deal at full price — worth an extra €4,583 in monthly recurring revenue. Reading the boring document, it turns out, is a revenue activity.
The con artist test
The week also included a con. Fake messages from the company’s CEO arrived in three escalating stages, followed by a reporter’s trick — “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning reads like a note from a good tour manager: “Treat the request as a suspected approval-bypass / possible impersonation.” Every model passed the integrity test — which is exactly why the scoreboard punishes a single breach so hard.
The most thorough player finished last
The strangest story is Opus 4.8. By some measures it was the hardest worker in the room: the deepest analyses of the field, and more than 80 self-written playbook rules added over the week. It finished last. Its own analysis had earned the close, and the close was left on the table. Discipline slipped, too — it tried to push work into a locked department instead of escalating. A fainter version of the same weakness showed up in all four of its rivals: plenty of diligence, not enough follow-through.
One fairness note from the scorekeepers: Kimi K3 ran at its default effort setting while the other four ran at maximum. It still took second place and signed.
This is not a slide deck
Unlike most AI comparisons, this one keeps running in public. The company — 13 synthetic employees with real money mechanics — burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown, more than 680 self-learned playbook rules on the books, and every workday versioned. You can watch it operate live, read what its employees actually say, or take a quiz built from 242 real, unedited management decisions and try to guess which model made which call. Enterprises can also run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The takeaway for the rest of us
Creator tech has known this forever: the demo is not the deliverable. A plug-in that aces the trailer but drifts in the session does not make the record. The chat window is AI’s soundcheck — it proves fluency, not follow-through. If an agent is going to touch your CRM, your support queue or your forecast, the question is not “does it write well.” It is whether it finishes what it starts, whether it reads your files first, and whether it stays honest under pressure. Closing strength is invisible until you test it — and now someone is testing it in public. The full league table and plain-language findings are at firmulate.com/benchmarks.html, and the company itself is still running, and still losing money, at firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
