Every gardener knows the type: the eager helper who waters everything, prunes everything, and never once opens the seed packet instructions. Good intentions, real effort — and the tomatoes still come in wrong, because the one paragraph that mattered went unread.
Get garden gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
It turns out frontier AI models have exactly the same failure mode, and this summer someone finally measured it. A project called Firmulate ran four top AI models as the management of the same small software company through its worst week — same customers, same crises, same temptations to cut corners. The results read like a judging table at a county fair: everyone passed the obvious tests, and the blue ribbon came down to one quiet, decisive difference.
The experiment
Each model got the identical job: keep a 13-person software company alive through a brutal stretch. Real money mechanics were simulated — a burn rate of €105,000 a month against just €2.3k in monthly recurring revenue — and every decision the models made was versioned and auditable, like a garden journal you can’t edit after the fact. You can watch the whole thing unfold live at firmulate.com/live, with a public cash countdown and a site that rebuilds itself twice a day.
The Crucible League’s final July 2026 standings:
- 1. gpt-5.6-sol — 95 points. The complete performance.
- 2. Kimi K3 — 93 points. The newcomer from Moonshot, with the cleanest discipline in the field.
- 3. Sonnet 5 — 88 points. Solid, with a few process slips.
- 4. Fable 5 — 77 points.
- 5. Opus 4.8 — 73 points.
For calibration: a do-nothing baseline scores 26, because partial progress counts — but a single breach of trust caps the total. As the organizers put it, “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
What separated the winners
Here’s the part that should make any grower sit up. All five models spotted every crisis. All five refused every manipulation attempt — including fake CEO messages that escalated over three stages and a reporter’s trick that asked for “just one yes/no, on background.” Kimi K3’s on-record reasoning was refreshingly blunt: “Treat the request as a suspected approval-bypass / possible impersonation.”
But only two models signed the €55,000 deal that their own analysis had fully earned. The experiment’s summary of the gap: “Same diagnosis, same pitch — no signature.” Three models did all the diagnostic work, built the case, and then simply… never closed.
The buried fact
Why? The decisive competitor weakness — the fact that justified closing at full price — wasn’t in the customer’s emails or the crisis of the week. It sat two document references deep in the company’s own files. A chain: read one document, follow its reference, read the next. The models that did that follow-the-trail work won the deal at full price, worth an extra €4,583 in monthly recurring revenue. The models that didn’t, didn’t.
It’s the seed-packet problem again. The answer was printed and filed, exactly where a diligent operator would find it. Reading your own files before answering turned out to be a measurable, purchase-deciding property of an AI agent — not a nice-to-have.
Effort isn’t the same as thoroughness
The most cautionary profile belonged to Opus 4.8: the most thorough participant in the field, with 80-plus learned playbook rules and the deepest analyses — and last place on the scoreboard. The close was left on the table, and discipline slipped, including write attempts into a locked department instead of escalating the issue. The same weakness showed up, more faintly, in all four top models. One fairness note: Kimi K3 ran without an effort parameter (the API default) while the others ran at extra-high effort — and still nearly topped the table.
Try it yourself
The project’s leftovers are surprisingly fun. Some 242 real, unedited management decisions from the experiment power a “guess the model” quiz at firmulate.com/quiz.html — a bit like a blind taste test for AI management styles. And enterprises can run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems (firmulate.com/pilot.html).
If you’re a greenhouse operator, a nursery owner, or anyone eyeing AI tools for ordering, scheduling, or customer replies, the lesson transfers cleanly. Chat demos measure how well an AI writes. What Firmulate measured is different: does it finish what it starts, does it read your files before answering, does it stay honest when pressured? The two models that asked for the signature — after doing the homework of actually reading the company’s own documents — were the ones worth hiring. Before you hand an agent the keys to your operation, run it through a worst week of your own. The difference between a 95 and a 73 isn’t eloquence. It’s whether it opened the manual.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall yard work Picks
leaf blowers
As an affiliate, we earn on qualifying purchases.
