Your coding benchmark measured a model you will never actually run

Anthropic’s guide to getting the most out of Opus 5.5 ends with a long checklist. You delete any line asking the model to think carefully. You list the design habits you want banned, such as cream backgrounds, italic accent words and pill-shaped buttons. You write rules in CLAUDE.md about when it may stop and ask. You decide what should happen when a safety classifier moves your conversation to an older model. None of those settings appears in a benchmark configuration. I think that turns Opus 5.5’s 66.4% on Terminal-Bench 4.0, against 55.8% for Fable 5.1, into a statement about a setup no team actually runs.

I opened the guide expecting launch-week marketing. What I found reads more like release notes for behaviour that moved. Opus 5 shipped on 24 July and Opus 5.5 on 22 September, about two months apart. In that window the model picked up habits that each need a patch in your prompts. The guide says it sometimes halts a long run to summarise the next step without taking it, or to offer to continue. It says that in long chats it can go back over earlier answers while handling a short follow-up, which slows replies, and it gives you an instruction to paste in so it stops. It says that without direction it falls back on a few default visual styles, and that telling it to avoid a generic look mostly swaps one default for another. I read these as fixes for drift between point releases.

The settings you control already swing the results more than the gaps on the vendor’s chart. Vellum reports that at medium effort Opus 5.5 scores 54.6%, ahead of GPT-6 Astra’s peak of 53.3%, at roughly 80% less per task attempt. MindStudio says medium effort matched or beat high and max on several benchmarks. CodeRabbit’s testing of Opus 5 found that more reasoning did not reliably produce better reviews, and that the default configuration found the most issues while its precision fell to 26.4%, with 110 nitpicks. Turning one knob reorders the leaderboard and changes how many useless comments your engineers wade through.

Then there is the classifier. When it fires, the request is quietly rerouted: The New Stack reports that most flagged cybersecurity requests go to Opus 4.8, and biology and frontier-LLM flags go to Opus 5. The guide adds that the check covers the whole conversation, files and search results included, so something pasted an hour ago can trigger the switch. Zapier ran AutomationBench without fallback models and counted each intervention as a failure, which put Opus 5.5 at 40.0% against GPT-6 Astra’s 41.4%. A team doing security audits in Claude Code could benchmark one model and get a different one several hours into a run.

Pass rates hide the rest. Sonar’s independent test gave Opus 5.5 87.7% across 544 HumanEval and MBPP tasks, against 88.6% for Opus 5, which is effectively a tie. Underneath, security blockers fell 53% and maintainability blockers fell 20%, while bug density rose 12%, from 576 to 644 per million lines, and concurrency findings rose 44%. If your codebase is full of threads, that tie is a regression.

Anthropic’s API documentation says existing Opus 5 prompts should perform well without changes, and it covers effort, safeguards, unattended tasks and frontend design in detail. GitHub’s chief product officer, Mario Rodriguez, says Opus 5.5 solved terminal tasks in Copilot CLI and VS Code in less than half the steps of Opus 5 while using fewer tokens. I believe both claims. But one practitioner guide points out that the API guide never mentions Claude Code by name, and the Claude Code advice lives in a separate blog playbook that tells you to rewrite your saved instructions. Rodriguez’s numbers come from GitHub’s own tools, which supports my point: the harness produced the result. Anthropic itself, per The New Stack, says that at this level a few benchmark points don’t translate into a noticeable difference in real-world use. On that, the vendor and I agree about its own chart.

So run the test yourself. Take every merged pull request from the past quarter in your repository where a bug was later found. Give each candidate model the pre-fix diff with your real CLAUDE.md at medium effort, and ask it to list only merge-blocking problems. Count bugs caught, false alarms, and runs that showed a notice starting “Switched to”. Keep the harness in version control. If the two-month cadence holds, I expect another Opus release with another checklist before the end of November 2026, and you will want to rerun it that afternoon.

Prompted by Getting the most out of Opus 5.5 in Claude and Claude Code / claude.dev Blog, claude.dev Blog.