
Have you noticed Claude or GPT giving worse answers than they did at launch? You are not imagining it. Performance does not always stay consistent, and changes to an app or its settings can make answers worse without any change to the underlying model. Anthropic and OpenAI have both acknowledged incidents of degraded performance, as Nate Herk explains. Find out why everyday results can differ from published benchmark scores, as well as how effort settings, hidden instructions and routing affect the answers you get, and what independent daily trackers can and cannot tell you.
What the Postmortems Show
TL;DR Key Takeaways :
- Anthropic published two postmortems covering weeks-long declines in Claude’s performance; in one, it said its evaluations had failed to capture the degradation users reported.
- Daily trackers have not caught model weights changing. Live Nerf detected no drop in Opus 5.5 during its first 50 days, though it says small changes can slip past its checks. Margin Lab recorded a decline from 65% to about 50%, attributed to a Claude Code app update rather than the underlying model.
- Effort settings affect results: Opus 5.5’s launch coding score was 57.8% at maximum effort, versus 52% for the version people often use. Anthropic changed Claude Code’s default effort from high to medium without initially telling users and later acknowledged the wrong trade-off.
- Instructions, routing and bugs can also degrade performance. Anthropic’s testing showed that a 25-word limit on Claude’s text between steps made it worse; in a Fable 5 benchmark, a safety filter sent nine of 12 tasks to Opus 4.8 instead.
- Effort, instructions and routing can change without notice. From what was found, no law requires labs to disclose reductions to those settings; Connecticut’s law instead requires disclosure of usage limits before sign-up or renewal.
Anthropic has published two postmortems covering periods when Claude got worse for weeks. In one, users reported that Claude Code felt less intelligent. In the other, Anthropic said its evaluations had failed to capture the degradation users reported.
The day after GPT-5 launched, OpenAI’s Sam Altman posted that something had broken behind the scenes, making the model seem less intelligent. Companies sometimes give users a usage reset after problems like these.
A launch benchmark can also reflect a different setting from the one you use day to day. Anthropic’s launch page lists a 57.8% coding score for Opus 5.5, while the version people often use scores 52%. The benchmarks used maximum effort, whereas the user default is medium.
What Daily Trackers Can Detect
An AI model contains billions of numbers called weights, which are adjusted during training. Quantization makes a model smaller and cheaper to run by rounding those numbers, potentially losing some detail.
Some people suspect that a large launch-day model is later replaced with a smaller, quantized one. Daily trackers have not caught model weights changing.
Live Nerf runs the same 78 questions through a model each day. It detected no drop in Opus 5.5 during its first 50 days, though Live Nerf says small changes can slip past its checks.
Margin Lab recorded a decline from 65% to about 50%, attributed to an update to the Claude Code app rather than the underlying model. Independent individuals built these daily benchmarks; the companies did not publish daily scores of that kind.
Unlock more potential in AI model performance changes by reading previous articles we have written.
- Al Benchmarks Investigated : Do Companies Tune Private Builds for Leaderboards, Then Ship Weaker Versions?
- Claude Opus 5.5 Expands Session Limits but Retains Weekly Caps
- Generate 15-Second Motion Graphics with Claude Opus 5.5 Text Prompts
- 6 Claude Code Levels Explained : Map Skills from Prompts to Agent Teams
- Anthropic Adds Persistent JavaScript Mods to Claude Code
Effort and Instructions
Effort controls how much a model can think before answering. Anthropic changed Claude Code’s default from high to medium to make it faster, initially without telling users. It later acknowledged that it had made the wrong trade-off.
You can turn effort up to maximum, but higher effort costs more time and money. Too little can produce worse results. Comparisons using the same prompt at different Opus 5.5 effort settings also showed a difference in output.
AI apps usually surround a model with hidden instructions called a system prompt, typically located in the harness running it. You can add your own instructions too, such as a CLAUDE.md file. The surrounding instructions differ by harness.
Anthropic added an instruction telling Claude to keep its text between steps to 25 words or fewer. Its own testing showed that the instruction made Claude worse. OpenAI’s Astra also fired old skills too often, getting in the model’s way.
Routing and App Problems
Routing determines which model responds to a request. When Fable 5 came back online, a benchmark showed its debugging score falling from 86 to 26. But nine of the 12 tasks never reached Fable: a safety filter sent them to Opus 4.8 instead.
Users initially were not told when that routing happened. In one check, Fable refused 14 of 63 questions and silently routed those questions to Opus 4.8. An answer could therefore appear to come from Fable when it came from another model.
Bugs and infrastructure can also affect results without a known change to model weights. Anthropic shipped a bug that wiped Claude’s thinking on every turn, causing it to forget what it was doing. OpenAI found misconfigured servers that made Astra worse for part of its traffic.
Launch Demand and Disclosure
Running models costs money and GPU resources for each token, and demand can rise sharply after launch. Altman said OpenAI’s API traffic roughly doubled in 24 hours when GPT-5 launched.
Nate Herk suspects that compute constraints and competition could give labs an incentive to provide less of a model after attracting customers, but says that possibility is an opinion and cannot be proved. Labs have already reduced effort to make models faster.
Perceived decline may also come partly from expectations. After a heavily promoted launch, repeated use can expose failures that stand out, particularly when you move from launch-week demos to hard work requiring context or expertise.
For customers paying $20, $100 or sometimes $200 a month, effort, instructions and routing can all change without notice. From what was found, no law requires labs to disclose reductions to those settings. Connecticut’s law instead requires disclosure of usage limits before sign-up or renewal, not disclosure that a model has gotten worse.
Media Credit: Nate Herk | AI Automation
Disclosure: Some of our articles include affiliate links. If you buy something through one of these links, Geeky Gadgets may earn an affiliate commission. Learn about our Disclosure Policy.