Every team calling model APIs should run its own LLM proxy, LiteLLM hack included

If your team pays for model APIs and the only record of what happened lives on the vendor’s servers, you are reading a meter installed by the company that sends the invoice. I think every team calling Claude, GPT or anything else over HTTP should put a proxy it runs itself in front of those calls, logging which key called which model from which client, the token counts, the cache hits and the status of every request. A developer made this case on his blog this week with considerably more swearing than I will use, and with a complaint I recognise: the analytics panels the labs offer are pretty charts, and he says he can’t make a decision with them.

The best argument for a second meter comes from Anthropic itself. In a postmortem published on 17 September 2025, the company described three infrastructure bugs that degraded Claude’s answers from August into early September. One sent some Sonnet 4 requests to the wrong servers. It started at about 0.8% of requests and peaked at about 16% on 31 August. Customers spent weeks arguing in public about whether the model had got worse, with nothing on their own side to settle it. A proxy would not have shown them Anthropic’s routing table. It would have given them a per-day baseline of their own requests and responses to compare against, which is a better position than vibes and screenshots.

Then there is money. OpenAI discounts cached input tokens by 50% for prompts of 1,024 tokens or more. Anthropic bills cache reads at about 10% of the base input price, so on Claude a cache miss can cost roughly ten times as much for identical input. The blog author says he can trace why a cache miss happened during a coding session in about 20 seconds by hand, because his proxy logged it. A daily spend chart on a vendor dashboard will tell you the bill went up on Tuesday. It will not tell you that someone moved a timestamp to the top of the system prompt.

I expected the vendors’ own tooling to make most of this redundant, and it goes further than the blog post admits. OpenAI and Anthropic both offer admin APIs that break usage and cost down by key and model, and Anthropic supports OpenTelemetry export for Claude Code. That is real data. It is also the vendor’s count, aggregated, one vendor at a time, and it describes what you were billed. When Anthropic announced weekly usage limits for Claude Code subscribers on 28 July 2025, saying fewer than 5% would be affected, subscribers had no independent token count to check that claim against, and a proxy logging tokens per key would have given them one.

The objection I take seriously is that a proxy is a fat target. At 10:52 UTC on 24 March 2026, litellm 1.82.8 was published to PyPI carrying a malicious .pth file, according to FutureSearch’s write-up. LiteLLM is one of the most widely used open-source LLM proxies. The file stole SSH keys and cloud credentials and spread across Kubernetes clusters; 1.82.7 was also compromised, and both were pulled the same day. Anyone who argues that a self-hosted proxy concentrates your secrets in one exploitable process is describing something that already happened.

I still come down on the side of running one, because the alternative is keys scattered everywhere. GitGuardian counted 23.8 million new hardcoded secrets in public GitHub commits during 2024, up about 25% on the year before, and found about 70% of secrets leaked in 2022 were still valid. A single main key pasted into every agent, harness and laptop is how you end up in that dataset. Virtual keys issued by a proxy can be revoked one at a time without rotating the credential everything else depends on. Nor is leaving analytics to the vendor a safe harbour: in November 2025 OpenAI disclosed that an attacker had got into Mixpanel, its analytics supplier, and exported names, emails, approximate locations and org IDs of API platform users. Telemetry is a target wherever it sits. I would rather it sat somewhere I could audit and patch.

The LiteLLM incident is an argument for how to run a proxy. Pin it by hash and never let it auto-upgrade from PyPI. Put it in its own namespace with a service account that cannot see the rest of the cluster, so a poisoned release finds model keys and nothing else. Then, at the end of next month, reconcile your proxy’s token totals per key against the vendor’s usage API, and take any gap you cannot explain to the vendor’s support desk with request IDs and timestamps attached.

Prompted by If you don't run your own LLM proxy, how the fuck do you even know whats going on?, jrdsgl.com.