Your startup is paying a reasoning model to write JSON it could answer in one token

I opened this post expecting ammunition for a column I had half written: startups should stop renting frontier models and fine-tune a small open one for their routing and classification calls. Privatemode’s write-up made me rewrite it. They fine-tuned nothing. They took GLM-5.3-Flash as it ships, and the model is hardly small. The vLLM recipe lists it at 320 billion parameters in total, 18 billion active per token.

The method is simple once you see it. Put the state and a numbered list of options into the prompt as JSON. End the assistant turn with a prefilled choice_index: so the next token has to be a number. Then ignore the token the model emits and read the log probabilities it assigned to each index token at that one position, using vLLM’s logprob_token_ids. Normalise over the options and you have a choice plus a probability for every alternative, from one forward pass and one output token.

On 28 text datasets against TypeSafe’s Jev, a model trained for exactly this job, the prompted GLM scored 10 wins, 10 losses and 8 ties, with a median gap of 0.7 percentage points in Jev’s favour and p = 0.64. From Germany the GLM setup answered in 180 ms to Jev’s 264 ms; from the US Jev won, 164 ms to 299 ms. Jev is cheaper at list prices, about EUR 16 per million decisions against EUR 62.

Now compare that with what I suspect most startups run in production: a chat call to a reasoning model that thinks for a while, writes a JSON object, and gets parsed. Privatemode ran that as a control on the same model. Letting GLM reason first lifted accuracy from 85.5% to 89.9% on two-option tasks and from 79.2% to 82.0% in the 21-to-80-option band. It also averaged 7.2 seconds per decision and cost about EUR 350 per million, against EUR 62. For ticket routing or clause tagging, I would give back those few points for that bill and answers that arrive in a few hundred milliseconds. The probabilities are a bonus the chat call never handed you: a coin-flip answer goes to a human, a near-certain one does not.

The price sheets push the same way. As of August 2026 the cheapest mainstream frontier API, GPT-5.6 Luna after OpenAI’s July price cut, charges $0.20 per million input tokens and $1.20 per million output. GLM-5.3-Flash lists at $0.04 input on OpenRouter. When the answer is a single token, the output column of the price sheet barely touches your invoice, and every token of reasoning you skip is money kept.

The fine-tuning crowd will say I have the order wrong, and they bring evidence. Bucher and Martini at the University of Zurich found in 2024 that smaller fine-tuned models consistently beat zero-shot prompted giants, GPT-4 and Claude Opus among them, at text classification. One newsletter reports a 3B model fine-tuned in 74 minutes that scored 81.7% on a legal-labelling task against 77% for Claude Sonnet 4.6. Privatemode’s own data agrees in places: on TREC with 42 fine-grained classes, the trained Jev held 85.6% while prompted GLM fell to 79.6%.

I accept all of that and still think fine-tuning belongs second. It needs labels, and labels are worse than teams assume. The benchmark found about 17% of banking77’s examples had two defensible answers, capping any system near 85%. If your taxonomy puts get_physical_card next to order_physical_card, a fine-tune will faithfully learn your confusion. The one-token setup gives you a calibrated baseline in an afternoon, and its probability spread shows which of your classes collide before you spend a week labelling. Fine-tune where the gap survives that. And if you want the model off the API entirely, Unsloth’s 1-bit quantisation fits in 102GB of memory but keeps only 71% of top-1 accuracy, which I would not bet a classifier on.

Two warnings from the same benchmark before you start. Renaming true and false on boolq cost GLM 20 points while the other systems lost under three, so your option names are part of your model and deserve their own test. And even at temperature 0, up to 3.5% of answers changed between identical runs on a busy server, so run every eval twice before believing a difference.

You can check this on your own traffic. Pull last month’s decisions that your product made through a chat completion. Replay them through a vLLM endpoint with the options numbered, choice_index: prefilled and logprob_token_ids set. If fewer than 3.5% of answers differ, the disagreement sits inside the noise Privatemode measured between identical runs, and every token your current setup writes beyond that single digit is one you paid for and threw away.

Prompted by Turn GLM-5.3-Flash into a Jev-like System One model, privatemode.ai.