Jev and the routing paradox: paying a smart model to pick a cheap one
LLM routing has a structural flaw — the classifier that decides which model to use is itself an LLM, so you pay generative prices for a decision. TypeSafe's Jev suggests a different shape for that layer.
- Enterprise AI
- LLM Routing
- FinOps
- Architecture
The router tax
Model routing is now standard practice in enterprise AI platforms. Send the easy requests to a small model, escalate the hard ones, and the bill drops without anyone noticing a quality change. The peer-reviewed RouteLLM work reported 85% cost savings while retaining 95% of GPT-4 quality in controlled evaluations. The idea is sound and the savings are real.
What routing architectures tend to under-account for is the router itself.
To decide which model should answer, something has to read the request and judge its difficulty. In most implementations that something is another language model. NVIDIA’s NeMo Switchyard, for example, summarises the last few conversation turns and has a classifier sort them into simple, medium, complex, or reasoning tiers before dispatching to the matching model.
That classifier is a generative model being asked to produce, in effect, one word. It thinks in tokens, and you are billed for the thinking. A reported comparison of classifier choices in Switchyard found the expected shape: pick a capable classifier and routing is accurate but expensive; pick a cheap one and it is affordable but slow or unreliable. Cost, latency, accuracy — choose two.
The tax compounds in the places that hurt most. A single-turn classification is cheap. An agent on its fourteenth step, or a support conversation twenty turns deep, means feeding a large history into the router before anything is answered. Input tokens are cheap individually and relentless in aggregate: a 32k-token context classified on every step is a real line item, and it is incurred whether or not the routing decision turns out to matter.
That trade-off is usually presented as a tuning problem. I think it is structural, and it comes from using a text-generation engine to produce something that was never text.
A different shape for the decision layer
TypeSafe released Jev on 15 September 2026, describing it as a “System One” model — fast, automatic, pattern-matching judgement, as distinct from the deliberate reasoning generative models perform.
It exposes three question types rather than a prompt:
- Choice — pick one of a set of options you declare up front, with a probability distribution across them.
- Score — place the input on an ordered rubric you define.
- Noul — a bounded yes-or-no, returned as a single probability between 0 and 1.
For routing, that maps almost too neatly: a Choice over your model tiers, and
optionally a Score on input complexity to inform the threshold.
The request shape makes the design intent obvious. Jev is available on
OpenRouter as typesafe/jev-1.13 through its
Decisions endpoint,
so this is testable today rather than blocked behind the early-access waitlist:
open_router.alpha.decisions.create(
model="typesafe/jev-1.13",
state={
"turns": 14,
"latest": "Refactor this module and explain the migration risk.",
"tools_available": ["repo_search", "patch"],
},
questions={
"tier": {
"type": "choice",
"instructions": "Which model tier should answer this?",
"criteria": {
"small": "Lookup, extraction, or a direct factual answer.",
"mid": "Summarisation or light multi-step work.",
"frontier": "Multi-step reasoning, code generation, or planning.",
},
},
"needs_tools": {
"type": "noul",
"instructions": "Does answering require calling a tool?",
"criteria": {
"true": "The request depends on repository or system state.",
"false": "The request can be answered from the conversation alone.",
},
},
},
)
Note that state is structured data, not a prompt string. You are handing the
model program state and asking typed questions about it.
What comes back is an answers map keyed by your own question names, each
carrying a typed value and its probability — a noul resolves to something like
0.92, a choice to a distribution across the options you declared. There is no
written rationale, and nothing to parse.
Two consequences matter architecturally.
Output billing goes to zero. Not “approaches zero” — there are no output tokens to bill, because the model emits none. Published pricing is $0.042 per million input tokens with output free. In one independent test cited by O’Reilly, a general-purpose model consumed roughly 910 output tokens reasoning its way to a single yes-or-no answer, against 85 for Jev — which were not charged. That is a property of evaluating a constrained answer space in parallel, not a pricing promotion someone can withdraw next quarter.
The answer space is closed. Because the options are declared in advance, the failure mode of a generative classifier simply does not exist: it cannot return a tier you never defined, emit malformed JSON, or wrap its answer in an apology that breaks your parser. You still have to handle a wrong decision. You no longer have to handle an unparseable one.
Calibration is the training claim. TypeSafe says Jev is trained by a method it calls Reinforcement Learning for Calibrated Decisions, positioned against RLHF: where RLHF optimises for answers human raters approve of, RLCD is claimed to optimise for a decision paired with a confidence that reflects how often it is actually right. If that holds, the probability is a usable number rather than decoration, which matters enormously for what follows.
Treat the name with care, though. As Forbes’ Lance Eliot notes, RLCD is a phrase coined by the vendor rather than an established technique, and it is adjacent to but not the same as existing work such as reinforcement learning with calibration rewards or with verifiable rewards. The method itself is undisclosed. TypeSafe’s headline figures — 20–200× faster, 40–400× cheaper — are vendor numbers for a product days old. The 910-versus-85 measurement is one small independent test. Both are interesting. Neither is a benchmark on your workload, and I have not run it on mine.
What changes if this holds
Routing becomes always-on rather than selective. Today you route where the savings justify the classifier’s cost, which in practice means high-volume paths. Remove that cost and every request can be classified, including the low-volume expensive ones where misrouting hurts most.
Calibrated probabilities become a control surface. This is the part least discussed and most useful. A generative classifier returning the word “complex” gives you no measure of its own confidence. A distribution does. Set a threshold, route confidently above it, escalate ambiguous requests to a stronger judge — routing stops being a classification problem and becomes a risk-managed one with a deliberate escalation path.
Latency stops scaling with reasoning depth. A chain-of-thought classifier’s latency grows with how much it chooses to think. A single parallel pass is reported at 70–500ms. That upper bound deserves attention if you are holding a p99 budget under a second: it is a synchronous block in front of every request, and unlike streaming generation there is no first token to hide it behind. Budget for the ceiling, not the median.
Decision-shaped work separates from generative work. Support triage, fraud flags, alert classification, invoice approvals, agent action selection, LLM-as-a-judge evaluations — none need prose. They are typed questions wearing a chat interface because that was the only interface available. One analysis estimates 40–60% of a typical frontier bill is this kind of work, and that moving that slice could cut frontier spend 30–70% before integration and evaluation costs. Treat that as a hypothesis to test against your own telemetry, not a number to forecast against.
How I would put it into production
Behind a gateway, like any other model dependency.
Route through a managed endpoint with per-workload token budgets and structured telemetry, so the decision layer is observable from day one. Trim context at that gateway rather than passing whole histories through: the router needs enough state to judge difficulty, not the transcript. This is the single biggest lever on the router tax, and it is entirely in your control.
Keep a generative fallback and exercise it. A single-vendor, early-access dependency on the hot path of every request is a concentration risk regardless of how good the model is. OpenRouter’s own cookbooks sketch two patterns worth studying — gating agent tool calls, and a verified cascade where a cheap model drafts and the decision layer decides whether the frontier model is needed at all. The second is more interesting than routing, because it moves the decision after the cheap attempt rather than before it.
Monitor calibration, not just accuracy. A classifier whose confidence scores drift becomes dangerous precisely because your escalation threshold is quietly doing less work than you think. Log the full distribution, not only the argmax.
Keep arithmetic out of it. TypeSafe documents counting and arithmetic as known weak spots, which is a useful piece of candour and a clear boundary: ask the decision layer semantic questions, and do the counting in code where it belongs. A router that has to tally tool invocations before choosing a tier should compute the tally itself and pass it in as state.
And accept the real trade-off: you lose the explanation. A generative classifier can tell you why it chose a tier. A typed answer with a probability cannot. For routing that is usually acceptable. For a decision a regulator may ask about, the absence of a rationale is a design constraint to plan around rather than discover during an audit.
The part worth keeping
Whether Jev specifically becomes the standard is not the interesting question. What is interesting is the recognition that a large share of what we currently ask language models to do is not language work at all, and that we have been paying generative prices for classification because the shape of the tooling made it the obvious thing to do.
Routing was the clearest example, because there the cost of deciding is measured directly against the cost of the thing decided. It will not be the last.