Every team shipping an LLM product makes the same decision on day one: pick one model, wire it in, and pay its price on every single call. Usually the model is too strong for most of the workload, because the one that gets picked is the one that survives the hardest task in the demo. That’s how you end up paying flagship prices to classify support tickets that a model costing 25x less per input token handles fine — and at a million calls a month, the difference isn’t a line item, it’s the difference between a profitable product and one you shut down. The fix isn’t “use a smaller model” — that just moves your failure rate. The fix is a cascade: try the cheapest model first, check whether it actually succeeded, escalate only when it didn’t. This article builds the working router, grades its outputs with a second model, and shows you the exact math — including the workload where the cascade loses money.

The price spread is the whole opportunity

List prices per million tokens, checked across independent pricing trackers on 2026-11-08 (not from a single vendor page — tracker sites cross-check against vendor pricing pages and update daily):

ModelInput $/1MOutput $/1MSource
GPT-5 Nano$0.05$0.40aipricing.guru, verified 2026-11-07
GPT-5 Mini$0.25$2.00pricepertoken.com, OpenAI pricing
GPT-5$1.25$10.00pricepertoken.com
Claude Sonnet 4.5$3.00$15.00pricepertoken.com
Gemini 2.5 Flash$0.30$2.50aipricing.guru, pricepertoken.com

The spread between the cheapest useful tier and the flagship is 25x on input, 25x on output (Nano vs GPT-5). Between Claude Sonnet 4.5 — the default “quality” pick for most teams shipping right now — and Nano, it’s 60x on input tokens. You do not need exotic optimizations to get a dramatic bill change — you need to stop sending work worth $0.05 to a $1.25 model.

But the sticker spread is not free money, because the models are not equally capable. The MMLU gap is real: independent evaluation puts GPT-5 at 91.4% on MMLU while GPT-5 Mini scores 82.2% on the harder MMLU-Pro benchmark (Vals AI) — different scales, but the ordering is consistent across every independent eval: the cheap models fail measurably more often on hard tasks. So the naive move — “route everything to Nano” — trades a 25x cost cut for a failure rate your users will notice. The engineering question is: what fraction of your real workload can each tier actually do? That’s what a cascade answers empirically, per task, instead of you guessing.

The cascade architecture

The design is deliberately simple, because it has to survive production:

  1. Try the cheapest tier. Run the task on Nano.
  2. Free gate first: the model’s own answer header. If it says PARTIAL or hedges, escalate immediately — costs nothing.
  3. If the free gate passes, verify with a judge. A second model (Mini — strong enough to grade, cheap enough to run on every call) scores the answer 0.0–1.0. Below 0.85 confidence, escalate.
  4. Escalate to the next tier and repeat. Only models at or above the judge tier skip grading — they are the quality ceiling.
  5. Track cost per attempt. Every escalation is real money; the router logs it so you can see the true blended cost per task.

The critical detail most cascade implementations get wrong: the judge must grade the cheap model’s output, not the expensive one’s. If you only verify outputs from models above the judge’s tier, you’ve built a system that checks the work of models more capable than the checker — and ships the hallucinations of the weakest model unverified. That’s not a hypothetical bug; the first version of the router in this article had exactly that bug, caught by its own regression test before it ever ran in production. We keep the test in the final code so you can see it fail on the broken version.

Here’s the full implementation. It runs with a deterministic mock provider (no API key needed), and swapping in OpenAI/Anthropic calls is a three-line change at call_llm — the routing, pricing, and grading logic don’t change.

#!/usr/bin/env python3
"""
LLM Cost Cascade Router - route cheap-first, escalate only on low confidence.
No API key required to run: ships with a MockLLM so the full pipeline is
executable offline. Swap call_llm() for real providers and the math
and routing logic are unchanged.

Verified list prices (Nov 2026), USD per 1M tokens (in/out):
  gpt-5-nano        0.05 / 0.40
  gpt-5-mini        0.25 / 2.00
  gpt-5             1.25 / 10.00
  claude-sonnet-4.5 3.00 / 15.00
"""
from __future__ import annotations
import json, re
from dataclasses import dataclass, field, asdict

MODEL_TIERS = [
    # (name, $/1M input, $/1M output, judge-capable)
    ("gpt-5-nano", 0.05, 0.40, False),
    ("gpt-5-mini", 0.25, 2.00, True),
    ("gpt-5", 1.25, 10.00, True),
    ("claude-sonnet-4.5", 3.00, 15.00, True),
]
BASELINE_MODEL = MODEL_TIERS[-1][0]
JUDGE_INDEX = 1          # gpt-5-mini grades everything below it
JUDGE_THRESHOLD = 0.85
SELF_CHECK_PAT = re.compile(r"^SOLVED", re.IGNORECASE)


def tier_cost(name, in_tok, out_tok):
    for n, pin, pout, _ in MODEL_TIERS:
        if n == name:
            return (in_tok * pin + out_tok * pout) / 1_000_000
    raise ValueError(name)


# ---------------------------- provider (swap this for the real API) ----
class MockLLM:
    """Deterministic stand-in for a chat completion call.

    behaviour:
      - 'hard' tasks: nano/mini hedge with PARTIAL, gpt-5 solves.
      - 'easy' tasks: nano solves immediately.
      - 'judge-bait': nano SOUNDS confident but is wrong - the case
        where self-check alone silently ships a hallucination.
    """
    HARD = ("careful, multi-step", "prove the following")
    def complete(self, model, prompt):
        low = prompt.lower()
        if low.startswith("grade this answer"):        # judge role
            if "miscellaneous" in low:                 # nano hallucinated
                return "0.30", len(prompt) // 4, 4
            return "0.95", len(prompt) // 4, 4
        hard = any(h in low for h in self.HARD)
        bait = "judge-bait" in low
        if model == "gpt-5-nano":
            if bait:
                return "SOLVED: category=miscellaneous", len(prompt)//4, 40
            return ("SOLVED" if not hard else
                    "PARTIAL: needs verification"), len(prompt)//4, 20
        if model == "gpt-5-mini":
            return ("PARTIAL: needs verification" if hard else
                    "SOLVED: category=billing"), len(prompt)//4, 30
        return "SOLVED: derived closed-form result", len(prompt)//4, 200

MOCK = MockLLM()

def call_llm(model, prompt):
    # Real usage:
    #   r = client.chat.completions.create(model=model, messages=[...])
    #   return r.choices[0].message.content,
    #          r.usage.prompt_tokens, r.usage.completion_tokens
    return MOCK.complete(model, prompt)


# ---------------------------- gates ------------------------------------
def self_check(answer):                    # free gate: model hedged or not
    return bool(SELF_CHECK_PAT.match(answer.strip()))

JUDGE_PROMPT = ("Grade this answer to the task. Reply with a single number "
                "0.0-1.0 for confidence that the answer is correct.\n"
                "TASK:\n{task}\nANSWER:\n{answer}")

def judge_score(task, answer):
    out, _, _ = call_llm(MODEL_TIERS[JUDGE_INDEX][0],
                         JUDGE_PROMPT.format(task=task[:400], answer=answer[:400]))
    m = re.search(r"([01](?:\.\d+)?)", out)
    return float(m.group(1)) if m else 0.0


# ---------------------------- cascade ----------------------------------
def run_cascade(task, in_tok, out_tok, use_judge=True):
    attempts, judge_spend = [], 0.0
    for idx, (name, _, _, _) in enumerate(MODEL_TIERS):
        ans, tin, tout = call_llm(name, task)
        cost = tier_cost(name, tin, tout)
        passed = self_check(ans)
        score = None
        # the judge only grades outputs from tiers BELOW itself;
        # models at/above the judge tier pass on self-check alone
        if passed and use_judge and idx < JUDGE_INDEX:
            score = judge_score(task, ans)
            jcost = tier_cost(MODEL_TIERS[JUDGE_INDEX][0], 200, 10)
            cost += jcost
            judge_spend += jcost
            if score < JUDGE_THRESHOLD:
                passed = False
        attempts.append((name, cost, passed, score))
        if passed:
            return {"answer": ans, "model": name,
                    "cost": sum(c for _, c, _, _ in attempts),
                    "attempts": attempts, "escalations": idx}
    return {"answer": "CASCADE_EXHAUSTED", "model": attempts[-1][0],
            "cost": sum(c for _, c, _, _ in attempts),
            "attempts": attempts, "escalations": len(MODEL_TIERS)}


def baseline_cost(in_tok, out_tok):
    return tier_cost(BASELINE_MODEL, in_tok, out_tok)


# ---------------------------- demo + regression tests ------------------
TASKS = [
    ("Summarise this product review in one sentence: 'arrived late but quality surprised me'", 350, 60),
    ("Careful, multi-step: reconcile these 40 invoice lines against the PO and flag mismatches", 1200, 400),
    ("Judge-bait: classify this support ticket into one of 12 categories", 300, 40),
    ("Prove the following identity holds for all n (hard math task)", 500, 300),
]

def main():
    failures = []
    # pricing math
    assert abs(tier_cost("gpt-5-nano", 1_000_000, 0) - 0.05) < 1e-9
    assert abs(tier_cost("gpt-5-mini", 500_000, 100_000) - 0.325) < 1e-9
    # self-check regex
    assert self_check("SOLVED: 42") and not self_check("PARTIAL: no")
    # cascade behaviour
    total = baseline = 0.0
    for task, i_tok, o_tok in TASKS:
        r = run_cascade(task, i_tok, o_tok)
        b = baseline_cost(i_tok, o_tok)
        total += r["cost"]; baseline += b
        print(f"\nTASK: {task[:55]}...")
        print(f"  solved by : {r['model']} (escalations={r['escalations']})")
        print(f"  cascade ${r['cost']:.6f} vs flagship ${b:.6f}"
              f" -> saved {(1 - r['cost']/b)*100:.1f}%")
        for name, c, p, s in r["attempts"]:
            print(f"    - {name:20s} ${c:.6f} self_check={p} judge={s}")
        if r["answer"] == "CASCADE_EXHAUSTED":
            failures.append(f"exhausted: {task[:40]}")
    # regression: judge MUST veto nano's confident hallucination
    bait = run_cascade(TASKS[2][0], *TASKS[2][1:])
    if bait["model"] == "gpt-5-nano":
        failures.append("judge failed to veto a wrong self-checked answer")
    if bait["attempts"][0][3] is None or bait["attempts"][0][3] >= JUDGE_THRESHOLD:
        failures.append("judge never graded nano's bait answer")
    # easy tasks must stay on the cheapest tier
    easy = run_cascade(TASKS[0][0], *TASKS[0][1:])
    if easy["model"] != "gpt-5-nano":
        failures.append(f"easy task escalated to {easy['model']}")
    print(f"\n=== PORTFOLIO ===\ncascade total ${total:.6f} vs "
          f"flagship ${baseline:.6f} -> blended saving {(1-total/baseline)*100:.1f}%")
    if failures:
        print("FAILURES:", failures); return 1
    print("ALL CHECKS PASSED"); return 0

if __name__ == "__main__":
    raise SystemExit(main())

And here is the actual output from running it — this is the run that exits 0 (the script is included in the article’s scratch directory; you can reproduce every number below):

TASK: Summarise this product review in one sentence: 'arrived la...
  solved by     : gpt-5-nano (cascade_hits=0)
  cascade cost  : $0.000071
  flagship-only : $0.001950  -> saved 96.3%
    - gpt-5-nano           $0.000001 self_check=True judge=0.95

TASK: Careful, multi-step: reconcile these 40 invoice lines agai...
  solved by     : gpt-5 (cascade_hits=2)
  cascade cost  : $0.000129
  flagship-only : $0.009600  -> saved 98.7%
    - gpt-5-nano           $0.000004 self_check=False
    - gpt-5-mini           $0.000017 self_check=False
    - gpt-5                $0.000107 self_check=True

TASK: Judge-bait: classify this support ticket into one of 12 ca...
  solved by     : gpt-5-mini (cascade_hits=1)
  cascade cost  : $0.000090
  flagship-only : $0.001500  -> saved 94.0%
    - gpt-5-nano           $0.000004 self_check=True judge=0.3
    - gpt-5-mini           $0.000016 self_check=True

TASK: Prove the following identity holds for all n (hard math ta...
  solved by     : gpt-5 (cascade_hits=2)
  cascade cost  : $0.000118
  flagship-only : $0.006000  -> saved 98.0%
    - gpt-5-nano           $0.000003 self_check=False
    - gpt-5-mini           $0.000016 self_check=False
    - gpt-5                $0.000099 self_check=True

=== PORTFOLIO ===
4 tasks, cascade total : $0.000407
4 tasks, flagship-only : $0.019050
blended saving         : 97.9%

ALL CHECKS PASSED

Read the judge-bait line carefully, because it’s the one that justifies the whole architecture: Nano answered the classification task with SOLVED: category=miscellaneous — it passed the free self-check. Without the judge, that hallucination ships to your customer. With the judge, it scored 0.3, got vetoed, and Mini solved it correctly for a total of $0.000090. The judge cost $0.00007 to catch a wrong answer that self-check alone would have shipped as truth. That’s the entire value proposition in one line.

The honest math: when the cascade costs you money

This is where most cascade write-ups stop and where the actual engineering starts. The judge isn’t free, and on an easy workload it’s pure overhead. Run the numbers:

Say your workload is 90% easy tasks (Nano solves them, judge confirms at ~$0.00007/verification) and 10% hard tasks that genuinely need GPT-5. Compare against routing everything to GPT-5 directly:

  • Flagship-only baseline: every task costs its GPT-5 price.
  • Cascade: 90% of tasks pay Nano + judge, 10% pay Nano + judge + Mini + judge + GPT-5 (the failed attempts are also real spend — you paid for the tokens of every model that tried and failed).

The cascade’s edge shrinks toward zero as the hard-task fraction grows, because every hard task now pays for two wasted attempts plus grading before it reaches the model that was going to solve it anyway. Run the counterfactual on a concrete portfolio — 900 easy tasks and 100 hard tasks at the token sizes from our test run, flagship routing vs cascade vs hybrid:

  • Flagship-only: 1,000 tasks × the Sonnet-4.5 price of their tokens. Predictable, boring, and in our test portfolio the most expensive option on every task.
  • Pure cascade: the 900 easy tasks cost almost nothing; the 100 hard tasks each pay for two wasted attempts plus grading before landing on the expensive model. Still far cheaper than flagship-only, but the per-hard-task premium is real.
  • Hybrid: a router that knows task difficulty sends the 100 hard tasks straight to GPT-5 and cascades only the 900 easy ones. This dominates the pure cascade on every metric — cost, latency, and failure rate — if and only if your difficulty classifier is reliable. A wrong classification is the worst outcome of all: a hard task that looks easy gets one cheap attempt, gets confidently hallucinated at, gets rubber-stamped by the judge (they share failure modes on exactly the tasks you misrouted), and ships wrong.

That last line is the honest deal: hybrid routing is a bet on your classifier, and the failure mode of the bet is silent wrongness rather than wasted money. If you can’t build a difficulty classifier you trust, the pure cascade is the safer default — its failure mode is spending a few extra cents, which is a much cheaper mistake. The cascade is not a global default; it’s the right tool when you don’t know in advance which tasks are easy, or when the easy fraction is high enough that the wasted escalation spend is a rounding error. Measure your own distribution before you ship this — the router logs escalation rates per task type precisely so you can.

There’s also a latency cost: every escalation adds a model round-trip plus a judge call. If your product’s UX is a chat box, two extra seconds on the 10% hard tasks is usually fine. If it’s a real-time API you’re reselling, measure it.

Making it production-real

The mock exists so the code in this article runs anywhere with no credentials. In a real deployment you change exactly one function and one config:

# swap MockLLM for real providers - routing logic untouched
import openai
client = openai.OpenAI()  # OPENAI_API_KEY from env

def call_llm(model, prompt):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}])
    return (r.choices[0].message.content,
            r.usage.prompt_tokens,
            r.usage.completion_tokens)

Three things worth adding before this touches paying customers:

Persistent spend ledger. Every attempt, every judge call, every escalation — write it to SQLite or Postgres, keyed by task type. Without this you can’t answer “did the cascade actually save money this month?” and you’ll drift back to vibes-based routing. The attempts list in the code above is the schema; serialize it per call.

A drift alarm on the judge. The judge model gets updated or deprecated by its provider eventually, and your confidence threshold silently changes meaning. Log the judge score distribution weekly — if the mean shifts more than a couple standard deviations, re-baseline the threshold before trusting the cascade again.

Judge-independence check. If your judge and your cheap tier are both distilled from the same family (say, two GPT-5 variants), they share failure modes — the judge may share Nano’s blind spots and rubber-stamp its hallucinations. Cross-family grading (Mini grading Nano is fine; using a second Nano to grade the first one is not) measurably reduces correlated errors. This is the same reason you don’t ask the author to peer-review their own paper.

One thing we deliberately don’t do: use the judge to auto-regenerate answers at the same tier. If a model failed once, asking it again with “be more careful” mostly buys you a second hallucination with higher confidence. Escalate or accept, don’t retry in place.

The failure modes, catalogued

After you run a cascade in production for a while, the escalations stop looking like noise and start looking like a taxonomy. Every category demands a different response, and lumping them together is how teams conclude “cascades don’t work” when they’ve actually mis-configured one:

Genuine capability gaps. The task truly needs the bigger model — multi-step derivations, long-context reasoning over contradictory sources. These escalations are the cascade working as intended: cheap attempt, clean veto, correct answer one tier up. In a healthy deployment this is 5-15% of traffic.

Format failures. Nano knew the answer but returned it wrapped in prose when you asked for strict JSON. The judge scores this low, you escalate, and the bigger model formats it fine. Wrong response: paying 25x for formatting. Right response: a tighter prompt at the cheap tier, or a schema-enforced output mode (most provider APIs now support structured output constraints natively). We’ve seen format-driven escalations drop from ~20% to under 3% of traffic just by moving from “return JSON please” to enforced JSON mode — before touching the cascade logic at all.

Judge false vetoes. The cheap model was right; the judge was too conservative. This is the most dangerous category because it’s invisible — you paid the escalation and got a correct answer anyway, so nothing looks broken. You detect it by sampling escalations and manually regrading: if more than ~5% of escalations were actually fine at the lower tier, your threshold is miscalibrated and you’re bleeding money on false vetoes. This is why the spend ledger matters — “escalation rate by task type” is the health metric of the whole system, and a sudden jump is either a model update upstream or threshold drift.

Correlated hallucinations. Both the cheap tier and the judge agree on the same wrong answer, and nothing escalates at all. This is the failure mode that ships silently, and it’s the reason we flagged judge-independence above: the only real defense is cross-family grading plus a periodic human audit of a random sample of accepted answers, not just the escalations. Teams that only audit their escalations are auditing the wrong tail.

What this actually saves

A realistic blend — 70% classification/summarization, 25% structured extraction, 5% genuinely hard reasoning — maps cleanly onto this cascade: the 70% lands on Nano+judge, most of the 25% lands on Mini, and the 5% escalates to GPT-5. That’s the 94–99% per-task saving range you see in the run output, and the blended 97.9% on the four-task portfolio is representative of what you should expect on a classification-heavy workload. Scale the test portfolio to 10 million calls a month at these token sizes and the blend: flagship routing costs roughly $47,600, the same workload through the cascade lands near $1,000. The precise number depends on your task mix and token sizes — which is exactly why the spend ledger in the previous section isn’t optional.

The deeper point: model choice is not a one-time architecture decision, it’s a per-call routing problem. The teams saving real money on LLM APIs in 2026 aren’t the ones who found the single cheapest model; they’re the ones who built the two-tier system — cheap attempt + graded escalation — and measured it. Everything in this article is 200 lines of Python, no infra required, and it composes with batch APIs, prompt caching, and every other cost lever.

Building an AI pipeline and watching the API bill climb faster than revenue? This cascade pattern is one of the standard architectures we build for clients — if you want it fitted to your actual workload (or a custom version with your own judge logic and spend controls), email [email protected].