A cascade’s savings are set by its first stage: every item the cheap model cannot settle
is paid for twice. Two earlier studies on a tabletop role-playing campaign pipeline evaluated
off-the-shelf non-generative decision models against a generative model [1], and designed a
Decision-2.0-Nox-4B → qwen3.8-flash-next cascade by replaying their frozen outputs, then
checked it live [2]. The first study also fine-tuned Nox on the game master’s (GM’s) rulings but
never put the tuned model in the cascade. Here we do, asking whether a domain-specialized
first stage removes more of the second. Because the frozen dataset caches Nox’s backbone
features, tuning its decision head and replaying the resulting cascade takes about a minute of
CPU time per variant, cross-fitted so that no item’s label reaches its own head or its own gate.
On triage, the tuned head lowered escalation from 33% to 25% and raised the cascade’s predicted
single-stream speed-up from 2.07× to 2.55×, while still missing no more real names than the
generative model. On fact typing the replay first showed the second stage almost vanishing,
then exposed why: every fact’s label is its entity’s label and entities recur, so a plain lookup of
the GM’s earlier rulings scores 99.0%, above every model. Held out by entity, the tuned Nox
alone tied the generative model on entities it had never seen (88.4% vs 88.3%). Holding out
by token on triage broke the missed-name constraint. We then served the tuned heads on the
real two-box system, with gates and predictions pre-registered by hash. Live, the triage cascade
scored +2.9 points above qwen (CI [+1.9, +4.2]; predicted +2.8), escalated 24.7% of tokens,
missed exactly as many names as qwen, and was 2.51× faster single-stream (predicted 2.38×);
the entity-disjoint fact-typing cascade escalated 4.3% and was 4.49× faster (predicted 4.36×) at
+1.1 points (CI [-0.1, +2.3]). The replay routed 100.0% and 99.9% of items as the live system
did. A specialized first stage removes most of the second on fact typing and a quarter of it on
triage, where the second stage stays because the tuned model alone drops twice as many real
names. Set beside Vela 2.0, a full fine-tune of the same Nox for safety and routing, the head-only
gain on our tasks (+5.5 AUC points on triage) is of the same order as Vela’s mid-range gains on
its own, at a small fraction of the effort; Vela itself, tuned for another domain, is a worse first
stage here than the Nox it came from.
The full paper can be found here https://drive.google.com/file/d/1QQD4cFtS4QS3wINR5WNXyV7r2MXidVjB/view?usp=sharing
Leave a Reply