Decision models are a recent class of non-generative language models: a backbone reads a state
once and a small head returns a probability for every option of every typed question (multiple
choice, yes/no, ordinal score), with no text generated. We test six of them: TypeSafe’s hosted
Jev, which introduced the SystemOne interface, and two open families released in October 2026,
Cloudflare’s Clef and vLLM Semantic Router’s Decision 2.0, served on a local NVIDIA DGX
Spark (GB10). We ask whether they can replace or assist a generative model in the judgment
steps of a working LLM pipeline for tabletop role-playing campaigns. Labels are the game
master’s own past rulings, frozen into one versioned dataset: 766 reviewed transcript-correction
cards, 864 triaged tokens, 497 spelling rulings and 115 entity-type merge decisions, plus 12 paired
NPC-behaviour scenarios. Decision models work where the label is a coarse property readable
from meaning: Clef 27B and Jev each forwarded about half of a transcript’s unknown-token pile
on a held-out split without discarding any of 84 real names, and five of six models moved NPC
behaviour in the expected direction on 11 of 12 paired scenarios against 5 of 12 for a published
dice-table system. They fall short where the label is a string-level judgment (choosing a canonical
spelling: 66–69% for the best decision models against 76% for a local generative model), a house
convention, or a consistency property across many items (typing extracted facts one at a time
created about as many split entities as it repaired; a model-free per-entity vote matched the
GM’s filing 32 times in 33 where it decided). Jev is the exception that shows what to ask for: on
spelling it abstained instead of guessing, with 28 confident wrong names against 109 for Clef 27B.
Fine-tuning a 4B decision model on the GM’s rulings recalibrated it (triage AUC 0.90 to 0.96)
but did not teach the conventions. We also report that an apparently strong verdict-prediction
result was an artifact of the classifier reading the upstream model’s own hedging, and that vendor
latency claims did not transfer to the GB10 (127 ms measured vs. 18.4 ms claimed).
The full paper can be found here – https://drive.google.com/file/d/1e0pzEsFrulo_QrLhIaIqhuz4YDujpQS0/view?usp=sharing
Leave a Reply