wrong tool

A game master who uses language models to prepare sessions of a long-running tabletop
role-playing campaign needs grounding documents: a current world state, campaign state, party
record, planning notes and per-character dossiers, distilled from dozens of recorded sessions and
fed to the models as context. Producing these documents is easy to start and hard to get right,
because the details that matter are exactly the ones generative models get wrong: who did what,
in what order, which name denotes which entity, and whether something happened at all. Over
seven months (March–October 2026) and six campaigns of up to 70 chapters, one practitioner
built, measured and retired twelve distinct approaches to the problem. We reconstruct them
from the git history, design documents and experiment write-ups of the repositories involved,
and organise them by two axes: the substrate a pipeline reads (narrated chapter prose, atomic
extracted facts, retrieval indexes, or the structured session summaries the GM already reviews)
and who makes the precision decisions (ordering, attribution, identity, scope). Approaches
that let a model make those decisions failed in ways that were silent and that compounded:
a consolidated extraction shrank a planning document from 513 to 109 lines; an atomic-fact
ensemble reported a slain antagonist as alive, and 98% of its 15,465 facts were seen by only
one sample, so agreement could not serve as a filter; a knowledge-graph library stored 1.18
characters of generated text per character of source and answered “what was the most recent
session” with session 8 of 11. The approach that the GM finally promoted parses the scene order,
entity sections and memorable moments already authored in reviewed session summaries, with
no model call (67 summaries, 409 scenes, 868 dossiers, byte-identical across runs), and uses a
model only to render the final prose; it also flagged 51 claims in the GM’s own tracking files
as unsupported by the record. We distil the history into ten cross-cutting findings and a set of
design rules for grounding long-horizon generative work.

The full paper can be found here – https://drive.google.com/file/d/1USvYt7moAUf8GbDzUYwW33I9HOsTu39D/view?usp=sharing

Decision models are a recent class of non-generative language models: a backbone reads a state
once and a small head returns a probability for every option of every typed question (multiple
choice, yes/no, ordinal score), with no text generated. We test six of them: TypeSafe’s hosted
Jev, which introduced the SystemOne interface, and two open families released in October 2026,
Cloudflare’s Clef and vLLM Semantic Router’s Decision 2.0, served on a local NVIDIA DGX
Spark (GB10). We ask whether they can replace or assist a generative model in the judgment
steps of a working LLM pipeline for tabletop role-playing campaigns. Labels are the game
master’s own past rulings, frozen into one versioned dataset: 766 reviewed transcript-correction
cards, 864 triaged tokens, 497 spelling rulings and 115 entity-type merge decisions, plus 12 paired
NPC-behaviour scenarios. Decision models work where the label is a coarse property readable
from meaning: Clef 27B and Jev each forwarded about half of a transcript’s unknown-token pile
on a held-out split without discarding any of 84 real names, and five of six models moved NPC
behaviour in the expected direction on 11 of 12 paired scenarios against 5 of 12 for a published
dice-table system. They fall short where the label is a string-level judgment (choosing a canonical
spelling: 66–69% for the best decision models against 76% for a local generative model), a house
convention, or a consistency property across many items (typing extracted facts one at a time
created about as many split entities as it repaired; a model-free per-entity vote matched the
GM’s filing 32 times in 33 where it decided). Jev is the exception that shows what to ask for: on
spelling it abstained instead of guessing, with 28 confident wrong names against 109 for Clef 27B.
Fine-tuning a 4B decision model on the GM’s rulings recalibrated it (triage AUC 0.90 to 0.96)
but did not teach the conventions. We also report that an apparently strong verdict-prediction
result was an artifact of the classifier reading the upstream model’s own hedging, and that vendor
latency claims did not transfer to the GB10 (127 ms measured vs. 18.4 ms claimed).

The full paper can be found here – https://drive.google.com/file/d/1e0pzEsFrulo_QrLhIaIqhuz4YDujpQS0/view?usp=sharing

A cascade’s savings are set by its first stage: every item the cheap model cannot settle
is paid for twice. Two earlier studies on a tabletop role-playing campaign pipeline evaluated
off-the-shelf non-generative decision models against a generative model [1], and designed a
Decision-2.0-Nox-4B → qwen3.8-flash-next cascade by replaying their frozen outputs, then
checked it live [2]. The first study also fine-tuned Nox on the game master’s (GM’s) rulings but
never put the tuned model in the cascade. Here we do, asking whether a domain-specialized
first stage removes more of the second. Because the frozen dataset caches Nox’s backbone
features, tuning its decision head and replaying the resulting cascade takes about a minute of
CPU time per variant, cross-fitted so that no item’s label reaches its own head or its own gate.
On triage, the tuned head lowered escalation from 33% to 25% and raised the cascade’s predicted
single-stream speed-up from 2.07× to 2.55×, while still missing no more real names than the
generative model. On fact typing the replay first showed the second stage almost vanishing,
then exposed why: every fact’s label is its entity’s label and entities recur, so a plain lookup of
the GM’s earlier rulings scores 99.0%, above every model. Held out by entity, the tuned Nox
alone tied the generative model on entities it had never seen (88.4% vs 88.3%). Holding out
by token on triage broke the missed-name constraint. We then served the tuned heads on the
real two-box system, with gates and predictions pre-registered by hash. Live, the triage cascade
scored +2.9 points above qwen (CI [+1.9, +4.2]; predicted +2.8), escalated 24.7% of tokens,
missed exactly as many names as qwen, and was 2.51× faster single-stream (predicted 2.38×);
the entity-disjoint fact-typing cascade escalated 4.3% and was 4.49× faster (predicted 4.36×) at
+1.1 points (CI [-0.1, +2.3]). The replay routed 100.0% and 99.9% of items as the live system
did. A specialized first stage removes most of the second on fact typing and a quarter of it on
triage, where the second stage stays because the tuned model alone drops twice as many real
names. Set beside Vela 2.0, a full fine-tune of the same Nox for safety and routing, the head-only
gain on our tasks (+5.5 AUC points on triage) is of the same order as Vela’s mid-range gains on
its own, at a small fraction of the effort; Vela itself, tuned for another domain, is a worse first
stage here than the Nox it came from.

The full paper can be found here https://drive.google.com/file/d/1QQD4cFtS4QS3wINR5WNXyV7r2MXidVjB/view?usp=sharing

So my original problem is this: I want campaigns that produce the kinds of moments we get, and to do that, I need to remember what happened and create beats that take time.

My solution was to write summaries, but that took me 3 hours for each 1.5-hour campaign. I ran 6 campaigns, and that led to burnout. LLMs can create summaries from Zoom, but with errors and hallucinations. Better than nothing, but if you need a precise set of events to drive an outcome, they don’t work.

Raw LLMs create shitty summaries. Commercial options created better summaries, but still required manual post-processing. The goal was a summary of the campaign, world state, party state, and planning so I can do better session prep and have more. And there it would have stood.

But my boss said You own the AI technology strategy for Nutanix Infrastructure I said, Okay. My desire + job = OPPORTUNITY.

So here’s where I started: (1) Had a bunch of handwritten summaries; (2) Had Zoom recordings. (3) Had Zoom transcripts; (4) Had purchased summaries via specialized systems.

My first attempt was to turn Zoom recordings into a narrative in one shot.

Failed.

Then I had my biggest aha – using an LLM to decide was the mistake. I switched to using the purchased summaries as the starting point. This was a huge win.

Better, but it still didn’t work.

I realized the problem was that the Zoom transcripts were dirty. So I created a spell-checking system. That seemed easy enough. My first attempt was manual. Then I started using LLMs. nbsp; Challenges: (1) What is ASR garbling; what is Zalthir? (2) How to detect Zalthir? So I built a very effective and pretty complex skill (check out https://lnkd.in/gMnegTBt).

That took a while to get working right. Originally, I had hundreds of things I had to clean up. Now, unless there is a new bit of canon, it takes about 5 minutes to do the manual review. Most of the time, the review is correct enough – in fact, if it weren’t for my OCD, I would let it work automatically.

Once the spell pass was complete, the next step was to fix the summaries. That’s when I had my next aha.

Step 1 – build a consistency checker that compared the previous canon + Zoom transcript to what was in the purchased summary. The skill is staged consistency (same repo)

Then, using that improved summary, I told the LLM to find details the first summary missed and create an enhanced summary. And that would verify the transcript, the summary, and the canon.

Net effect, the system now tracks many more details than ever. That lets me create far more detailed summaries and far more interesting session prep.

TLDR:

Where it started: Start with handwritten or generated summaries, and use LLMs to extract facts and create a new summary from the transcript.

Where it landed: Start with a Zoom transcript, add a second transcript, create a transcript from a Zoom recording, curate canon, curate the summary, and cross-check => much better summaries.