wrong tool

You are finite. Zathras is finite. This is wrong tool.

How Fable made the case for Survivability as a first-order design principle

by Leave a Comment

I wrote about the notion of survivability before – survivability requires portability. If your application is not portable – meaning it can’t use alternative infrastructure then your application is contingent on someone else. And that if you are contingent on someone else, you are already dead.

If I were a European government and a military/political leader, the argument DJT made would be the following: we can destroy any business that depends on US frontier models at a moment’s notice. The decisions can be “because the US government doesn’t like Anthropic,” to “somebody bought Anthropic,” to “whatever crazy decision the US made”

Any critical infrastructure can not depend on a US-based model any more than it could depend on China. Therefore, European AI researchers and governments will start spending beaucoup dollars to build alternatives.

The interesting thing is that open-source models are increasingly good enough. The problem is that even if you have the hardware, the MODEL is what the US government banned.

The only way to guarantee sovereignty is to have a copy of the model weights locally and to use the model without communicating with anyone under the jurisdiction of the US government.

The case for -on-prem-ai- was made by the US government’s actions for any institution that has to serve customers who won’t be very impressed with their vendor saying, ” Well, the reason I can’t serve you in France is that the US government made a decision.

As a Frenchman, my reaction would be – “You are a French company, I am paying money to you, and I want that physical good, and you can’t sell it to me?”

nutanixist 31 : why orchestration is a dead end for agentic infrastructure

by Leave a Comment

I have been spending a lot of time on the next phase of system administration, agentic system administration. How can you use Agentic flows to manage infrastructure?

And Keith Townsend has put together an excellent mental model that I want to focus on. Not to critique it, but to ask: what is required of the infrastructure to provide what he calls the 2C reasoning layer?

My view on LLMs is that they are probabilistic theorem provers. If you can shrink the facts down narrowly enough, the theorem prover will probably know the right answer because the probability of you having a unique enough problem in practice is zero.

Where they fail is when the set of facts is too large, there are too many options, or the facts change mid-execution.

For anyone who has Claude, Codex, or Cursor, imagine having the systems try to update your code while someone else is simultaneously editing it. The results would be ugly.

There are three elements to facts:

  • (1) They must be facts. Every path to the fact must produce the same fact.
  • (2) The semantics of the facts must be encoded.
  • (3) The policies of the facts must be encoded.


A natural, convenient, and reasonable way to get those facts is through an MCP server. But it’s more than just an MCP server that naively transforms the API set. Here are the layers I view as essential.

The semantic layer is useful when I am trying to power off a host. Powering off a host can be a fact, but the claim that the cluster may crash when you power off a host is semantics. And that semantics is deterministic. But if you don’t provide it, you are allowing the probabilistic theorem to infer that, and it may not.

The policy layer is what says, “You can power down a host on Tuesday.”

But both the policy and semantic layers need a deterministic ground-truth layer that provides a single view of the host’s state. And that the host’s state must remain the same regardless of how the API’s state is queried. And that any change in the host state is reflected everywhere.

If it’s not, then the semantic layer can’t know if the host can be shut down, the policy layer doesn’t know if the host is the one that can be shut down, and so the layer above the MCP can only guess, and when the consequences of guessing are a catastrophic layer, it all breaks.

Nutanix’s infrastructure is not an orchestration layer of sharded, incoherent state; instead, it is built on a single global transactional and resilient database that provides local autonomy and global consistency, serving as the foundational layer required by any 2C system.

To make my point, let’s look at Proxmox and OpenShift. Please note that any errors are mine, and if I got this wrong, I would love to correct it.

Let’s first look at OpenShift

This looks good until you see that last layer. The challenge is that, because of how K8S works, until the operation completes successfully, the system remains in an indeterminate state. So, although the kubernetes API says “the system is thus,” or rather the ‘desired state is thus,’ it can not guarantee when it says that the state hasn’t shifted.

Again, is the host down, or is it on its way to being down? The k8s layer doesn’t know. And so the MCP server has to either wait for the k8s layer to find out and then act, or reach around the k8s layer.

Worse, any operator can reverse a decision the MCP server made. So the MCP server is dealing with a state that can change behind its back, and that change can break the decisions that an agent sitting on top of the MCP server made.

From the agent’s perspective -> I saw facts, I made a decision, the decision failed. The problem is that the facts changed while the agent wasn’t looking. So it goes back and asks again, and gets different facts, so it does something different.

The probabilistic theorem prover is looking at a set of facts that are continuously changing

Now let’s look at Proxmox

Again, the MPC server has control planes with overlapping state. It must figure out who to query and when, and orchestrate an operation that is not transactional. So from the agent’s perspective, the fact is malleable. And a malleable fact is destructive for a probabilistic theorem prover.

After the MCP server returns, the actual ground state is unknown.

So the agent is always wondering why the thing didn’t work. And the answer is because the facts changed. But then, the agent has to start again. And then it has to create a plan that can account for every possible change, or it has to give up.

75 architecturalist papers: types are the right answer

by Leave a Comment

I have come here to gloat and observe.

When I was at VMware, a lot of the code written in Python was being used in production. In my mind, that was an awful technical decision because VMware was shipping a packaged appliance. Every CPU cycle we spent on Python was a CPU cycle the customer was spending, essentially imposing a tax on them for our convenience. The issue I had wasn’t just that we were spending it for our convenience, but that we were using a technology that was intrinsically unsafe due to its lack of type safety.

So, I would engage in intense debates and arguments with some of the Python enthusiasts. They would vehemently assert that Python was robust, reliable, and even superior. They would go on to explain that it was easier to code in and prototype with Python.

I couldn’t deny that if you’re trying to build a prototype, Python’s simplicity, speed, and ease of use are undeniable advantages.

But when you finally ship code, you are no longer prototyping. You know what you’re shipping, so you’re no longer experimenting. So that value is zero.

Fast forward a few years, and I end up at Nutanix. I’m looking at LLMs, AI, and generated code, and I’m wondering about my choices in terms of backend languages. I did some research and realized that Python is actually a very good language to use because the token cost is very low since it’s very expressive. Also, for personal projects and the kinds of things I was doing, most of the libraries and tooling that I want, use it.

So, having done my research and having given it some thought, I said, “Okay, Python is the language.” I figured I don’t really have to learn all of the vagaries of Python, and the performance is going to be dominated by the LLM. If I ever get to the point where this matters, I’ll figure it out.

I kind of lived in that happy bubble until I got this project that I downloaded from GitHub, a really great project. (https://github.com/nashsu/llm_wiki) The author was using JavaScript and Vue for the front end, and to my surprise, they were using Rust on the back end.

The back end of their server was entirely written in Rust, and I was like, “Well, that makes no sense.” First, I thought it was a case of an engineer trying to be clever, cute, or experimenting. At the end of the day, it’s an open-source project. They’re not being paid, and if they want to do something in Rust because that’s what they want to do, that totally makes sense.

But then another friend of mine told me that, well, actually, that’s a new emerging pattern. The reason is that using Rust is a much more token-efficient way of writing code once you have product feature lock. In other words, once you’ve shipped version 1, every subsequent iteration costs fewer tokens because of the context Rust gives to the computer and the fact that it catches a whole bunch of bugs as a result of the type safety. So, the net effect is you’re using fewer tokens per iteration afterward.

That made me smile because literally, the AIs figured out exactly what I had figured out 15 years ago, and what Niklaus Wirth figured out 50 years ago. And now people figured it out on their own.

Back then in 2015 when I got in front of all those Python engineers and told them that they would never write another Python service as long as I was architect and they should all go learn Go, I felt kind of bad. I was basically making an assertion with no data. But now I’m happy to say I was right, and yes, I’m going to gloat about it.

But.

It’s kind of cool to see that there are certain facts about software engineering that are not tied to human cognition but to cognition and information in and of itself. That to me is the most interesting thing about this.

74 architecturalist papers: cheap experiments

by Leave a Comment

Notes from moving a D&D distillation off the API and onto two boxes I own.

The run that hung

I was running the whole campaign through a new extraction pipeline. Fifty-six sessions of Out of the Abyss, fanned across two DGX Sparks. Most of it ran fine. Then four units at the tail hung for an hour and forty-four minutes on one box, while the other Spark sat there doing nothing.

I was sure I knew what it was. vLLM degradation, the decode running away toward the token ceiling. So that’s where I went looking.

That wasn’t it. The metrics showed zero preemptions and an almost-empty cache. Nothing was slow. The laptop driving the run had gone to sleep. The Python processes froze with their sockets to vLLM half-open, and when the laptop woke the sockets were dead and the reads behind them never came back.

Nothing had degraded. A read was just hanging forever. That isn’t a slow box and it isn’t an error. You can’t wait it out and you can’t retry it. You put a clock on it.

I’d had a timeout in an old script. A rewrite dropped it. I put it back, added a read timeout on the client, and moved on.

That’s the kind of bug this whole project was made of. The bug isn’t the interesting part. The interesting part is that I could afford to chase it at all.

What I was actually doing

The job hasn’t changed since I last wrote about this tool.

Take 56 campaign sessions and distill them into a world_state.md — a compact doc that says who’s alive, who’s angry, what’s on fire right now. The thing I scan before every session.

The old version did it with the Claude API. Chunk the bible, extract per chapter, one big synthesis call. It worked.

It also costs real money every run. So I ran it once, looked at the output, and lived with what I got.

That last part is the whole problem. I didn’t see it until the meter went away.

How the ensemble actually happened

I, of course, would like to say that I read the research, used my brilliant insight, and designed a five-lens ensemble.

And that would be … well, less than true?

It started with one prompt.

Read a chapter, emit the facts. Six thousand tokens at a time. I ran it. It gave me some facts.

Then I ran it again.

Different facts.

I asked about it and got the standard answer: that’s normal, it’s non-determinism, the model isn’t deterministic, and you live with it. Which is true, and is also the kind of true that makes you stop thinking.

So I didn’t quite accept that. What if I just do it multiple times?

So I ran it twice and took both. And the union was better than either run alone — not because the model got better, but because two samples of a noisy process see more than one. That’s where self-consistency came from.

Then, looking at what I had, I saw that facts were clearly being missed. Things in the text that just weren’t getting emitted. So a sweep was added to get every named fact.

Then I noticed timelines were getting dropped. A whole sequence of events would collapse into a single fact, and the when would vanish.

Which sounds easy to fix until you sit with it. What is a number? What is a timeline? “Three days later” is a fact. “On the eighth day” is a fact. “After the rockfall” is a fact about ordering with no number in it at all. Figuring out what we even meant by temporal took its own round of tests. That’s the temporal lens, and it was not obvious.

Then a subtler one. The facts were interpreting when they should have been recording. The model would read a scene and tell me what it meant. It wouldn’t tell me that the character held another’s hand; it would tell me that the character liked the other. I wanted facts, not meaning.

And now I had too many facts. So time to merge.

Merge showed me something I couldn’t see before. The sweep pass, the one told to be exhaustive, was packing several facts into one. It read like one fact, if you think one fact is a sentence with three facts connected by and statements. So back to the prompt: one fact equals one state-change, with worked wrong-and-right examples. The tell turned out to be the ellipsis. If the model reaches for … to stitch a quote together, it bundled. Split it. That alone took bundling from 15% to 1%.

And only then, with a prompt stack that actually worked, did the real engineering question show up:

How do I run this?

Five prompts, multiple samples each, against two boxes.

So I had five prompts, but how do I run them correctly? The problem is that now any bug in how the prompts are run is going to cost money. Make a bug in your orchestration logic that comes from my pocket. That hurts. That would have stopped me there. I needed orchestration that fans all of it across both Sparks and merges it deterministically and I couldn’t afford a bug, unless, the cost of doing a run was free. And it was, *free*. So I could build the ensemble. And debug it. Like when the laptop hung.

Then I tried it on a few sessions. Then on six. Then, on the whole campaign.

By the end, the union of those passes was as good as the distill script.

This is the price of iteration

Read that sequence back and count the steps.

One prompt. Notice non-determinism. Run it twice. Notice missing facts. Add sweep. Notice dropped timelines. Argue about what a timeline even is. Add temporal. Notice interpretation creeping in. Split recording from inference. Build the merge. Discover the sweep is bundling. Fix the prompt. Then build the ensemble to run everything. Then scale it up in three steps.

Not one of those steps was a plan. Every single one was a reaction to something the previous run did wrong, which I could only see because I’d run the previous one and stared at the output.

That’s the part the meter kills.

Each of those steps is a run. On the API, each run is a charge, and most of them produce a result whose entire value is “huh, that’s wrong in a new way.” A null result, which costs you money. It made you smarter, but it cost you money. You can’t justify “let me run it twice to see if the non-determinism is hiding recall” when the second run costs money and might just confirm the first. So you don’t. You take the first okay-ish output, and you stop.

And if you stop at “run it twice,” you never see the missing facts, so you never add the sweep, so you never hit the bundling problem at merge, so you never write the bundling fix. The chain exists only because each link was cheap enough to follow on a hunch.

The ceiling wasn’t Qwen. Qwen could always do this. The ceiling was how many times I could afford to be wrong on the way to finding out.

That’s the price of iteration. When you’ve already paid it as capex, you can afford to pay it over and over.

What beat what

The local models didn’t win. Qwen did not outwrite Claude. The final synthesis still runs on the big API model because it’s low-volume, cross-entity, quality-critical, and not where I want to save money. And by the time it runs, the input has been reduced from 537K tokens to about 40K, so it’s a cheap call.

Local to explore. API to finish.

The better world_state came from the chain, and the chain only exists because each run was cheap enough to follow on a hunch. The old API result was never the API’s ceiling. A well-iterated API run could probably have matched it. I just couldn’t afford to be wrong that many times in a row, because under a fixed budget, every null result is pure loss, and the chain is mostly null results until the end.

But it’s even worse, because I had already distilled the data, with Anthropic’s API, I couldn’t justify trying to make it better.

A meter doesn’t only cap how many experiments you run. It biases you toward timid ones. It makes the weird, probably-won’t-work shots feel irresponsible.

Owning the box doesn’t make you smarter. It stops punishing you for looking.

Where this doesn’t hold

And here’s where we have to be careful.

Utilization is carrying that word “free.” The power was pennies, but the amortized cost per run is only low if the boxes stay busy. An idle Spark has a high effective cost per token. For an occasional personal pipeline, the money case is thin, and the real reasons are data locality, iteration freedom, and learning.

The engineering is the actual bill. Speculative execution, the timeout, the dead-socket hunt, that’s labor the API path externalizes to Anthropic’s SRE team. Own the serving, and you pay it yourself, in time, and you pay it again every time the stack drifts. It was the highest real cost here, by a lot.

And free experiments erode discipline. A budget imposes a crude “think before you spend” rigor. Remove it, and it’s easy to run 50 variants when 5 would have told you the same thing. Local doesn’t free you from experimental discipline. It just shifts the responsibility for supplying it from the meter to you.

Plenty of organizations are sitting on idle GPUs. The question worth asking isn’t whether to buy hardware. It’s whether there’s a sunk, underused inference asset you can route verifiable bulk work onto that’s reviewable, iterative, bounded, and fine to run in minutes instead of milliseconds.

Aggregation hit all four. It went local. Synthesis hit one. It stayed on the API.

Eight pull requests were what it took to find that line, one cheap experiment at a time. Including the four that did nothing, and the one where I was sure it was vLLM, and it was a laptop taking a nap.

That one was free, too.

The pipeline lives in CampaignGenerator — ensemble_extract.py, synthesise_world_state.py, spell_canon.py, and the per-lens prompts under config/agents/. The robustness work is PRs #64–#71. This is the retrospective, not the code.

Introducing TurbovecDB an alternative to ChromaDB?

by Leave a Comment

Over the past few months, I have been working on a system I call “CampaignGenerator,” and one of the critical questions for that system is: how do you store state and make state retrieval fast?

When I discovered MemPalace, what drew me to it was its simplicity, locality, and elegant architecture.

What drove me bonkers was ChromaDB’s performance and stability.

Along the way, I got a DGX Spark, and that got me thinking about TurboQuant. And how that might make my DGX Spark’s more useful.

And as I kept looking around, I discovered a project called Turbovec And that then made me go, hmm.

Could Turbovec replace the HNSW system that ChromaDB used, which made it so slow? And could it do so without breaking any of MemPalace’s guarantees?

So Claude and I started experimenting. And the results were promising and exciting.

But Turbovec is only part of ChromaDB. To have a full database, you need a stable store.

So what the heck, I decided to create a database, TurbovecDB. That was actually an experiment in using Claude.

I asked Claude to review what MemPalace required and determine how to build a database that would meet its constraints.

Turns out that the requirements of MemPalace are reasonably straightforward, and so Claude determined that the solution was to build an API layer on top of SQLite, use Turbovec instead of HNSW, and voila, turbovecdb v0.1

I am not going to argue that what I built is a masterpiece of software engineering; I am going to note that this was relatively straightforward and shows the power of interfaces.




AI and programmers learning again what Wirth figured out in 1974, types and structure mattters.

by Leave a Comment

The thing about trying to turn TTRPG content into something that can be used in a game is that TTRPG content is a book of art.

Most folks who don’t play TTRPGs don’t realize that the content isn’t a set of tables but rather a piece of art. Everything from the cover, the art, the font, and the language makes or breaks the TTRPG content.

When using it as part of a system for creating campaign content, different parts need to be handled separately.

Want to change a “one-shot” to fit into a campaign? The story itself is the first part you have to change.

Want to make it harder? Then you just need the mechanical details of the monsters.

General-purpose RAG doesn’t work. And shoving a 120 MB PDF to get 10KB is a great way to waste time and make Anthropic’s shareholders and Nvidia’s shareholders wealthier.

So the first step is to transform the PDF into a structured document that can be used to make subset queries.

And because TTRPG is no different from any other discipline, it turns out this problem is why SAP and others have moved away from pure RAG toward creating structured documents that the Agent can interact with.

As a computer scientist, what makes me laugh? The original promise of LLMs was that it could read unstructured data, infer structure and make decisions.

And here we are discovering – yet again – that computers are great at operating on structure, not on systems with no structure.

Nikolas Wirth would be laughing.

AI Can Write the Code, But as Every Evil Overlord Knows, Every Plan Needs a Five-Year-Old to Vet It

by Leave a Comment

Having observed how AI accelerates code development and simplifies the discovery of design and architecture, the next bottleneck is review.

The challenge with review is that it is ultimately a judgment call, and that improves over time. When an organization chooses to remove junior engineers because it sees only senior engineers as capable of conducting reviews, it is actually lowering its productivity threshold. The challenge you faced with reviews is that they are ultimately a human-biased artifact

Human beings bring their experience and their whole personalities into the review. Having a mix of senior and junior people from diverse backgrounds increases the likelihood that reviews catch defects. This is not novel, new science. It goes to pretty much every organizational principle since the hundred things I would do if I were the evil overlord list.

The list says if I were an evil overlord, I would have a five-year-old on staff whose only job was to tell me if a plan was silly and then fix it.


When you eliminate that diversity your effectively reducing the scope of your ability to detect problems Not the easy problems, but the really hard problems Furthermore, if your group is not diverse in its experiences, what will happen is that it will become less capable of tolerating dissent So although initially, you will get a productivity win over the long-haul, you will suffer from the collapse of your ability to detect issues.

As an industry, we know this very well, so when I see organizations that decide to go all in on a pattern where they retain a small number of similar people to do reviews and keep only them on the team, I see a team in danger of model collapse. They will have a short-term efficiency gain and a long-term failure as the Blindspot magnifies

I have seen this pattern at three different jobs where a senior technical team had a very homogeneous background and missed obvious trends because they saw them as alien to their experience In fact, it was one of those jobs that led me to conclude that the best thing a company can do is periodically, purge, their senior technical ranks, so as to ensure a fresh flow of insight, if the company is not growing sufficiently so that a large new body of technical leaders can be promoted to the highest levels

73 architecturalist papers: survivability – a framework for software and culture

by 1 Comment

Working Thesis

Software encodes culture. Culture increasingly depends on software. Therefore, the survivability of software systems determines the survivability of the societies that rely on them.

Origin of the Idea

I did not arrive at this concept through academic study.

I was exposed to cultures that had already survived:

  • through family
  • through relationships
  • through lived experience

These were systems that:

  • persisted across disruption
  • adapted without disappearing
  • survived even when institutions failed

At the same time, my professional life was focused on:

  • backup
  • disaster recovery (DR)
  • business continuity (BC)

At some point, the connection became unavoidable:

The same principles used to keep software systems alive are the principles that allowed cultures to survive.

Once seen, it cannot be unseen.

The Core Question

What allows a system—cultural or technical—to persist over time, especially under disruption?

Not:

  • correctness
  • performance
  • reliability

But:

Survivability

Definition of Survivability

A system is survivable if it can be re-created, adapted, and continue functioning after its assumptions about the environment break.

The Survivability Framework

A system survives only if it satisfies four properties:

1. Portability — Can it move without breaking?

Definition:

The ability to operate across environments without fundamental redesign.

Key Idea:

If it cannot move, it cannot survive change.

Failure Mode:

  • vendor lock-in
  • environment-specific assumptions
  • tight coupling to platform

Diagnostic Question:

How long would it take to run this somewhere else?

2. Redundancy — Can it be re-instantiated independently?

Definition:

Multiple independent ways to recreate and operate the system.

Key Idea:

Copies are not enough—independent capability is required.

Failure Mode:

  • single team knowledge
  • “hero engineers”
  • backups that cannot be restored

Diagnostic Question:

If the original team disappeared, could this system be rebuilt?

3. Sovereign Substrate — What do you control?

Definition:

The degree to which a system controls its environment.

Key Idea:

You cannot rely on what you do not control.

Failure Mode:

  • external pricing/licensing control
  • platform dependency with no exit path
  • reliance on opaque infrastructure

Diagnostic Question:

What parts of this system could be taken away from us?

4. Identity Preservation — Can it change without dissolving?

Definition:

The ability to evolve while maintaining coherence and purpose.

Key Idea:

Too rigid → extinction

Too fluid → loss of identity

Failure Mode:

  • breaking changes
  • fragmentation
  • drift without shared understanding

Diagnostic Question:

What must remain true for this to still be the same system?

The Hidden Layer: Assumptions

Every system encodes assumptions about its environment.

Survivability depends on how wrong those assumptions can be before the system collapses.

Portability Without Assimilation

Core Idea

A system must be able to move into a new environment without becoming indistinguishable from it.

The Trade-Off

StrategyBenefitCost
AssimilationMaximum local optimizationLoss of independence
PortabilitySurvivability, flexibilityFriction, constraints

Key Insight

The more perfectly a system fits its environment, the less likely it is to survive a change in that environment.

Important Clarification

Portability is not:

  • avoiding all platform features
  • building lowest-common-denominator systems

Portability is:

  • controlled integration
  • clear boundaries
  • replaceable dependencies

You can use the environment—you just cannot become it.

The Cost of Portability

Portability introduces:

  • friction
  • constraints
  • reduced short-term optimization

This is real.

But:

Portability trades short-term efficiency for long-term survivability.

Refined Principle

Assimilation optimizes for stability. Portability prepares for change.

Cultural Survivability (Observed Patterns)

From lived experience and exposure:

Systems survived when they were:

  • carried by people
  • replicated across many independent actors
  • able to move across environments
  • able to change without losing identity

They did not survive because they were:

  • stable
  • protected
  • centrally controlled

Critical Insight

A culture written down is not alive. It survives only when it is continuously executed by people.

Mapping Culture → Software

Cultural PropertySoftware Equivalent
Carried by peoplePortability
Many independent communitiesRedundancy
Spaces of self-governanceSovereign substrate
Continuity through changeIdentity preservation

Backup, DR, and Culture

BDR ConceptCultural Equivalent
BackupDistributed memory (people, practices)
Disaster RecoveryAbility to re-establish in new environments
Business ContinuityMaintaining function during disruption
Infrastructure ControlCultural autonomy

Key Realization

Backup and disaster recovery are not just technical practices—they are survival strategies for any system that must persist.

Modern Risk: Software as Cultural Substrate

Today, society depends on software for:

  • identity
  • finance
  • communication
  • knowledge

Which means:

We are encoding culture into systems that may not be survivable.

The Risk

Many modern systems:

  • are not portable
  • lack redundancy
  • do not control their substrate

Therefore:

We are building cultural systems that cannot survive disruption.

Survivability Debt

Every time you sacrifice portability, redundancy, or control for convenience, you incur survivability debt.

This debt is only paid when:

  • the environment changes
  • assumptions break

The Engineer’s Blind Spot

Most engineers optimize for:

  • performance
  • integration
  • correctness

But rarely ask:

Will this system survive if the environment changes?

The Survivability Test

Evaluate any system:

  1. Can it move?
  2. Can it be rebuilt independently?
  3. Does it control its environment?
  4. Can it evolve without losing identity?

If any answer is “no”:

The system is fragile—even if it appears reliable.

Core Principles

  • Survivability > Reliability
  • Optionality > Optimization
  • Independence > Convenience
  • Execution > Preservation

Final Insight

Systems do not survive because they are correct. They survive because they can adapt, be carried, and be re-created across changing environments.

One-Line Summary

If culture runs on software, then software survivability becomes a prerequisite for cultural survival.

Closing Perspective

I did not learn about survivability from systems that worked.

I learned it from seeing what remained when systems failed.

And now the question is:

Are we building systems that will remain?

Next Steps (Optional Expansion)

  • Case Study: Vendor lock-in and infrastructure dependency
  • Case Study: Long-lived cultural systems
  • Practical: Survivability audit checklist for engineers
  • Patterns: Designing for portability without losing capability

If you want, I can next:

  • convert this into a PDF or nicely formatted essay,
  • expand each section into full chapters, or
  • build a practical engineering checklist/toolkit from this.

the nutanixist 31: why portability matters as much as backup and dr for business continuity: lessons from the broadcom acquisition of VMware

by Leave a Comment

red white and blue abstract painting
Photo by Jazmin Quaynor on Unsplash

People often talk about resilience in software as if backup and DR are enough. What the Broadcom acquisition of VMware made me understand is that it isn’t.

Software survivability depends on three things working together: portability, physical ownership, and backup. Without all three, you are trapped. And once you are trapped, you are vulnerable.

Software is just text. Like human culture, it is inert without a physical substrate to store, execute, and restore it. That means survivability is not just a question of whether your code exists somewhere. It is a question of whether you can actually bring it back to life under adverse conditions.

To understand this, it helps to break any software system into two parts:

  1. Infrastructure
  2. Application or business logic

An application cannot run without infrastructure. But infrastructure is itself software, with its own dependencies, constraints, and failure modes. So survivability must be evaluated across the entire stack, not just at the application layer.

1. Backup Only Matters If You Can Restore It Yourself

A backup is only real if it can be restored without relying on the original vendor.

If recovery depends on the original authors of the code, the original cloud provider, or the original platform vendor, then you do not truly control your backup. You are renting recoverability, not owning it.

This matters most in precisely the situations where recovery is most important. If the vendor cannot or will not support you, then your backup is functionally useless. If the vendor’s recovery capacity is constrained, you should assume you are a low priority. During a black swan event, many customers may need restoration simultaneously. That is exactly when centralized recovery support breaks down.

If your restore plan assumes the vendor will be available, willing, and sufficiently staffed under crisis conditions, then it is not a restore plan. It is hope.

Furthermore, if your plan assumes that any relationship you have with a vendor cannot be disrupted because of events in the world, that is not a plan; that is a hope.

2. You Need Ownership of Physical Infrastructure

Even if you can restore independently, you still need somewhere to restore to.

If you do not own physical infrastructure, you have no guarantee there will be a place to run your system when you need it most. And this is not just about owning some hardware in the abstract. You need the right infrastructure, with the right capacity, available on the right timeline.

A recovery plan that depends on finding infrastructure during a crisis is not much of a recovery plan, either.

But ownership and restoration still are not enough.

Physical infrastructure can be destroyed. It can be denied. It can become unreachable. That can happen through deliberate action, geopolitical conflict, sabotage, or natural disaster. If the physical substrate disappears, your software, however well-designed, becomes unusable.

This is why survivability cannot stop at backup and infrastructure alone. There has to be another layer of freedom.

3. Portability Is What Prevents Lock-In From Becoming Captivity

Every software system contains two layers: infrastructure and business logic. Application portability is the ability to move that business logic across infrastructures.

This does not mean portability is free. It is not transparent, and it is never without cost. But it must be possible.

That distinction matters. A system does not need to move instantly or effortlessly. It does need to be capable of moving.

Without portability, your application is fused to a specific infrastructure stack. And once that happens, your bargaining power disappears. Pricing changes, policy changes, licensing changes, geopolitical shifts, or vendor instability all become existential problems rather than procurement problems.

Portability is what turns dependence into choice.

It does not happen accidentally. It requires discipline, architectural restraint, and a willingness to forgo some short-term convenience to preserve long-term freedom.

A Case Study in the Cost of Missing Portability

As the vCenter architect from 2015 to 2023, I believed VMware had a sacred trust with its customers. VMware’s attitude was that we did not lose customers. We worked hard to keep them.

That mindset led many customers to assume they did not need portability. They had backups. They had physical infrastructure. What they lacked was a portable application layer.

The hidden assumption was that the business relationship itself was fixed.

It was not.

The relationship existed only as long as capital markets agreed with it. Once the markets decided that the existing customer relationship no longer aligned with their interests, the relationship changed. And when that happened, the absence of portability became painfully visible.

Customers suddenly found themselves with no real negotiating leverage. Prices rose, and many had little choice but to pay. Porting away was either prohibitively difficult or practically impossible.

Why?

Because portability had never been built. No portable layer existed. And portability does not emerge by accident. If you do not explicitly design for it, it will not be there when you need it.

The Real Test of Survivability

A survivable software system is not one that merely runs well under normal conditions. It is one that can survive broken relationships, failed vendors, destroyed infrastructure, and changed incentives.

That requires three things:

  1. Backup you can restore yourself
  2. Ownership of physical infrastructure sufficient to restore onto
  3. Portability of the application layer across infrastructures

Remove any one of these, and your resilience is incomplete.

Backup without self-service restore is dependency.
Infrastructure without portability is entrapment.
Portability without infrastructure is theory.

Survivability begins when all three are present at once.

Rich Plots, Real Improvisation

by Leave a Comment

What Alice Saw

I shared a tool in a Discord channel the other day, and a new DM named Alice messaged me back:

“I’m gonna be so honest. This is Greek to me. I have no clue what you’re showing or what problem it solves. I’m a new DM.”

Fair

So I sent her a Google Doc I use to prep my Out of the Abyss campaign. Here’s a short snippet.

Score NameNPC/FactionCurrent ValueNext ThresholdWhat Triggers Next
Zuggtmoy’s WeddingZuggtmoy / Neverlight GroveElevated — 2 increases (Blingdenstone expansion + Basidia’s evacuation removed internal resistance)Wedding completion / Araumycos unionContinued party absence from Underdark; no faction opposes Zuggtmoy; fungal spread reaches critical mass
Juiblex RebirthJuiblexLow-Moderate — 1 increase (declared intent to consume Zuggtmoy’s domain)Juiblex manifests a new physical form or begins attacking Zuggtmoy’s territoryZuggtmoy’s wedding weakens her defenses; time passes without intervention


A minute later she replied:

“Ooooooooh. It’s for your campaign story. I thought it was a software thing, not creative.”

She was right. I’d explained the machinery before I’d explained the use.

So here’s the useful version:

I use AI to help me maintain campaign canon across long-running games, but I do not let it decide what counts as canon.

That distinction is the whole system.

What Alice saw was a planning document. At the top were four campaign clocks: four villain plots advancing in the background, each one changed by something the party had done, or failed to do, at the table.

Under that were faction states, NPC dossiers, and plot notes: who knows what, who wants what, what the party has learned, and what is changing offscreen.

I scan that document before every session. In a minute or two, I know what moved, why it moved, and what pressure is building in the world if the party does nothing.

I can’t run the campaign I want to run without this doc.

I also can’t write it by hand. Not across two campaigns. Not across a year of sessions. Not with dozens of named NPCs, each dragging their own history behind them.

This essay is about how I got it written anyway.

What Kind of Game I Want

I want a particular kind of game.

I want players making strange, committed, character-driven choices that I could not have predicted in advance.

That’s not abstract for me. I played Baldur’s Gate 3 cold, no guide, no walkthrough, and made a series of choices as Shadowheart that apparently almost nobody makes. Not because I was optimizing for rarity, but because they made sense for the version of her I was playing.

That’s what I want at the table: not correct choices, but real ones.

I run two D&D campaigns: a heavily modified Out of the Abyss campaign for the Ember Vanguard, and a Dragon of Icespire Peak / Lost Mine hybrid set in Phandalin. I actively push both parties to go somewhere I didn’t plan for. If the plot I prepped isn’t the plot they want, that’s fine.

That’s the deal.

Pick any two

The problem is that I want three things at once.

First: deep prep. Texture. NPCs with interiors. Villains whose behavior today is a consequence of something they chose eight sessions ago. Plots that keep moving, whether the party is watching or not.

Second: flexibility. When the party walks past the dungeon I built, something has to be where they actually went, and it has to feel like it was always going to be there.

Third: consistency. The villain I run in session 24 has to behave like the villain I ran in session 6. If I forget what Shal already said, did, or knew, the illusion cracks.

For a long time, that combination felt impossible.

Prep deep, and the moment players deviate, you’re improvising on top of prep that no longer applies. Prep loose, and the world gets thin. Try to prep every branch, and you end up burning the time you were trying to save.

And over a long campaign, the hardest problem is quieter: you lose track of what actually happened. The next scene drifts a little. Then a little more. Nobody stops the game to point out the contradiction. The fiction just gets lighter.

That’s the part people don’t say out loud. When the party walks past four hours of prep, I’m not upset because they missed it. I’m upset because I burned four hours on something that no longer matters. That lost time becomes thinner prep for the next session, then less energy in the session after that. Players feel it too. They become less willing to push into unplanned territory if they can sense I’m paying for it.

Before LLMs, I had more or less given up on getting all three.

What I Tried First

I wrote summaries from memory. That works for one campaign, maybe. It breaks fast when there’s too much to hold.

Then I tried using an LLM to write the summaries. Better than nothing, but imprecise in ways I didn’t always catch, and the errors showed up later, when they were harder to spot and more expensive to fix.

Then I found GMAssistant.app. That was a real improvement. It gave me solid summaries of what happened. But a D&D session isn’t just a sequence of actions. It’s dialogue, tension, implication, half-finished intentions, weird emotional turns. The recap could get the action right while still losing the feel of the session.

So I went further. I combined GMAssistant recaps with verbatim VTT transcripts from our Zoom calls. Then I built tooling around that. Then more tooling around the tooling. Six weeks, maybe two months, of real work.

I thought I was solving the record problem.

I was. Partly.

The Failure Mode

What I didn’t realize until it nearly cost me a scene was that I was also building a new kind of failure.

A few months into Out of the Abyss, I was designing the endgame around an earlier scene. A PC named Daz had come across evidence implicating a major NPC. My planned encounter assumed he had taken that evidence with him.

I checked the LLM-generated recap to confirm. It said Daz had discovered the evidence. Good enough, I thought, and I kept writing.

Then, by accident, I re-read the original session summary.

What had actually happened was narrower. Daz had looked at some unusual books on a shelf and left the room. He hadn’t opened them. He hadn’t taken them.

Noticed had become discovered in the paraphrase. And discovered had quietly shaded, in my head, into obtained.

If I had run the encounter as written, I would have retconned my own campaign.

That’s the failure mode.

The model hadn’t lied. It had paraphrased. But the paraphrase was fluent. It read like canon. It read so much like canon that I stopped checking the source.

And once that paraphrase enters the next stage of the pipeline, it hardens. A summary becomes a dossier entry. A dossier entry becomes a threat score. A threat score shapes the next session. Small errors do not stay small.

My first version of the tool had bought me depth and flexibility at the cost of consistency, and it had done it invisibly.

The obvious fix would be to go back to the source every time. But if I have to re-read everything every time, I don’t need the tool.

What Actually Worked

What actually worked was putting myself back in the middle.

Now I use the same loop at every layer.

First, the model reads what I can’t read quickly: a transcript, a stack of summaries, a year of sessions. It gives me candidate structure: NPC lists, draft dossiers, scene candidates, recap material.

Second, I review that structure. I fix names. I merge duplicates. I cut paraphrases that slipped into invention. I restore what matters and remove what doesn’t.

This step is not optional. This step is the work.

Third, the model takes the reviewed structure and renders it as prose: a dossier, a narrative recap, a planning document, or a threat tracker.

The model is strong at the first and third steps. It is unreliable at the second. Scope, attribution, ordering, what counts as canon, what matters dramatically: those are creative decisions. Those stay with me.

Skip that middle step, and the errors compound. Keep it, and the whole loop holds.

That’s the system.

What I Have Now

The threat tracker Alice saw is the direct output of it. Session material gets extracted into per-NPC dossiers. I review and reconcile them. Then the tool synthesizes the planning document from that reviewed canon.

That’s how I get villain clocks that stay consistent across a year of play. It’s how I get session recaps that read like narrative chapters without drifting into invention. It’s how I get pre-session cheat sheets I can trust.

It’s also how I keep two campaigns inside one world without the whole thing collapsing under its own weight.

What I have now is not magic. It’s just finally the right division of labor.

I have a threat tracker that tells me, before every session, which villain plans have moved and what moved them.

I have session recaps that read like narrative chapters, but only after I verify the details that matter.

I have NPC dossiers where Captain Tolubb is one NPC, not three duplicate spellings pretending to be different people.

I have two campaigns sharing one coherent world across a year of play.

And most importantly, I no longer mind when the party walks past the dungeon.

Because the prep I’m doing now is not the kind that gets wasted.

If You Want This Too

If you want this for your own game, the practical lesson is simple:

Use AI to read, sort, summarize, and draft.

Do not use it to silently decide what is true.

That part stays with you.

The tools are free. They’re also crude. I’m one GM iterating on my own campaigns, not a product team, and the learning curve is not zero.

I don’t make money on any of this. It’s a hobby. My goal is simply for more of us to have better tools, and for the people trying to make a living doing this to actually make a living.

So if this sparks something for you, build the version you want. If something I built is useful, take it. Fork it. Break it. Send it to a friend who runs games.

I would rather more of us had working tools than fewer.

The last essay was for people interested in the machinery: the loop, the trust layers between documents, and the searchable index I use to query reviewed content mid-session.

This one was for Alice.