People often talk about resilience in software as if backup and DR are enough. What the Broadcom acquisition of VMware made me understand is that it isn’t.

Software survivability depends on three things working together: portability, physical ownership, and backup. Without all three, you are trapped. And once you are trapped, you are vulnerable.

Software is just text. Like human culture, it is inert without a physical substrate to store, execute, and restore it. That means survivability is not just a question of whether your code exists somewhere. It is a question of whether you can actually bring it back to life under adverse conditions.

To understand this, it helps to break any software system into two parts:

1. Infrastructure
2. Application or business logic

An application cannot run without infrastructure. But infrastructure is itself software, with its own dependencies, constraints, and failure modes. So survivability must be evaluated across the entire stack, not just at the application layer.

1. Backup Only Matters If You Can Restore It Yourself

A backup is only real if it can be restored without relying on the original vendor.

If recovery depends on the original authors of the code, the original cloud provider, or the original platform vendor, then you do not truly control your backup. You are renting recoverability, not owning it.

This matters most in precisely the situations where recovery is most important. If the vendor cannot or will not support you, then your backup is functionally useless. If the vendor’s recovery capacity is constrained, you should assume you are a low priority. During a black swan event, many customers may need restoration simultaneously. That is exactly when centralized recovery support breaks down.

If your restore plan assumes the vendor will be available, willing, and sufficiently staffed under crisis conditions, then it is not a restore plan. It is hope.

Furthermore, if your plan assumes that any relationship you have with a vendor cannot be disrupted because of events in the world, that is not a plan; that is a hope.

2. You Need Ownership of Physical Infrastructure

Even if you can restore independently, you still need somewhere to restore to.

If you do not own physical infrastructure, you have no guarantee there will be a place to run your system when you need it most. And this is not just about owning some hardware in the abstract. You need the right infrastructure, with the right capacity, available on the right timeline.

A recovery plan that depends on finding infrastructure during a crisis is not much of a recovery plan, either.

But ownership and restoration still are not enough.

Physical infrastructure can be destroyed. It can be denied. It can become unreachable. That can happen through deliberate action, geopolitical conflict, sabotage, or natural disaster. If the physical substrate disappears, your software, however well-designed, becomes unusable.

This is why survivability cannot stop at backup and infrastructure alone. There has to be another layer of freedom.

3. Portability Is What Prevents Lock-In From Becoming Captivity

Every software system contains two layers: infrastructure and business logic. Application portability is the ability to move that business logic across infrastructures.

This does not mean portability is free. It is not transparent, and it is never without cost. But it must be possible.

That distinction matters. A system does not need to move instantly or effortlessly. It does need to be capable of moving.

Without portability, your application is fused to a specific infrastructure stack. And once that happens, your bargaining power disappears. Pricing changes, policy changes, licensing changes, geopolitical shifts, or vendor instability all become existential problems rather than procurement problems.

Portability is what turns dependence into choice.

It does not happen accidentally. It requires discipline, architectural restraint, and a willingness to forgo some short-term convenience to preserve long-term freedom.

A Case Study in the Cost of Missing Portability

As the vCenter architect from 2015 to 2023, I believed VMware had a sacred trust with its customers. VMware’s attitude was that we did not lose customers. We worked hard to keep them.

That mindset led many customers to assume they did not need portability. They had backups. They had physical infrastructure. What they lacked was a portable application layer.

The hidden assumption was that the business relationship itself was fixed.

It was not.

The relationship existed only as long as capital markets agreed with it. Once the markets decided that the existing customer relationship no longer aligned with their interests, the relationship changed. And when that happened, the absence of portability became painfully visible.

Customers suddenly found themselves with no real negotiating leverage. Prices rose, and many had little choice but to pay. Porting away was either prohibitively difficult or practically impossible.

Why?

Because portability had never been built. No portable layer existed. And portability does not emerge by accident. If you do not explicitly design for it, it will not be there when you need it.

The Real Test of Survivability

A survivable software system is not one that merely runs well under normal conditions. It is one that can survive broken relationships, failed vendors, destroyed infrastructure, and changed incentives.

That requires three things:

1. Backup you can restore yourself
2. Ownership of physical infrastructure sufficient to restore onto
3. Portability of the application layer across infrastructures

Remove any one of these, and your resilience is incomplete.

Backup without self-service restore is dependency.
Infrastructure without portability is entrapment.
Portability without infrastructure is theory.

Survivability begins when all three are present at once.

Notes from running D&D with LLMs.

The trilemma

I run two D&D campaigns. Out of the Abyss, heavily modified. And a Dragon of Icespire Peak / Lost Mine hybrid set in Phandalin.

Between them: hundreds of pages of session summaries, dozens of NPCs, two plots branching for over a year.

Every GM I know wants three things at once.

A campaign that’s deeply prepped — with texture, with NPCs who have agendas, with plots that advance under their own momentum whether the party notices or not.

A campaign that’s flexible — that bends when players go somewhere you didn’t plan for. Which is always. Players are the point.

A campaign that’s consistent — where session 24 honors session 6, where NPCs don’t change their minds between games unless something actually happened, where nobody at the table says “wait, didn’t we already kill that guy?”

Prepped. Flexible. Consistent.

Pick two.

That’s the GM’s trilemma. Prep deep and you’re brittle — the party walks past your dungeon and you’re either railroading or improvising on top of prep that no longer applies. Prep loose and the world feels thin. Compromise and you get both.

This essay is about the tooling I built to stop picking two.

Not a product pitch. Working notes from a GM on what broke, what worked, and the pattern that ended up holding three things together.

---

Why this used to be impossible

Before LLMs, the leg that went first was consistency.

Prep depth is achievable. It just costs hours per session. Flexibility within deep prep is achievable too — deep prep gives you material to pivot to.

What isn’t achievable, not by anyone I’ve ever met, is holding the full state of a long-running campaign in your head.

A campaign with eighteen sessions and forty named NPCs is a body of information larger than memory. You remember what stuck. You forget the cleric from session 11. You half-remember the stone giant’s oath and improvise around the gap. Two sessions later, you’ve accidentally rewritten what he said.

It’s not a failure of effort. It’s bandwidth.

The standard responses are well-worn. Run shallower campaigns so the state fits in your head. Run published modules and let the book do the remembering. Keep a wiki you don’t have time to update. Keep a journal you don’t have time to re-read.

All of these work, to a point. None of them break the trilemma. They just lower the ceiling so the trilemma bites less.

I wanted the ceiling back.

---

Where LLMs break

LLMs are, on their face, the missing piece. They can read everything I can’t.

My first version of the pipeline was exactly that. Summaries in, prep out. The LLM reads everything. I read its output. Done.

It wasn’t done.

A few months into Out of the Abyss, I was designing an endgame encounter that leaned on an earlier scene. A PC named Daz had come across evidence implicating a major NPC. The encounter I was drafting assumed he’d taken the evidence with him.

I checked the LLM-generated recap. It said: Daz discovered the evidence. Good. I finished the encounter.

Then, by accident, I re-read the original session summary. What actually happened: Daz saw the books on a shelf, noted they looked unusual, and left them there. He hadn’t taken them. He hadn’t opened them.

Noticed had become discovered. Discovered had, in my head, shaded into obtained.

The encounter I’d built was internally consistent. If I’d run it, it would have been a retcon. Players notice retcons. Some small fraction of the fiction’s weight leaks out.

The interesting part is the mechanism.

The LLM hadn’t lied. It had paraphrased. And the output was fluent. It read like canon. It read like canon so smoothly that I stopped going back to check.

That’s the failure mode. The LLM’s output doesn’t announce that it’s a paraphrase. It reads like a record.

If I ask the LLM to also do the next step — generate the encounter from the paraphrased recap — it will happily do so. The paraphrase hardens into a scene with dialogue and stakes, pointed at my players. Errors don’t stay small. They compound.

So the naive version traded away consistency to gain depth and flexibility. And the trade was invisible.

The obvious fix — verify every output by hand — erases the value. If I re-read the source every time, I don’t need the LLM.

The real question was different. Could I structure the pipeline so the LLM does only what it’s reliable at — rendering verified structure — and a human review step sits at every point where precision matters?

That’s what the rest of this is about.

---

The extraction layer I don’t own

Before any of my tooling runs, something has to turn the raw session — hours of unstructured speech on a Zoom VTT — into usable signal.

I don’t do that part. Other people have built good tools for it. GMAssistant.app. Saga20. A handful of others in the AI-assisted D&D space.

My own pipeline leans on GMAssistant directly. The session doc generator uses its recap as an authoritative anchor for scene extraction. Without it, extraction is an unguided scan across the transcript, and unguided scans miss things.

The ecosystem matters. Crediting it matters. The tools I built sit on top of a layer other people built.

That said: the output of the VTT extraction layer is still an LLM extraction. Fluent. Plausible. Sometimes wrong in ways you won’t catch without reading the source.

The review beat applies to their output too. Everything downstream of the VTT passes through my gate before anything else runs.

---

The loop

The pattern I ended up with wasn’t elegant the first time. I didn’t arrive at it by principle. I arrived at it by iterating on two live campaigns and paying attention to where things kept breaking.

The shape my iteration settled into is three beats.

Extract. An LLM reads something I can’t read fast — a transcript, a session summary, a stack of extractions — and returns candidate structure.

Review. I read the candidate structure. I fix it. I merge it. I throw out what’s wrong. I add what’s missing. This is not optional. This is the thing.

Render. An LLM takes the reviewed structure and produces readable prose — a narrative recap, an NPC dossier, an encounter document. It’s rendering inside a structure I’ve verified.

That’s it.

The LLM is strong at beats one and three. It’s unreliable at beat two — scope, ordering, attribution, what counts as canon. Those are precision decisions. Those are mine.

Skip the review beat and errors compound. The first LLM’s paraphrase becomes the second LLM’s input. Two LLMs downstream, the original detail is unrecognizable. Nobody notices because the output is fluent.

Keep the review beat and the loop holds. The LLM does what it’s good at. I do what I’m good at. The content that comes out has actually been seen by a human.

I now run this loop at every layer of my pipeline. There are four seams where it lives. They’re worth walking through.

---

Seam one: NPC dossiers

The party meets an NPC in session 4. They reference him again in session 9 with a slight typo on the name. In session 15, someone else mentions a character who may or may not be the same person.

Without help, I now have three fragments of one NPC and no single view.

The script I use — planning.py –build-dossiers — extracts per-NPC information from every summary. Each dossier lives in its own file. Extract.

Then I read them. And half the time I find duplicates under different names. Captain Tolubb. Cap. Tolubb. Tolubb. Three files, one NPC.

I merge them. Pick the canonical file. Fold the names into an aliases: block at the top. Reconcile any content differences by hand. Delete the losers. Review.

When I run planning.py –synthesize on the reviewed dossiers, the synthesizer uses the aliases to rewrite every occurrence of “Cap. Tolubb” in the raw extracts to “Tolubb” before the LLM sees the text. The final planning doc treats him as one NPC. Render.

Skip the review beat — the merge, the alias recording — and synthesis treats every variant as a distinct NPC. You get the fragmented result back, now laundered into a clean-looking document. Exactly what the merge was supposed to fix.

The script can extract. I have to review. The script can render.

Three beats.

---

Seam two: session recaps

After a session, I have a VTT transcript, a GMassist-style recap, and a few hours of memory.

The session doc generator runs passes one through four — consistency check against campaign state, enhancement of structured sections, narrative plan, per-scene character extraction. Extract.

Then I stop the pipeline. Each scene’s extraction is written to a file. I open them in an editor. I read them against the VTT source. I add dialogue the extractor missed. I cut lines the extractor invented. I adjust emphasis when it got the emotional beat wrong. Review.

When I’m satisfied, I run pass five — narration — from the reviewed extractions. One character, one scene at a time. Render.

What comes out is a session document that reads like a novel chapter. First-person per character. Style-matched to their voice. Dialogue that was actually said, because I verified it in beat two.

If I skipped the review beat, the narration would be fluent and wrong in ways I couldn’t see without checking the VTT. Same failure mode as the Daz scene, one layer deeper.

---

Seam three: grounding docs

The bible of my campaign is the accumulated session summaries. Too long to read every time. Too important to not use.

distill.py and campaign_state.py extract from the bible. World state. NPC states. Completed quests. Open threads. Extract.

I read the generated world_state.md and campaign_state.md. I fix the things that are wrong. I add the things the extractor missed. I cut the things that are no longer true. Review.

Those reviewed docs are now the grounding context for every downstream prep script. prep.py reads them first. Everything I generate is rendered against a world state I’ve verified. Render.

This is the least visible seam. It’s also the one that makes all the others work. Prep generated from bad grounding is bad prep, no matter how good the prep prompt is.

---

Seam four: cross-campaign canon

I don’t just run two campaigns in parallel. I run two campaigns that share a world.

Group 2’s actions become Group 1’s history. Group 1’s consequences become Group 2’s present. This is fun and also a lot to keep straight.

I keep a notes/canon/ directory. Cross-campaign events go there when they happen. The party in one campaign did X; here’s what the other campaign’s world now has to account for. Extract. (Often by hand, not by LLM.)

I review the notes before they touch either campaign’s grounding docs. Often I’m the only person with enough context to know what an event actually means across the shared world. Review.

When I’m sure, I promote the relevant facts into the appropriate campaign’s world_state.md and NPC dossiers. Render.

notes/ is staging. Neither campaign’s palace indexes it directly. The canon gate runs across both worlds.

---

The query that kept failing

The loop gives me clean content. Reviewed dossiers. Verified grounding docs. Session recaps I trust.

Clean content in files is also passive content.

Four months into running OotA, mid-session, the party doubled back through a village they’d passed through five sessions earlier. Someone mentioned that bartender we met. I had six seconds before my table noticed I didn’t know.

I tried to ask NotebookLM.

“Hey NotebookLM, what’s the name of this person in this village the party met?”

The answer came back confident, fluent, and wrong. Wrong name. Wrong village. Wrong session.

NotebookLM doesn’t know what “the party” means. It has no persistent roster of my PCs. It has no trust layering — an NPC mentioned in a planning draft weighs the same as an NPC in a session summary, and the planning draft is speculative. It summarizes when I need it to return hits. It loses temporal context — met in session 4 looks the same as mentioned in passing in session 11.

Every time I tried to use it mid-session, the same thing happened. Fluent. Confident. Wrong.

This is the naive LLM problem from the Daz scene, one layer up. NotebookLM is doing extract + render with no review beat in between, no trust awareness, and no sense of which documents count as what.

So I had clean content I couldn’t query reliably. I’d solved the consistency leg with the loop and then rediscovered the same failure mode at the retrieval layer.

The flexibility leg wasn’t going to come from a better prompt.

---

Memory palace

The metaphor for what I built is older than computing. A memory palace. A building in your head where you place things so you can walk through and find them again.

Mempalace is that, externalized. A searchable index over the reviewed content the loop produces.

The important word is reviewed. The palace doesn’t index my raw summaries. It doesn’t index speculative notes. It doesn’t index the unrevised output of LLM extraction. It indexes what the loop has blessed.

The palace has structure. Three layers, different trust levels.

Narrative. The campaign bible, split into chapters. Authoritative. What actually happened at the table. If the palace tells me the narrative wing says X, X happened.

Chronicle. LLM extractions of the bible, organized by time. Search accelerator. Fast way to find the right session or the right window. Not authoritative on its own — the paraphrase problem still applies — but good at answering when did this matter.

Reference. Reviewed grounding docs, NPC dossiers, world state. Working reference. What’s currently true. Stable between sessions.

A query crosses the wings. Find the NPC in reference (who is this). Check chronicle for the time window (when did this matter). Verify in narrative (what actually happened).

That last step is the one NotebookLM doesn’t have. NotebookLM returns a paraphrase and calls it an answer. The palace returns a paraphrase and tells you the source paragraph. You can verify, in seconds, before committing.

Trust layering plus source retrieval plus structure-aware search. That’s the mechanism.

---

Prep time: the Daz problem, solved

Back to the Daz scene.

The question I should have run, before building the endgame encounter: what did Daz do with the evidence in that session?

In the palace: search the chronicle wing first — find the session where the evidence came up. Then pull the actual scene from the narrative wing. Read the two paragraphs. Three minutes, maybe.

What I would have found: Daz looked at the books, said something in character about them being unusual, and walked out of the room without touching them.

The encounter I was building would have been redesigned on the spot. Probably better for it — he knew and did nothing is a more interesting story beat than he took the evidence, for the kind of character Daz is.

That’s the consistency leg, live.

I now do this before every major prep beat. Anything I’m about to build downstream canon on gets a palace check first. Five minutes. Catches drift before it ships.

The seams of the loop feed the palace. The palace feeds the next round of prep. Clean content in, clean questions answered, clean content back out.

---

Mid-session: what I’m building toward

This is where the essay has to be honest.

I’ve been running the loop long enough to trust it. Prep-time palace use is in my workflow now. Consistency leg, back.

Mid-session use is the next chapter. I haven’t run mempalace at a live table yet — it’s new. What I’ve done before now is variants of the NotebookLM attempt, and I’ve reported how those went.

Here’s what mid-session use is designed to do.

“Hey palace, what’s the name of the bartender in Phandalin the party met in session 4.”

The palace knows the party — my PCs are listed in reference. It knows bartender maps to NPCs in the Stonehill Inn. It knows session 4 is a narrative-wing filter. It returns the two paragraphs where the encounter happened.

I read the name off the screen. I use it at the table. The scene keeps moving.

That’s the flexibility leg. Not the LLM generates an answer. The palace retrieves the passage and I read it. The retrieval is fast because the index is built on reviewed content. The answer is trustworthy because I’m looking at the source, not a paraphrase.

The design is done. The architecture matches the failure modes I diagnosed from NotebookLM. The at-the-table running-time is the thing I haven’t proved yet.

I’ll write the follow-up when I have.

---

Cross-campaign: the ceiling

The most interesting queries cross campaigns.

Group 2, years ago, captured Lolth as part of a deal with Vhaeraun. That action triggered the demon lord incursion. This is the premise of Out of the Abyss, Group 1’s campaign.

One campaign’s climactic decision is another campaign’s opening premise. When Group 1 asks why are the demon lords here, the correct answer involves Group 2’s choices from a year ago.

In notes-on-my-laptop world, this falls out of date the first time I forget to update one campaign after something happened in the other. It has fallen out of date before.

With a palace per campaign and shared canon in notes/canon/ that gets promoted into both, the coherence is maintainable. Query the Group 2 palace for Lolth. Query the Group 1 palace for Gromph Baenre. Both hits trace back to the same event. The world stays one world.

This is the ceiling the trilemma moves when it breaks. Not just consistent campaigns. Interlocking campaigns. Depth that compounds across groups and years.

No commercial tool does this. It’s not the market. It’s also not a hard problem — once the loop is solid and the palace is structured, cross-campaign canon is mostly an organizational question.

---

Trust is emergent

One thing I didn’t plan: the trust hierarchy fell out of the loop on its own.

A document’s trust level is how many review beats it has survived.

Raw VTT — zero review beats. Useful, authoritative in a literal sense (it’s the recording), but hard to consume.

LLM extractions of the VTT — one LLM beat, zero review beats. Accelerator, not truth.

Reviewed session summaries — one LLM beat, one review beat. Authoritative for what happened at the table.

Grounding docs synthesized from reviewed summaries — one more LLM beat, one more review beat. Working reference, trustworthy for planning.

I didn’t design this. I built the loop to fix a problem. The hierarchy emerged because different documents have been through different numbers of review beats, and I can tell the difference when I’m reading them.

The palace respects the hierarchy because it’s built on the output of the loop. The wings correspond to trust levels. Search accelerates by wing. Verification crosses wings.

If I’d tried to design the trust hierarchy up front, I would have gotten it wrong. I got it right by letting it fall out of the work.

---

The canon gate

One rule I don’t bend.

Nothing enters the palace without passing a review beat.

notes/ is staging. Arc drafts, encounter sketches, NPC ideas, speculative plot threads — all of it stays in notes. None of it gets indexed.

When a note becomes canon — when it’s earned its way into the campaign — it gets promoted to the appropriate grounding doc or dossier. Then the palace mines the new file. Then it’s searchable.

Not before.

The reason is the same reason the review beat exists at all. The palace’s value is that its answers are trustworthy. Start indexing speculative material and the next mid-session query returns a fluent, confident answer based on something I was thinking about doing, not something that happened. Exactly the NotebookLM failure mode, now in my own tool.

The gate is the thing. Everything else is support.

---

Things I tried that didn’t work

Worth listing. None of these are sermons — they’re my mistakes.

Indexing everything. Early on I mined every document I had into the palace. Every session summary. Every extraction dir. The published module. The result was search results dominated by the module and diluted by extraction redundancy. I now index only content with unique retrieval value. Module text lives outside the palace, accessed via the 5etools MCP instead.

Auto-promoting notes. I briefly had a script that watched notes/ and mined anything new into the palace. This took about a week to start returning speculative material as authoritative. Ripped it out.

Treating session_doc narration as record. The first-person per-character narrations are great to read. They are not the record of what happened. They’re a render of a review of an extraction. If I query the palace and get a hit from a narration file, I still go check the original summary. Same failure mode as Daz — fluent doesn’t mean source.

One-shot LLM prep. Feeding the LLM the whole summaries file and asking for an encounter. Works for shallow prep. Shreds consistency at depth. The loop exists because this doesn’t scale.

None of these were dumb ideas in the moment. They were natural things to try. They all break the same way — by letting the LLM do the architect’s job somewhere in the chain.

---

What I have. What’s next.

What I have, today, across two campaigns:

Clean content that a human has actually looked at. NPC dossiers with merged aliases. Grounding docs I trust. Session recaps where the dialogue is dialogue that got said.

A palace built on that content. Prep-time queries that catch the Daz problem before it ships. Cross-campaign coherence that holds across a year of play.

A consistency leg that’s back. A depth leg that’s real.

What I don’t have yet, honestly: live mid-session evidence that mempalace solves the flexibility leg. The NotebookLM attempts told me what doesn’t work and why. The palace is designed against those failure modes. I haven’t yet queried it under table pressure with a party watching.

I’ll write the follow-up when I have.

What I think this generalizes to: any long-form creative project with continuity requirements and LLM help. Novelists with series bibles. Worldbuilders with decades of notes. Researchers with years of reading. The loop doesn’t care that it’s D&D. GMs happen to be a good test case because campaigns combine volume, continuity, and table pressure.

I’m going to write more about the architectural side of this. Control-plane thinking applied to creative tooling. What the trust hierarchy looks like as a system. Why the extract/review/render pattern is a general answer to LLM-assisted knowledge work, not a D&D trick.

Those essays will be longer on the architecture and shorter on the dice.

This one was for GMs.

---

The ecosystem

Tools I lean on or built. One line each.

GMAssistant.app — turns Zoom VTT transcripts into structured session recaps. First extract beat in my pipeline.

Saga20 — similar space, different approach to VTT signal extraction. Worth knowing about.

CampaignGenerator — my tooling. Session prep, session doc generation, grounding doc synthesis, NPC dossier building. The scripts that run the loop. ([github link])

mempalace — the palace itself. Wing/room architecture, trust-layered retrieval, MCP-accessible. Build guide: MEMPALACE_HOWTO.md. ([github link])

5etools MCP — published module text stays here, out of the palace, accessed on demand.

notes/canon/ — not a tool, a convention. Shared cross-campaign history. Portable across workspaces.

---

Follow-up essay, after I’ve run this at a live table, will report.

One thing I do not understand is how enterprise companies will rely on infrastructure they cannot restore without help from the vendor from a backup.

My belief is that a system exists only if you, without the vendor, can restore it from a backup.

A system that cannot be restored from backup doesn’t exist.

It is contingent on other forces that can destroy it at any point in time.

And that, as a result, relying on that system for anything that must survive the contingent force is irresponsible.

So, for example, if your business relies on a system that cannot be restored from backup and that system lives in a data center, then you are saying that your business is contingent on the availability of that data center.

Some will argue that, well, you could always restore things if you have enough time, and my answer is yes, you can if you have enough time, but what is enough time?

If the system is complex enough, time can be months.

And if the time when it needs to be restored is measured in days, then the fact that it could be restored in months is irrelevant.

Recently, I read about a man whose son locked his entire Google account. As a result, all of his emails, his contacts, and his contracts were gone. And with taxes due in a few weeks, he could not file taxes, he could not reach out to customers, and he could do nothing. Bills could not be paid. Why? Because he had no backup. He had critical data that was owned by Google. Not by him. And when Google decided he no longer had access to it, it was lost.

That anyone would allow themselves to be in that situation is a mystery to me.

When I joined VMware, I discovered how we tested back in 6.0. We created a new vCenter, backed it up, restored it, and declared success. I asked the team to take a backup of a system that was running, and try a restore, and guess what, the restore didn’t work.

It struck me as mind-boggling that with that VCenter in 2015, it could not be effectively restored from a backup.

Over the years, I struggled to make infrastructure, in particular, VCF, something that could be restored from backup, but what I discovered is that nobody cared.

I don’t mean nobody at VMware cared. I mean, the customer base didn’t care.

This utterly confused me. I could not believe it.

It was only when I went to a customer that I realized that the customer didn’t trust the backup. It trusted VMware to do whatever it took to restore a system.

And what I realized was that the constraint on the customer was how many engineers VMware had on staff to recover backups when the backup procedures failed utterly.

In many ways, the VMware engineering team was insurance for the entire industry should customer systems fail spectacularly.

If you depend on VMware engineering, you are at the whims of whoever runs it and whether their interests are aligned. And to be 10000% fair, this is true of any vendor on the planet.

Broadcom, by stripping the engineering team of redundancy, created a scenario in which a catastrophic failure requiring a large surplus of engineers would result in extraordinarily bad outcomes for the world as a whole.

Why? Because the only backup that any customer can rely on effectively is the VMware engineering team, and that VMware engineering team is smaller.

What has happened is that the industry has said, “ I don’t need to have a backup because I can trust VMware.”

But you can only trust VMware as long as VMware’s interests and yours are aligned.

And when that is no longer the case, the fact that you don’t have backups that you can restore from without VMware means that you are at the mercy of VMware’s business priorities.

And this isn’t about VMware; this is about any company.

Backups you can restore from are your insurance policy if the Vendor fails you. If you can’t prove you can restore, your single point of failure is another business that can change on a dime.

And for me, as someone who had to restore a business from a backup in 12 hours or risk an IPO, the idea that you wouldn’t have your own backups and rely on somebody else is unbelievable and unfathomable.

When you look at AI systems, they are really good at getting to an approximate answer. If you think about randomized algorithms, they don’t return optimal answers, but they do return good enough answers most of the time with a lot less effort. When a product manager wants to create a product using AI and isn’t an engineer, they can get a pretty good, approximately correct answer. But it’s approximately correct.

In fact, about 4 years ago, my buddy and I chatted about this. AI is clever, meaning it gets to an answer. AI is not correct, meaning it doesn’t produce the precise correct answer. Sometimes you need the precise correct answer, and that’s where the LLM fails.

The workflow between product management and engineering is going to look like this; PM provides an approximate answer that gives a good-enough view of what the feature is. The PM team uses the LLM to create a plan and to render the code. The goal is to gain line of sight into the feature and validate its value.

Engineering takes that feature, and then creates a better plan that is informed by engineering expertise, and attributes the PM team that is narrowly focused on the feature, doesn’t have to, and should not have to care about. The detailed plan is then used to render the code. The old world was PM produces a document, then engineering creates a plan, renders code, PM reacts to the code, PM changes design, and …

The new world is PM produces a prototype rendered to code automatically from a plan created by the AI tool. When PM is satisfied with the prototype, engineering reviews the tool and then creates a more detailed, precise plan to generate the code.

Engineering may use AI to create part of the plan, but the evaluation of the plan and its correctness will contain insight and details that PM doesn’t and shouldn’t care about. Is it possible for PM to go end-to-end? Yes, but I also believe that the less you rely on expertise, the more likely you are to experience poor outcomes over time.

Anyways, the way I look at it is this: Use AI coding assistants to get an approximate answer that meets your goal. Take the approximate answer and refine the plan until it is more correct. Use the corrected plan to render better code