wrong tool

You are finite. Zathras is finite. This is wrong tool.

Nutanixist 33: delivering K8S deployment simplicity for VMs with no tradeoffs

by Leave a Comment

I want to talk today about a really radical innovation that Nutanix built that is very unique to our platform, and the side effect of a feature the team built. So we released a new capability in 7.6 called Projects, which allows you to do what we thought was just tenant isolation. Essentially, you could group a set of resources and isolate them from another set of resources for tenants. But what we didn’t realize that the way we built it let us do something far more powerful.

Let me explain the traditional problem with tenancy. The traditional problem with tenancy is that you have a resource and need to partition it between multiple tenants. The problem is that the resource partitioning can’t be done at the resource level, but has to be done at a higher level because you can’t go and integrate all the databases the underlying resources have. Very concretely, if you look at what VCF automation does, what it does is it says, “Well, since I can’t modify NSX, vSAN, and vSphere to have a single database such that all the resources are multi-tenant all the way down to ESX, it built a new layer called the Supervisor and then created namespaces. So as long as you go through that layer, you have tenancy, but if you go below that layer, you lose the tenancy information.

The image below is an illustration of a blog I wrote that shows the traditional layering.

Why is that important? It matters because the virtual machine state is now stored in a variety of locations. It’s stored in vCenter, it’s stored in ESX, it’s stored in NSX, it’s stored in vSAN, and it’s stored in the VCF automation layer, and it’s stored in VCF. Now, this isn’t meant to pick on that product. I think that’s actually a really credible solution and, to be quite honest, one that I’ve advocated for over the years. And certainly is part of the long-term vision of the Supervisor that some of the brilliant architects I got to work with articulated.

The issue, though, is that it doesn’t really provide you with an encapsulation oll of the resources that is portable and as reliable and robust as the virtual machine. In other words, I can have a virtual machine exist even if all the other software goes away. What Nutanix did was say, “Hey, what if we created a multi-tenancy layer below the virtual machine? Or, more importantly, such that the multi-tenancy layer was intrinsic to the virtual machine and the resources.” So what we did was we modified the underlying database that the flow product, the AHV virtual machine produc,s and the storage productuses, also known as Prism Central and Prism Element, to have tenancy baked in. As a result, a virtual machine only exists when it is part of a tenant. It can’t exist without a tenant.

So that sounds cute, but why is that useful? The reason it’s useful is that, all of a sudden, we created the first usable deployment spec for a virtual machine. The problem with a virtual machine and a deployment spec for it, is that it’s not just the virtual machine configuration, but it’s all of the storage, the networking policies, the identities, the categories, and the relationships to other virtual machines. And today, there’s no way to isolate that information and have that information be as robust as the VM configuration, except with Nutanix, because the same place that stores the virtual machine state is as robust as the same place that stores this configuration.

For the first time, you can now say this virtual machine and all of its configuration, both the VM configuration and its environmental configuration, are stored in one location and can be retrieved in an atomic way. Why is this powerful? Because what it says is that projects allow me, as an enterprise, not only to support multi-tenancy, but to support isolation of configuration, not for security, but just for operational simplicity. In other words, much like containers have allowed me to have multiple processes run inside of a Linux box that have a certain degree of isolation, this allows me to have configuration isolation of virtual machines at the operator level in a way that gives me confidence that operator A cannot mess around with the configuration of operator B. Or even if operator A and B is me, that if I do something to VM 1, it will not affect VM 2.

That is a very powerful simplification because it reduces infrastructure complexity from managing all 500 virtual machines at once to managing them in reasonable subsets, without deploying additional Prism Central infrastructure.

77 architecuralist: the rapid evolution of AI scale-out

by Leave a Comment

The other day, I remarked that it was possible that Apple’s R&D budget for 2025 equaled the entire R&D budget for tech in 1980. At the time, I thought I was exaggerating, but I then asked Gemini (see below) to go do some research, and what I uncovered was that I was not.

When looking at the current technology trends and moat of a company like Anthropic or OpenAI or any of the AI vendors, it’s easy to assume that they have a very large moat. The problem, though, is a moat is a function of money and resources, and the scale of the resources necessary to turn a scale-up problem into a scale-out problem is staggering.

But when you think about the total amount of dollars that exist in engineering investment and the willingness of pretty much everybody to throw whatever it takes to solve the problem, you realize that it becomes entirely possible to turn what should’ve been a 40-year problem, namely how do we go from scale-up AI to scale-out AI, to a four to five-year problem, and that’s what we’re seeing. Effectively, we’re seeing the attack of scale-out that took 40 years to complete, taking less than five.

What that tells you is that the era of the frontier models may be coming to a premature close. I don’t begrudge the end of that era because as the investment increases and the scale-out increases, the value declines. So, do I think it’ll go to zero? No, but I don’t think it will be where the action is in a year or so.

Here is the budget analysis from Gemini

Comparing tech-only R&D in 1980 to Apple’s 2025 R&D budget makes the contrast even more stark.

Apple’s FY2025 R&D spend was $34.55 billion.

In 1980, total U.S. industrial R&D in the technology sector—which encompassed information technology, mainframe/minicomputers, semiconductors, telecommunications, and office equipment—was roughly $8 billion to $12 billion in nominal terms.

Apple alone in 2025 spent 3 to 4 times the nominal total R&D budget of the entire 1980 tech industry.

Nominal vs. Inflation-Adjusted Comparison

Perspective1980 Total Tech R&DApple FY2025 R&DTakeaway Nominal Dollars~$8 billion – $12 billion$34.55 billionApple’s budget today is 3x–4x larger in raw dollars. Real (2025) Dollars~$32 billion – $48 billion$34.55 billionApple’s single budget roughly matches the purchasing power of the entire 1980 tech sector.

Key Historical Drivers

  • Fewer, smaller players in 1980: In 1980, industrial tech R&D was dominated by a few giants (IBM, AT&T Bell Labs, Texas Instruments, Xerox PARC). Even IBM—the undisputed titan of the era—spent roughly $1.5 billion on R&D in 1980.
  • Apple’s current scale: Apple’s $34.55 billion budget is driven by proprietary silicon, AI infrastructure, custom displays, sensors, and operating systems—demonstrating how single mega-cap tech firms today deploy capital at a scale that exceeds entire national industries four decades ago.

The Future of AI and Scale-Up Computing

by 1 Comment

One of the things that has been bothering me when I look at AI is that the advantages of the current foundation model hinge on their use of the most expensive part of computing infrastructure, which is global memory with low latency. What do I mean? If you look at what Fable does, at the core of it, it’s looking at billions of parameters or trillions of parameters, and then trying to infer the next token from those trillions of parameters. That is expensive because the hardware to provide that kind of speed and efficiency and bandwidth is some of the hardest hardware to build.

Historically, there was another kind of computing that had the same problem, and that was scale-up computing, essentially. How do I build a bigger, faster processor that would get better and better over time? What happened was that at some point, we realized we couldn’t scale up processors anymore. Instead, we had to scale them out by having more cores. In parallel, we discovered that trying to scale up applications was incredibly hard, so we had to scale out applications as well. That trend started in 1970 and ended in 2024 when scale-out applications became just the norm of how you build applications. Very few people are doing anything that is pure scale up.

AI is at, let’s say, year 4 of this trend where scale-up applications are what frontier models provide, and then scale-out is what all of the various tricks, techniques, tools, and mechanisms that exist for everything else. Now, the question if you’re an AI fanboy is not whether the foundation models do or do not win. That’s actually orthogonal to my thought, which is that what if the real problem is that the 40-year event of the disruption of scale-up computing suddenly takes not 40 years, but 5 years? The question there is, if it does, what does that mean?

What it means is that the value of the kind of technology that enables the Fable-like systems doesn’t meet the value because the scale-out systems, which rely on less efficient hardware, provide good enough results, or even worse, provide enough good enough results such that the investment necessary to deliver the next high-end model and the hardware necessary to support it are simply not as valuable. This trend disrupted the entire RISC processor market. It disrupted one of my early employers, SGI.

I’m tempted to say this might take 40 years, but I’m also cognizant of the fact that if I look at the total investment in R&D today and compare it to the total investment in 1980, I wouldn’t be surprised to discover that Apple’s R&D budget in 2025 equals the total tech R&D budget of 1980. Maybe I’m off, but it doesn’t feel a ridiculous thing to say.

nutanixist 32 the compounding advantage of architecture

by 2 Comments

I’ve talked a lot about the advantages of Nutanix’s architecture, and at some point I’ve appealed to reason, I’ve appealed to revenue, I’ve appealed to existence, but I haven’t really explained the nuts and bolts of why. This past week, I conducted an experiment on using various AI technologies to design a feature, and the consequences of that experiment provided me an excellent way to surface the architectural advantages of our system.

Our system, as those who have read the Nutanix Bible know, is essentially a distributed system with a collection of microservices sitting on top of a six nines available database. But one of the pieces that people don’t know about as much is Ergon, which is essentially our task system. What Ergon does is it provides a mechanism for you to create a task, and then it creates sub-tasks, and everybody can synchronize on those tasks. You create a task, and then that task gets turned into sub-tasks, and everybody knows exactly where everybody else is with respect to the top-level task. The other property of the task system is it’s idempotent. In other words, if I create a task, the same task will only be created once.

Now that we know that system exists, what does it mean? In a traditional way of building microservices, let’s say you have 3 microservices and they have to coordinate amongst each other without a transactional layer like Ergon, you have to account for the possibility that any one of those microservices might see some state that is modified unexpectedly. Why? Because there’s no way to grab a global lock on that state. But with Ergon, we can actually grab a global lock. So state transitions are very prescribed. The only way the state can move is if you have that task, and that task has specific gates, and those gates have to follow particular flows.

Although I said global lock, it’s not really a global lock. It’s a task-level lock. Essentially, what we’re doing is we’re saying for this task, all of the state variables that would be manipulated are manipulated under the agreement of this task flow, and they are manipulated in this order.

You know where you are in the task by looking at which subtasks have been completed and which subtasks have not yet been completed. You know if the task has failed by looking at the task state.

So compare the two models. In one case, every service has to be aware of every other service’s potential to interact with it and is unaware of whether that service has completed its work or not. Even if it knows that a particular service has completed its work, it doesn’t know if some other service knows that it has completed its work. So it has to guard against the possibility that some other service thinks that it’s in a different state than what another service says.

Whereas here, all you have to account for is that everybody knows what the current state of the workflow is, everybody knows who has completed and who hasn’t completed, and so you don’t have to have the level of testing and checking and error conditions that you would have otherwise. It’s why distributed transactions and databases are so valuable. It’s why transactional semantics and databases are so incredibly important and simplify software development.

So it’s not that we’re doing something novel in this, it’s merely that we’ve done it for infrastructure.

76 architecturalist: why human software engineers are always necessary

by Leave a Comment

I reject the idea that humans have no role to play in the future of software development. Not because I think AI will fail to become more powerful, flexible, or capable. It will. I reject it because the value of software is ultimately determined by humans, not by AI.

This is not a new question. Asimov asked it long before LLMs: if another intelligence can make better choices for humanity, should humanity let it? The answer his stories return to is uneasy but clear. Humans may accept help from another intelligence, even brilliant help, but they rebel against surrendering the right to decide what is good for them. The Wachowskis asked the same question in The Matrix and arrived at the same answer. A system can optimize for survival, comfort, or order and still miss the human point entirely.

As long as the human universe is the center of our universe, only humans can decide what is good for humans. Another intelligence may decide what is good for itself. It may even discover solutions for humans that humans would not have found on their own. But the final judgment still belongs to the human.

In practice, this means an LLM can produce a clever algorithm, a cleaner abstraction, or a faster implementation. But whether that solution is worth the token cost, whether it handles the corner cases that matter, whether it satisfies the usability expectations of the people who will rely on it, and whether it is worth maintaining are judgments only humans can make.

So, what is the role of humans in software engineering? It is the same role the human has always had. Software engineers did not exist to build software for machines. We existed to build software for other humans. Our job was always to translate human needs into instructions a computer could execute. That translation remains valuable because the difficult part was never only making the machine do something. The difficult part was deciding what was worth doing.

There is always a cost to producing anything. There are easy ways and hard ways. There are choices that make future work easier and choices that make it harder. Those are engineering trade-offs. But beneath every trade-off is a human value judgment: what matters, who it matters to, what risk is acceptable, what complexity is justified, and what outcome is actually good.

AI may change the mechanics of software development. It may change the speed, the leverage, and the shape of the work. But it does not remove the human role, because the human role was never merely typing code. It was understanding human needs deeply enough to decide what the machine should be asked to build.

How Fable made the case for Survivability as a first-order design principle

by Leave a Comment

I wrote about the notion of survivability before – survivability requires portability. If your application is not portable – meaning it can’t use alternative infrastructure then your application is contingent on someone else. And that if you are contingent on someone else, you are already dead.

If I were a European government and a military/political leader, the argument DJT made would be the following: we can destroy any business that depends on US frontier models at a moment’s notice. The decisions can be “because the US government doesn’t like Anthropic,” to “somebody bought Anthropic,” to “whatever crazy decision the US made”

Any critical infrastructure can not depend on a US-based model any more than it could depend on China. Therefore, European AI researchers and governments will start spending beaucoup dollars to build alternatives.

The interesting thing is that open-source models are increasingly good enough. The problem is that even if you have the hardware, the MODEL is what the US government banned.

The only way to guarantee sovereignty is to have a copy of the model weights locally and to use the model without communicating with anyone under the jurisdiction of the US government.

The case for -on-prem-ai- was made by the US government’s actions for any institution that has to serve customers who won’t be very impressed with their vendor saying, ” Well, the reason I can’t serve you in France is that the US government made a decision.

As a Frenchman, my reaction would be – “You are a French company, I am paying money to you, and I want that physical good, and you can’t sell it to me?”

nutanixist 31 : why orchestration is a dead end for agentic infrastructure

by Leave a Comment

I have been spending a lot of time on the next phase of system administration, agentic system administration. How can you use Agentic flows to manage infrastructure?

And Keith Townsend has put together an excellent mental model that I want to focus on. Not to critique it, but to ask: what is required of the infrastructure to provide what he calls the 2C reasoning layer?

My view on LLMs is that they are probabilistic theorem provers. If you can shrink the facts down narrowly enough, the theorem prover will probably know the right answer because the probability of you having a unique enough problem in practice is zero.

Where they fail is when the set of facts is too large, there are too many options, or the facts change mid-execution.

For anyone who has Claude, Codex, or Cursor, imagine having the systems try to update your code while someone else is simultaneously editing it. The results would be ugly.

There are three elements to facts:

  • (1) They must be facts. Every path to the fact must produce the same fact.
  • (2) The semantics of the facts must be encoded.
  • (3) The policies of the facts must be encoded.


A natural, convenient, and reasonable way to get those facts is through an MCP server. But it’s more than just an MCP server that naively transforms the API set. Here are the layers I view as essential.

The semantic layer is useful when I am trying to power off a host. Powering off a host can be a fact, but the claim that the cluster may crash when you power off a host is semantics. And that semantics is deterministic. But if you don’t provide it, you are allowing the probabilistic theorem to infer that, and it may not.

The policy layer is what says, “You can power down a host on Tuesday.”

But both the policy and semantic layers need a deterministic ground-truth layer that provides a single view of the host’s state. And that the host’s state must remain the same regardless of how the API’s state is queried. And that any change in the host state is reflected everywhere.

If it’s not, then the semantic layer can’t know if the host can be shut down, the policy layer doesn’t know if the host is the one that can be shut down, and so the layer above the MCP can only guess, and when the consequences of guessing are a catastrophic layer, it all breaks.

Nutanix’s infrastructure is not an orchestration layer of sharded, incoherent state; instead, it is built on a single global transactional and resilient database that provides local autonomy and global consistency, serving as the foundational layer required by any 2C system.

To make my point, let’s look at Proxmox and OpenShift. Please note that any errors are mine, and if I got this wrong, I would love to correct it.

Let’s first look at OpenShift

This looks good until you see that last layer. The challenge is that, because of how K8S works, until the operation completes successfully, the system remains in an indeterminate state. So, although the kubernetes API says “the system is thus,” or rather the ‘desired state is thus,’ it can not guarantee when it says that the state hasn’t shifted.

Again, is the host down, or is it on its way to being down? The k8s layer doesn’t know. And so the MCP server has to either wait for the k8s layer to find out and then act, or reach around the k8s layer.

Worse, any operator can reverse a decision the MCP server made. So the MCP server is dealing with a state that can change behind its back, and that change can break the decisions that an agent sitting on top of the MCP server made.

From the agent’s perspective -> I saw facts, I made a decision, the decision failed. The problem is that the facts changed while the agent wasn’t looking. So it goes back and asks again, and gets different facts, so it does something different.

The probabilistic theorem prover is looking at a set of facts that are continuously changing

Now let’s look at Proxmox

Again, the MPC server has control planes with overlapping state. It must figure out who to query and when, and orchestrate an operation that is not transactional. So from the agent’s perspective, the fact is malleable. And a malleable fact is destructive for a probabilistic theorem prover.

After the MCP server returns, the actual ground state is unknown.

So the agent is always wondering why the thing didn’t work. And the answer is because the facts changed. But then, the agent has to start again. And then it has to create a plan that can account for every possible change, or it has to give up.

75 architecturalist papers: types are the right answer

by Leave a Comment

I have come here to gloat and observe.

When I was at VMware, a lot of the code written in Python was being used in production. In my mind, that was an awful technical decision because VMware was shipping a packaged appliance. Every CPU cycle we spent on Python was a CPU cycle the customer was spending, essentially imposing a tax on them for our convenience. The issue I had wasn’t just that we were spending it for our convenience, but that we were using a technology that was intrinsically unsafe due to its lack of type safety.

So, I would engage in intense debates and arguments with some of the Python enthusiasts. They would vehemently assert that Python was robust, reliable, and even superior. They would go on to explain that it was easier to code in and prototype with Python.

I couldn’t deny that if you’re trying to build a prototype, Python’s simplicity, speed, and ease of use are undeniable advantages.

But when you finally ship code, you are no longer prototyping. You know what you’re shipping, so you’re no longer experimenting. So that value is zero.

Fast forward a few years, and I end up at Nutanix. I’m looking at LLMs, AI, and generated code, and I’m wondering about my choices in terms of backend languages. I did some research and realized that Python is actually a very good language to use because the token cost is very low since it’s very expressive. Also, for personal projects and the kinds of things I was doing, most of the libraries and tooling that I want, use it.

So, having done my research and having given it some thought, I said, “Okay, Python is the language.” I figured I don’t really have to learn all of the vagaries of Python, and the performance is going to be dominated by the LLM. If I ever get to the point where this matters, I’ll figure it out.

I kind of lived in that happy bubble until I got this project that I downloaded from GitHub, a really great project. (https://github.com/nashsu/llm_wiki) The author was using JavaScript and Vue for the front end, and to my surprise, they were using Rust on the back end.

The back end of their server was entirely written in Rust, and I was like, “Well, that makes no sense.” First, I thought it was a case of an engineer trying to be clever, cute, or experimenting. At the end of the day, it’s an open-source project. They’re not being paid, and if they want to do something in Rust because that’s what they want to do, that totally makes sense.

But then another friend of mine told me that, well, actually, that’s a new emerging pattern. The reason is that using Rust is a much more token-efficient way of writing code once you have product feature lock. In other words, once you’ve shipped version 1, every subsequent iteration costs fewer tokens because of the context Rust gives to the computer and the fact that it catches a whole bunch of bugs as a result of the type safety. So, the net effect is you’re using fewer tokens per iteration afterward.

That made me smile because literally, the AIs figured out exactly what I had figured out 15 years ago, and what Niklaus Wirth figured out 50 years ago. And now people figured it out on their own.

Back then in 2015 when I got in front of all those Python engineers and told them that they would never write another Python service as long as I was architect and they should all go learn Go, I felt kind of bad. I was basically making an assertion with no data. But now I’m happy to say I was right, and yes, I’m going to gloat about it.

But.

It’s kind of cool to see that there are certain facts about software engineering that are not tied to human cognition but to cognition and information in and of itself. That to me is the most interesting thing about this.

74 architecturalist papers: cheap experiments

by Leave a Comment

Notes from moving a D&D distillation off the API and onto two boxes I own.

The run that hung

I was running the whole campaign through a new extraction pipeline. Fifty-six sessions of Out of the Abyss, fanned across two DGX Sparks. Most of it ran fine. Then four units at the tail hung for an hour and forty-four minutes on one box, while the other Spark sat there doing nothing.

I was sure I knew what it was. vLLM degradation, the decode running away toward the token ceiling. So that’s where I went looking.

That wasn’t it. The metrics showed zero preemptions and an almost-empty cache. Nothing was slow. The laptop driving the run had gone to sleep. The Python processes froze with their sockets to vLLM half-open, and when the laptop woke the sockets were dead and the reads behind them never came back.

Nothing had degraded. A read was just hanging forever. That isn’t a slow box and it isn’t an error. You can’t wait it out and you can’t retry it. You put a clock on it.

I’d had a timeout in an old script. A rewrite dropped it. I put it back, added a read timeout on the client, and moved on.

That’s the kind of bug this whole project was made of. The bug isn’t the interesting part. The interesting part is that I could afford to chase it at all.

What I was actually doing

The job hasn’t changed since I last wrote about this tool.

Take 56 campaign sessions and distill them into a world_state.md — a compact doc that says who’s alive, who’s angry, what’s on fire right now. The thing I scan before every session.

The old version did it with the Claude API. Chunk the bible, extract per chapter, one big synthesis call. It worked.

It also costs real money every run. So I ran it once, looked at the output, and lived with what I got.

That last part is the whole problem. I didn’t see it until the meter went away.

How the ensemble actually happened

I, of course, would like to say that I read the research, used my brilliant insight, and designed a five-lens ensemble.

And that would be … well, less than true?

It started with one prompt.

Read a chapter, emit the facts. Six thousand tokens at a time. I ran it. It gave me some facts.

Then I ran it again.

Different facts.

I asked about it and got the standard answer: that’s normal, it’s non-determinism, the model isn’t deterministic, and you live with it. Which is true, and is also the kind of true that makes you stop thinking.

So I didn’t quite accept that. What if I just do it multiple times?

So I ran it twice and took both. And the union was better than either run alone — not because the model got better, but because two samples of a noisy process see more than one. That’s where self-consistency came from.

Then, looking at what I had, I saw that facts were clearly being missed. Things in the text that just weren’t getting emitted. So a sweep was added to get every named fact.

Then I noticed timelines were getting dropped. A whole sequence of events would collapse into a single fact, and the when would vanish.

Which sounds easy to fix until you sit with it. What is a number? What is a timeline? “Three days later” is a fact. “On the eighth day” is a fact. “After the rockfall” is a fact about ordering with no number in it at all. Figuring out what we even meant by temporal took its own round of tests. That’s the temporal lens, and it was not obvious.

Then a subtler one. The facts were interpreting when they should have been recording. The model would read a scene and tell me what it meant. It wouldn’t tell me that the character held another’s hand; it would tell me that the character liked the other. I wanted facts, not meaning.

And now I had too many facts. So time to merge.

Merge showed me something I couldn’t see before. The sweep pass, the one told to be exhaustive, was packing several facts into one. It read like one fact, if you think one fact is a sentence with three facts connected by and statements. So back to the prompt: one fact equals one state-change, with worked wrong-and-right examples. The tell turned out to be the ellipsis. If the model reaches for … to stitch a quote together, it bundled. Split it. That alone took bundling from 15% to 1%.

And only then, with a prompt stack that actually worked, did the real engineering question show up:

How do I run this?

Five prompts, multiple samples each, against two boxes.

So I had five prompts, but how do I run them correctly? The problem is that now any bug in how the prompts are run is going to cost money. Make a bug in your orchestration logic that comes from my pocket. That hurts. That would have stopped me there. I needed orchestration that fans all of it across both Sparks and merges it deterministically and I couldn’t afford a bug, unless, the cost of doing a run was free. And it was, *free*. So I could build the ensemble. And debug it. Like when the laptop hung.

Then I tried it on a few sessions. Then on six. Then, on the whole campaign.

By the end, the union of those passes was as good as the distill script.

This is the price of iteration

Read that sequence back and count the steps.

One prompt. Notice non-determinism. Run it twice. Notice missing facts. Add sweep. Notice dropped timelines. Argue about what a timeline even is. Add temporal. Notice interpretation creeping in. Split recording from inference. Build the merge. Discover the sweep is bundling. Fix the prompt. Then build the ensemble to run everything. Then scale it up in three steps.

Not one of those steps was a plan. Every single one was a reaction to something the previous run did wrong, which I could only see because I’d run the previous one and stared at the output.

That’s the part the meter kills.

Each of those steps is a run. On the API, each run is a charge, and most of them produce a result whose entire value is “huh, that’s wrong in a new way.” A null result, which costs you money. It made you smarter, but it cost you money. You can’t justify “let me run it twice to see if the non-determinism is hiding recall” when the second run costs money and might just confirm the first. So you don’t. You take the first okay-ish output, and you stop.

And if you stop at “run it twice,” you never see the missing facts, so you never add the sweep, so you never hit the bundling problem at merge, so you never write the bundling fix. The chain exists only because each link was cheap enough to follow on a hunch.

The ceiling wasn’t Qwen. Qwen could always do this. The ceiling was how many times I could afford to be wrong on the way to finding out.

That’s the price of iteration. When you’ve already paid it as capex, you can afford to pay it over and over.

What beat what

The local models didn’t win. Qwen did not outwrite Claude. The final synthesis still runs on the big API model because it’s low-volume, cross-entity, quality-critical, and not where I want to save money. And by the time it runs, the input has been reduced from 537K tokens to about 40K, so it’s a cheap call.

Local to explore. API to finish.

The better world_state came from the chain, and the chain only exists because each run was cheap enough to follow on a hunch. The old API result was never the API’s ceiling. A well-iterated API run could probably have matched it. I just couldn’t afford to be wrong that many times in a row, because under a fixed budget, every null result is pure loss, and the chain is mostly null results until the end.

But it’s even worse, because I had already distilled the data, with Anthropic’s API, I couldn’t justify trying to make it better.

A meter doesn’t only cap how many experiments you run. It biases you toward timid ones. It makes the weird, probably-won’t-work shots feel irresponsible.

Owning the box doesn’t make you smarter. It stops punishing you for looking.

Where this doesn’t hold

And here’s where we have to be careful.

Utilization is carrying that word “free.” The power was pennies, but the amortized cost per run is only low if the boxes stay busy. An idle Spark has a high effective cost per token. For an occasional personal pipeline, the money case is thin, and the real reasons are data locality, iteration freedom, and learning.

The engineering is the actual bill. Speculative execution, the timeout, the dead-socket hunt, that’s labor the API path externalizes to Anthropic’s SRE team. Own the serving, and you pay it yourself, in time, and you pay it again every time the stack drifts. It was the highest real cost here, by a lot.

And free experiments erode discipline. A budget imposes a crude “think before you spend” rigor. Remove it, and it’s easy to run 50 variants when 5 would have told you the same thing. Local doesn’t free you from experimental discipline. It just shifts the responsibility for supplying it from the meter to you.

Plenty of organizations are sitting on idle GPUs. The question worth asking isn’t whether to buy hardware. It’s whether there’s a sunk, underused inference asset you can route verifiable bulk work onto that’s reviewable, iterative, bounded, and fine to run in minutes instead of milliseconds.

Aggregation hit all four. It went local. Synthesis hit one. It stayed on the API.

Eight pull requests were what it took to find that line, one cheap experiment at a time. Including the four that did nothing, and the one where I was sure it was vLLM, and it was a laptop taking a nap.

That one was free, too.

The pipeline lives in CampaignGenerator — ensemble_extract.py, synthesise_world_state.py, spell_canon.py, and the per-lens prompts under config/agents/. The robustness work is PRs #64–#71. This is the retrospective, not the code.

Introducing TurbovecDB an alternative to ChromaDB?

by Leave a Comment

Over the past few months, I have been working on a system I call “CampaignGenerator,” and one of the critical questions for that system is: how do you store state and make state retrieval fast?

When I discovered MemPalace, what drew me to it was its simplicity, locality, and elegant architecture.

What drove me bonkers was ChromaDB’s performance and stability.

Along the way, I got a DGX Spark, and that got me thinking about TurboQuant. And how that might make my DGX Spark’s more useful.

And as I kept looking around, I discovered a project called Turbovec And that then made me go, hmm.

Could Turbovec replace the HNSW system that ChromaDB used, which made it so slow? And could it do so without breaking any of MemPalace’s guarantees?

So Claude and I started experimenting. And the results were promising and exciting.

But Turbovec is only part of ChromaDB. To have a full database, you need a stable store.

So what the heck, I decided to create a database, TurbovecDB. That was actually an experiment in using Claude.

I asked Claude to review what MemPalace required and determine how to build a database that would meet its constraints.

Turns out that the requirements of MemPalace are reasonably straightforward, and so Claude determined that the solution was to build an API layer on top of SQLite, use Turbovec instead of HNSW, and voila, turbovecdb v0.1

I am not going to argue that what I built is a masterpiece of software engineering; I am going to note that this was relatively straightforward and shows the power of interfaces.