wrong tool

You are finite. Zathras is finite. This is wrong tool.

the nutanixist 21: architecture is why Pure and nutanix could deliver a great solution in record time

by 2 Comments

One of Pure Storage’s more remarkable achievements was its integration with vVOLs. They spent years integrating and making vVOLs work. And, without a doubt, a significant part of the reason this was easy was due to the work the Pure team did. 

people hands reaching out
Photo by Priscilla Du Preez 🇨🇦 on Unsplash

But this is why it was easy for Nutanix and Pure. 

vVOLS was the answer VMware had to how to enable storage vendors to provide VM data management that was integrated into VMware’s policy-based management framework. 

That’s a mouthful.

In 2008, NetApp introduced a product called SnapManager for Virtual Infrastructure, which revolutionized how people discussed storage integration with VMware. Instead of seeing storage as independent of compute, it was presented as an integrated operational workflow. The VI admin using SMVI could directly integrate with NetApp storage to take snapshots and provision storage. 

In 2011, Nutanix introduced HCI, which provided VM-level data management that bypassed the operational concerns of storage administrators by removing them from the equation entirely.  

In 2012, VMware introduced policy-based storage management, along with the first incarnation of vVOLs in 2015, to enable policy-based management of storage. 

What VMware aimed to do was enable the entire storage ecosystem to integrate with the vSphere control plane, providing the operational value of VM data management in a consistent, vendor-agnostic manner.

Effectively, the goal was to move around competitors like Nutanix and NetApp by commoditizing the VM data management. And make vSphere the way you manage data, with storage vendors acting like providers. 

It was a good idea, and it was on the cusp of greatness, but for what I can only imagine were misguided, petty reasons, VMware canceled it.

Many of the challenges of vVOL were inherent to vSphere, making integration very difficult. 

vSphere doesn’t have a cluster control plane, and VMFS does not have a single control point for I/O; the VMFS IO path is in the kernel. 

So, what were vVOLs? Without getting too deep into the weeds, what VMware did for vSAN was add a new path in the core storage stack of vSphere. That layer was then integrated into the vSAN cluster control plane. That same interface was then externalized to the partners. 

And that was the problem. 

The storage partner was tasked with the complicated problem of building a clustered storage control plane. Why? Because vSphere, as I have explained elsewhere, doesn’t have a clustered control plane and allows independent hosts to make independent decisions that the control plane must react to. 

When vMotion occurred, the VASA provider was involved in the operation as it had to unmount a LUN from an ESXi host and then remount it on another host. 

But it was messier. Because vSphere cannot guarantee the number of hosts that will connect to a storage array or the number of LUNs that will be mapped, the VASA controller must manage any limits. 

And then, finally, due to VMFS limits, the number of vVols that could be connected to any host was limited. 

For Nutanix, these problems didn’t exist. Due to the clustered control plane, we could ensure that the number of LUNs connected to a storage controller remained within the limits agreed upon by Pure and Nutanix. Because our IO path was in user space, we could mount and map every virtual disk on every host. And because of our clustered control plane, during a Live Migration (Nutanix’s vMotion), we could handle the re-routing of the IOs without requiring the external storage provider to do the fencing for us. 

Unlike vVOL, which requires the storage vendor to build a clustered control plane from the basic primitives of the per-node file system, the storage vendor integrates with a cluster control plane and operates on cluster-level semantics. 

That is why our integration was fast. 

And more importantly, why our integration delivers more value and better availability than the dearly departed vVOLs. 

the architecturalist 63: nutanix was the correct answer

by 1 Comment


In 2012, while at Zynga, I had a moment of clarity that the way we had thought about infrastructure up to that point was wrong. That our focus on making a single node more and more available was a dead end.

I wrote about this on Quora, and it was picked up by Forbes, which gave me 1 minute of fame.

And I wrote this:

person using magnifying glass enlarging the appearance of his nose and sunglasses
Photo by Marten Newhall on Unsplash

NetApp’s engineering spent a lot of time worrying about hardware availability and making hardware appear to be much more resilient than it actually was.

And yet, these guys like Facebook, Twitter, and Google didn’t think that was important.

Which was mind-boggling. How else can you write software if the infrastructure isn’t perfect? “What were you people doing?” I thought.

So what drove me to find another job was that somehow, people were building meaningful applications that didn’t need component level availability. Something was changing…

Which brings me to what was changing.

What was changing, and this only became obvious after I joined Zynga, was that the old model was dead.

In a world where you have thousands of servers, depend on services that change all of the time, the notion that the application can be provided the illusion of perfect availability is, well, foolish.

In fact, applications have to be architected to understand failures. Failures are now as important to software as thinking about CPU and Memory and Storage. Your application has to be aware of how things fail and respond to those failures intelligently.

I believe that the next generation of software systems will be built around how do you reason about failure, just like the last was about how do you reason about CPU and Memory and Storage.

For the last 13 years, I have been wondering what the correct answer is. One school of thought believed that the correct answer was to treat everything as a database transaction. What if we made infrastructure transactional?

As a result, numerous attempts were made to develop management applications that updated the model of the world in a database and attempted to force the real world to conform to that model. I even invented one and published a paper that described such a system.

And they kind of worked.

The general idea was that you had an API that updated a database, and then a set of controllers that would go and modify the world to conform to the database. And if they ever detected an inconsistency between the world and the database, they would go and correct the system to conform to the database.

And those systems failed to deliver on transactional infrastructure.

When you invoke an API, the database gets updated, and the world converges, but here’s the rub: the world can diverge. And you wouldn’t know.

Let me provide an example from vCenter, a product with which I am very familiar.

Let me be specific – you tell vCenter to power on a VM. vCenter updates its database, then communicates with ESXi, and the VM is powered on.

But is the VM powered on?

You don’t know, because a user can log into ESXi and power off the VM.

In effect, ESXi has its own database and API. And that API and database can be used to change the state of the system.

To make matters worse, if a network partition occurs, the VM will be powered on, and vCenter cannot determine if the VM is powered on or not.

Therefore, any piece of code written must account for three states: “Yes, No, and I don’t know.”

Now, if it’s only one client calling vCenter and doing one thing at a time, that’s manageable. However, if you are working with workflows that depend on the VM being powered on, for example, powering on the VM, moving it, and so on, then for every step, you must account for the possibilities of ‘yes’, ‘no’, and ‘maybe’. And that handling all the different kinds of ‘maybe’ makes writing the control plane tricky.

And when I was at Zynga, I would like to believe I had identified this problem, but I had no idea how to solve it.

For years, I thought the only path forward was the desired state. In short, you express an intent, and the system converges to that intent. But the problem with that model is that expressing things as a sequence of operations is more convenient than simply describing intent. The problem with intent is that if you need to express two different contingent intents, how do you do that? And yes, you could, but pretty soon, you have one massive intent that describes the entire universe.

And so the approach, although promising, never materialized.

And then I ended up at Nutanix. I have also noted that Nutanix has a distributed database at its core, which is part of the puzzle. However, as I mentioned earlier, it’s only a part.

There were three more.

The second was the ability to have a parent database with multiple child databases, and that the parent database would always receive updates in the correct write order.

The third was soft transactions. This is critical because the system must perform reliably and be able to tolerate failures.

But the piece of the puzzle that eluded me was the need for two magical pieces of technology: the first was AHV, a stateless operating system, and the other was Stargate, a clustered IO path.

What Stargate guarantees is that the cluster knows which disk is being connected to, and it provides a point of control for the disk. It is not possible to change the state without Stargate knowing. And so, for a cluster, Stargate can prevent anyone from accessing disks and assert who is accessing them.

The second is AHV, which, when it reboots, doesn’t remember what it was doing before it rebooted. Therefore, AHV cannot run any workload without the cluster knowing what the workload is.

When you combine all five pieces of technology, you have the answer to the question I posed.

The infrastructure, by design of the datapath and system components, only has two answers to any operation: “Yes, I completed, and No, I didn’t.” And either is definitive. There exists no other possible answer to the question.

Once you have such a system, it becomes possible to implement two services that control the OS and the datapath that can assume the behavior of the infrastructure is binary.

And once you do that, you can build a system of APIs that always return yes or no to any question.

This then allows you to combine APIs into workflows that can be trivially designed. What do I mean?

Suppose I have a workflow that must call 5 APIs. We model this as a single workflow comprising five tasks.

In transactional infrastructure, after each API returns a response, I know what the environment must be. And therefore, if it says “Yes”, I can advance to the next step knowing that it is “Yes.” In other words, if Task 1 is completed successfully, I can easily advance to Task 2.

So let’s consider the alternative. Task 1 is to power on a VM. Task 2 is to attach a network to the VM. If Task 1 declares success, Task 2 might fail because someone behind the scenes shut down the VM. Now, Task 2 must handle an error. But what does this mean for the workflow? Did the workflow fail? Well, it didn’t. What happened was that the environment changed in a way that the workflow was unaware of.

So let’s look at the workflow state –
Task 1 – power on VM – success
Task 2 – Attach Network – Failure because the VM is not powered on.

This is a contradiction. How could Task 1 succeed and Task 2 fail? This is a contradiction because the workflow didn’t account for another system changing the state of the VM behind the scenes. And because the change occurred outside of the system, the program interacting with the APIs cannot determine why it has a contradiction.

To understand what happened, you need to build yet another system that monitors both the workflow and the system that can be changed outside the workflow’s control.

Intent-based systems attempted to work around this by retrying, but, as I mentioned, they had their own issues, the most significant being an infinite retry loop.

Ultimately, the only solution was to make it impossible for the system to be changed that was not under the control of the control plane.

And that’s what the folks at Nutanix did.





the nutanixist 19: the arrogance of the broadcom shift in cloud credits

by Leave a Comment

I usually don’t discuss business models, but what Broadcom did is a good example of how thinking you have an irreplaceable product and not understanding your customers can cause problems.

One of the main challenges with VMware was the variation in business models, which made license portability difficult.

That variation also led engineering teams to go to great lengths to avoid collaborating.

In many ways, VMware was like several companies in one, each with its own distinct business model, selling layered products on top of vSphere.

Changing this corporate structure has been the main driver behind Broadcom’s changes to VMware’s product setup.

At the same time, we live in a very complex world. Corporations have very complicated budgets.

The core of selling goods is to meet your customers where they are.

Broadcom’s goal is to remove customer choice. The idea is that by forcing customers to work with the team that owns VCF credits, workloads will be pushed back into on-premises environments or not moved to the cloud.

Here’s how it works:

Think about “Big Corp Co.” with two teams. Team A wants to run some workloads in the cloud, while Team B is responsible for on-premises virtualization. Team A has workloads on vSphere that need to be migrated to the cloud, and the most cost-effective way to do this is to utilize some of their corporate credits.

But now, Team A can’t use those credits.

Since running the stack on VCF requires VCF credits, they will be directed to the internal VCF team, Team B, which will tell them that instead of running in the cloud, they should run on-premises.

Team A might protest, but Team B, which controls the budget, will explain that there isn’t enough budget for the cloud deployment and offer an on-premises alternative.

Therefore, Team A, because they are forced to use VCF for certain reasons, will theoretically have to move any workload that could have run in the public cloud back on-premises.

This approach assumes Team A has no options.

And that’s where this model of limiting choices fails.

Customers always have options.

And when you force someone to do something, they will quickly find ways to choose differently.

It’s why Nutanix has been adding many customers lately.

the nutanixist 18: relying on infrastructure instead of application specific availability

by Leave a Comment


From 1996-2009, it was believed that application availability was an infrastructure issue, so improving reliability meant making infrastructure more resilient. Tandem NeverFail symbolized the ideal of reliable infrastructure, with systems like the Origin 2000, featuring a single system image and NUMA, representing the peak of scalable computing. However, in the mid-1990s, Gregory Pfister’s book “In Search of Clusters” argued that building such infrastructure was too complex, advocating instead for clustering.

At the time, this idea seemed absurd. When distributed systems were first being deployed in the mid-1990s, it seemed truly crazy.

As a result, infrastructure vendors continued to focus on making single systems more resilient.

When the cloud emerged, infrastructure architects like myself viewed it skeptically because of its lack of guaranteed availability. “How would applications run on it?” we wondered.

What we didn’t realize was that software naturally seeks to operate on cheaper hardware, and because of this, new technologies have arisen to make that easier.

For me, the pivotal moment came in 2009, when Cafeville was running on an effectively 1000-node cluster. The team combined various components with some critical innovations.

This marked the beginning of an era where availability shifted from being purely an infrastructure concern to an application problem because infrastructure became less reliable.

My critique of vSphere and similar systems that aren’t natively clustered is that they are inherently less reliable than what applications require. Consequently, application teams must write code assuming infrastructure instability rather than depending on the system’s reliability.

What do I mean by infrastructure instability?

In the pre-Cloud era, infrastructure was assumed to either work or fail. In the cloud era, uncertainty in infrastructure was acceptable if the application, its developers, and the operations team could identify what went wrong—uncertainty was tolerated.

The problem was that this increased the cost of maintaining and supporting applications and slowed down development, as teams spent more time on infrastructure issues than on the applications themselves.

At Zynga, when my team provided a reliable infrastructure, team sizes decreased, and productivity for the game teams increased.

Our team ensured there would be no ambiguity about how the infrastructure was performing. We provided guarantees.

By stating that infrastructure needs to be more robust, I mean that it must ensure the application and its components are operational, that data is available, that no infrastructure changes have occurred, and that the system can recover if needed from a backup without requiring the system to be rebuilt.

And that, in an era of clustered applications, clustered infrastructure that can give those guarantees by default, like Nutanix, is the only way forward.

the architecturalist 62: people develop tools, software is a means not an end

by Leave a Comment

In 1994, I was told by a visionary professor of Computer Science that I was a fool for going into CS because the combination of component software design and offshoring was going to eliminate jobs.

I remember being pale in the face and sticking with it. At the time I graduated, there were 13 CS graduates, of whom two had cross-disciplinary fields. That class had the guy who invented Hadoop, and the folks who invented dtrace, and me (yes, I am putting myself in the same breadth, but that’s because we graduated at the same time).

Thirty-one years later, I see the same kind of fear-mongering.

The notion that computers will do software engineering or that there is a finite demand for engineered products remains the dumbest and most ignorant take in the history of takes.

AI is just the latest iteration in making each unit of software we write more efficient. In the 1980s, it was the move from assembly. In the 1990s, it was the move to garbage-collected programming languages. In the 2000s, the emergence of databases, hypervisors, and the web occurred. In 2008, it was the emergence of public cloud.

Does that mean that there aren’t dislocations and changes? No. In fact, in those transformations, jobs stopped existing, and folks had to retrain. And some of it was unfun.

But the idea that tool-making, design, and construction don’t require human beings is the fevered dreams of AI advocates.

the nutanixist 17: the consistent nutanix cloud platform

by Leave a Comment


In an earlier post, I observed how VCF’s state was like the cat in Schrodinger’s box. It was impossible to determine the state because each observer had a different view of the state, and there was no consensus protocol.

Nutanix’s engineering team took a fundamentally better approach than VCF.

As always, read the https://nutanixbible.com for more details.

Both the Nutanix Cloud Platform and VCF have the same problem: how do you share state in a distributed system?

In particular, given a set of programs in a distributed system, they must all agree on the value of any shared state. If they don’t agree and don’t know that they don’t agree, then each program will make different decisions based on its view of the value. For more https://lnkd.in/gVHePnME)

Where it gets gnarly is when you have failures. And that for everyone to agree on the value, everyone has to see the same updates to the value in the same order (there are variations on this requirement).

For example, suppose I have three databases, and each database has a copy of my bank account.

My starting balance is 100$, I deposit 100$ and withdraw 150$

With no consensus protocol, the following is possible:

Database 1 thinks I have 50$
Database 2 adds an overdraft charge because it saw the withdrawal of 150$ after it saw the deposit of 100$
Database 3 thinks I have a balance of 200$ because it never saw the withdrawal

With a consensus protocol, there is only one possible outcome, namely that each database thinks I have the 50$ in my bank account.

Both VCF and NCP are distributed systems. VCF has a set of central databases (NSX db, vCenter Postgres, Operations) and a set of edge databases on ESX hosts. NCP has a single centralized database and a set of edge databases in the form of clusters.

I already discussed VCF, so today, let’s focus on NCP.

So how does Nutanix maintain consensus between the central database and the clusters?

Each cluster database notifies the central database of all updates in the cluster in the order they were made.

As a result, the central system always has a complete and consistent view of the state of the cluster.

And all of the products built on the central system have a single, consistent view of the state of each other and the clusters.

This doesn’t sound like much, and it’s why restore actually works on NCP.

When I restore Prism Central from a backup, it has a consistent view of every cluster. It is not possible (modulo a bug) that the backup will contain a state of the environment that never existed. Nor is it possible for different services to have a different view of the environment.

It’s why you can restore from backup and recover an environment, whereas with VCF, you must do a rebuild.

The problem with this system, however, is not just backup, it fundamentally affects scale and availability.

Why does this matter?

Obviously, because backup 🙂

But it also affects scale. And correctness.

The VCF system works because, although theoretically the databases are out of sync, the system is working very hard to keep them in sync. And so as long as changes to the environment occur less frequently than the time needed for each database to figure out what is going on, the system works.

So what? Doesn’t every consensus protocol impose some cost? Yes, but the VCF consensus protocol, such that it is, doesn’t guarantee that the state is consistent; it says the state should be consistent. So if you scale the system incorrectly, instead of the system becoming slower, it will behave incorrectly. 

the nutanixist 15: schroedinger’s VCF

by 1 Comment

In my previous post, I talked about irreducible control planes. And hand-waved their implications, but those implications explain why Nutanix is so fundamentally different.

brown cardboard box on white table
Photo by Sahand Babali on Unsplash

An irreducible control plane is the minimal control plane that must exist for a service to exist. For VMs on ESX, that’s ESX. With the ESX host running and access to the ESX host, you can perform any actions you want on a VM.

Every other control plane adds value but is not essential.

vCenter is an optional system. And thus, the first peculiar part of vSphere is the split brain between vCenter and ESXi. vCenter does not control ESXi entirely. It has to account for the possibility that the state of an ESXi host changed while it was running.

In effect, vSphere states that if ESX is running and you can authenticate to the ESXi host, you have complete control of the system, even if all other software is not running. It also states that you can always log into the ESXi host and take action, even if all other systems are running.

Thus, ESXi and vCenter are permanently running in a split-brain mode, and it’s vCenter’s job to react to what happened on ESXi.

The design principle is a good one. That’s not a bad design principle. And I have argued that a singular design principle is why ESXi is the best system for running traditional single VM legacy workloads, where all you care about is running the VM.

But if you care about doing anything other than the VM running, then how vCenter knows about the state of the ESXi host is super interesting. If vCenter and ESXi have different views of the state of the host, and those views get out of sync, what happens?

For example, when an ESXi host gets disconnected from vCenter, and a host in the cluster that the ESXi failed, and vSphere HA starts a VM on that disconnected ESXi host, what happens?

What does vCenter know and when did it know it?

And if vCenter doesn’t know and you can’t explain what it knows and when it knows it, how does a system layered on vCenter know what vCenter knows?

So, for example, NSX relies on vCenter to know things about the ESXi host. And if vCenter has wrong information, then NSX has bad information.

And maybe that -might- be okay, if NSX and vCenter had consistent but wrong information.

But they don’t.

vCenter talks to ESXi via hostd and vpxa. And those agents don’t return all the information on a host. And they are certainly not the only way to manipulate the host state.

NSX has its agent, which returns information about the networking state, and that information is returned to NSX.

Now NSX depends on what vCenter thinks is running on ESXi to make decisions about the state of the VMs running on ESXi.

So the complete state of the system is:

What NSX thinks the ESXi host thinks.
What vCenter thinks the ESXi host thinks.
What NSX thinks vCenter thinks the ESXi host thinks.

And nobody can explain precisely the consistency guarantees of what each system thinks.

And thus we have Schrödinger’s VCF.

ESXi has one point of view, vCenter another, and an NSX a third.

And most of the time, things are okay, but when they aren’t, it’s impossible for any mere mortal to tell you what exactly the state of the system is.

To demonstrate that point, when you recover a vCenter from backup, the NSX system can’t just trivially recover, because the world suddenly changed under its feet. From ESXi, it has one point of view, and from vCenter, it has an entirely different point of view. And until vCenter is reconciled with the current ESXi state, and then NSX is reconciled with the new state of vCenter, the system is in an indeterminate state.

That flaw in the system is precisely what Nutanix’s architecture eliminates and simultaneously addresses the fundamental problem of availability of the cluster and autonomy of the cluster when Prism Central is unreachable.

the architecturalist 61: recovery from backup is AWOL in most DR planning

by Leave a Comment

Photo by Markus Winkler on Unsplash


Recently, I have been noodling about the qualitative difference between Ransomware and other forms of cyber attacks.

Ransomware is fundamentally a form of reversible sabotage. When someone encrypts your data, the data is there, it’s just destroyed. When you pay them, the data becomes undestroyed.

If you have a backup that has some of the data that is unencrypted and safe, then the question becomes:

_ How much data will I lose?
_ How much will that affect my business?
_ How long will it take to recover my business given the lost data?

Because the current active data is destroyed, the first step is to determine how much data you still have. To do that, you need to restore your data from a backup.

The problem with doing a restore is that the bad guys will have penetrated several of your systems, and so if you blindly restore a backup, the bad guys will destroy that copy.

So you need to restore the backup in a safe environment.

And so a critical question in any ransomware event is – “What is the last known good backup?”

And once you have that backup, the next question is – “How long will it take to rebuild the information that has been lost?”

Having, once again, entered the enterprise storage space, what I find interesting is how DR planning and preparation doesn’t include DR preparation around “recovery from backup.”

Not from yesterday’s backup but from a backup that is 30, 60, or even 90 days old.

If you routinely wargame that problem, you’ll uncover databases that are not backed up. Yes, I know a huge company that 6 years ago had super critical production databases in a DR configuration but no backup.

But more importantly, when a ransomware event occurs, you can quickly determine whether it’s worth paying the fee.

Testing backup recovery of old backups isn’t a nice-to-have in today’s world; it’s a necessity.

Or, as I like to say it, the IT teams that practice recovery from backup will be the only ones that are currently employed.

As to why I am so invested in this topic, at Zynga, two weeks before our IPO, a game of ours got destroyed because of an operator error.

Thankfully, we were able to recover from backup in less than 12 hours.

That experience taught me to think about backup differently.

And it’s why when I say Single Point of Failure, I think about recovery from backup not a server crashing.

When I talk to IT professionals and technical leaders, this need for backup to work is not understood.

the nutanixist 14: the PSC, identity and architecture

by 1 Comment

blue and white desk globe on green grass field during daytime
Photo by Guillaume de Germain on Unsplash

When I joined VMware, the company had just released version 6.0, and along with it, the external PSC (Platform Services Controller).

The idea behind the PSC was to centralize a set of core services, upon which the rest of the product portfolio could be built, including vCenter.

The problem was availability.

vCenter 6.0 depended on the PSC to be available so you could log in.

If the PSC were down or unreachable, you would be unable to log in.

You encountered a peculiar complexity: vCenter was up, but the PSC was unreachable, and you couldn’t log into vCenter to resolve the issue with the PSC (and yes, vSphere HA can’t help).

In short, the failure domain of the PSC and vCenter was distinct, and because they were different, your infrastructure only worked when the PSC and vCenter were available and reachable.

As you add more vCenters to a PSC, in what became known as the MxN configuration, for your entire environment to work as expected, everything had to be working.

So? If your environment consists of 10 vCenters and one PSC, and the connection to the PSC is broken for any vCenter, then that vCenter isn’t functioning correctly.

So? Just use the local account.

Except, the whole point of the PSC was not to use local accounts.

Because not being able to log into vCenter was unacceptable, customers had to maintain both local accounts and PSC accounts.

So what? It’s just an occasional thing. Except it isn’t. As the number of vCenters increases, the probability of any one vCenter being unable to reach its PSC increases, which means some part of your infrastructure is always requiring you to use a local password.

Where things got messy was during the restoration from backup. The PSC didn’t just include your identities; it also included tag definitions. If you restored a PSC from backup, then from the point of view of all the vCenters, it appears to have reverted to an earlier state. And so the tag definitions could suddenly disappear. And if they did, then any operational tooling that depended on the tags would break.

From 6.5 to 7.0, the vCenter team rectified that architectural flaw. We worked to make the transition as seamless as possible, driven by one consideration: never break the fault domain between vCenter and login.

What surprised me, according to the VCF 9.0 documentation, which I probably misread, is that VCF now requires a single centralized broker and also requires local identities.

So while VCF has gone around and around on this topic, Nutanix chose a different path; instead of relying on a single global broker, each cluster and system has its own broker, which can federate with one another.

I wondered why both companies took different paths. At the core, it comes down to the fact that Nutanix considers the cluster the irreducible control plane, while VMware considers ESX. A cluster can have an identity broker, whereas every host cannot—details in architecture matter.

the nutaxinist 13: x86 virtualization may not be what you think it is, bare metal is roaring back and why you need a different platform like AHV

by Leave a Comment

I wrote this a while ago, and since then, I have learned a great deal more about what makes AHV special. And although I talk about the database here, it’s not just about the database; it’s also about the kind of OS and the availability models of that system.

Photo by Jonny Gios on Unsplash

One of the more enduring mysteries about x86 virtualization is how profoundly misunderstood it is by those who are distant from it, considering its widespread use.

For folks with a passing understanding, they assume that something intercepts every instruction between the actual processor and the workload and translates it on the fly.

Except that’s not been the case from more or less the beginning.

What VMware and most other vendors did was virtualize a processor’s control instructions, not the workload instructions.

A processor has a set of instructions and capabilities for running workloads and a set of capabilities and instructions for managing the hardware.

The OS-processor interface is peculiar and continuously evolving. By its very nature, it was initially engineered to assume that only one OS ever interacted with it.

VMware virtualized that OS-Processor interface, enabling multiple different OSs to run on the same x86 hardware.

Once the processor’s control plane was virtualized, it became possible to build an OS (ESXi) that treated VMs as first-class abstractions.

ESXi enabled far more sophisticated control and sharing of the physical resources. It could do that because the control plane was virtualized, and when it needed to interfere with the running guest, it was able to.

Nowadays, every OS does the same thing—it virtualizes the OS-processor interface and, using that abstraction, can run multiple VMs on a single processor.

Unfortunately, we take this for granted because it is an astonishing technical result, and we are too impressed with it.

And given that every OS on the planet, including many free ones, and that customers wanted to run mixed workloads and they chose to use an inferior form of virtualization, the cognitive dissonance between I like bare metal and virtualization isn’t good enough, hurt my head.

So I dug into it.

They are saying that a modern application’s control plane is a distributed system. They want a distributed infrastructure control plane on which multiple applications can rely. Virtualization does provide a mechanism for sharing a server, but that’s not useful without a distributed infrastructure control plane that applications can share.

The industry-leading virtualization does not have a clustered control plane. So, customers naturally look towards bare metal Kubernetes (K8s) because it has a distributed database.

And then the same customers use kube-virt to create VMs. The shift to bare metal is thus not about virtualization, but about what control plane virtualization you need. Today’s applications require the infrastructure control plane to be virtualized.

The next generation of infrastructure management will depend on vendors who figure out how to virtualize the interface from the K8s API server to the underlying infrastructure itself.

To achieve this, you need a distributed database that is more reliable than etcd. Why? Because etcd is what you don’t have to pay for.

Fortunately, Nutanix has one of those.