the nutanixist 15: schroedinger’s VCF

In my previous post, I talked about irreducible control planes. And hand-waved their implications, but those implications explain why Nutanix is so fundamentally different.

An irreducible control plane is the minimal control plane that must exist for a service to exist. For VMs on ESX, that’s ESX. With the ESX host running and access to the ESX host, you can perform any actions you want on a VM.

Every other control plane adds value but is not essential.

vCenter is an optional system. And thus, the first peculiar part of vSphere is the split brain between vCenter and ESXi. vCenter does not control ESXi entirely. It has to account for the possibility that the state of an ESXi host changed while it was running.

In effect, vSphere states that if ESX is running and you can authenticate to the ESXi host, you have complete control of the system, even if all other software is not running. It also states that you can always log into the ESXi host and take action, even if all other systems are running.

Thus, ESXi and vCenter are permanently running in a split-brain mode, and it’s vCenter’s job to react to what happened on ESXi.

The design principle is a good one. That’s not a bad design principle. And I have argued that a singular design principle is why ESXi is the best system for running traditional single VM legacy workloads, where all you care about is running the VM.

But if you care about doing anything other than the VM running, then how vCenter knows about the state of the ESXi host is super interesting. If vCenter and ESXi have different views of the state of the host, and those views get out of sync, what happens?

For example, when an ESXi host gets disconnected from vCenter, and a host in the cluster that the ESXi failed, and vSphere HA starts a VM on that disconnected ESXi host, what happens?

What does vCenter know and when did it know it?

And if vCenter doesn’t know and you can’t explain what it knows and when it knows it, how does a system layered on vCenter know what vCenter knows?

So, for example, NSX relies on vCenter to know things about the ESXi host. And if vCenter has wrong information, then NSX has bad information.

And maybe that -might- be okay, if NSX and vCenter had consistent but wrong information.

But they don’t.

vCenter talks to ESXi via hostd and vpxa. And those agents don’t return all the information on a host. And they are certainly not the only way to manipulate the host state.

NSX has its agent, which returns information about the networking state, and that information is returned to NSX.

Now NSX depends on what vCenter thinks is running on ESXi to make decisions about the state of the VMs running on ESXi.

So the complete state of the system is:

What NSX thinks the ESXi host thinks.
What vCenter thinks the ESXi host thinks.
What NSX thinks vCenter thinks the ESXi host thinks.

And nobody can explain precisely the consistency guarantees of what each system thinks.

And thus we have Schrödinger’s VCF.

ESXi has one point of view, vCenter another, and an NSX a third.

And most of the time, things are okay, but when they aren’t, it’s impossible for any mere mortal to tell you what exactly the state of the system is.

To demonstrate that point, when you recover a vCenter from backup, the NSX system can’t just trivially recover, because the world suddenly changed under its feet. From ESXi, it has one point of view, and from vCenter, it has an entirely different point of view. And until vCenter is reconciled with the current ESXi state, and then NSX is reconciled with the new state of vCenter, the system is in an indeterminate state.

That flaw in the system is precisely what Nutanix’s architecture eliminates and simultaneously addresses the fundamental problem of availability of the cluster and autonomy of the cluster when Prism Central is unreachable.