wrong tool

I recently had a conversation with a friend of mine about a company he worked for years ago that had no backups for its production environment.

The team argued that they didn’t need backups because they had synchronous mirroring.

The person pulled their hair out and asked, “So if a table gets corrupted, what then?”

What I find odd in the IT industry is that we think of SPOF as a hardware failure.

With modern systems, the total outage after a hardware failure lasts only minutes. Although that’s unfortunate, it’s not disastrous.

In contrast, recovering from a backup can take days.

Furthermore, recovering from a backup can result in data loss.

Therefore, the cost, complexity, and downtime associated with backups are so high that people consider backups the most crucial part of their availability strategy.

But they don’t.

I would say, “But this is a SPOF,” and folks responded, “Well, that is 5 minutes of downtime, not a big deal.”

And I realized that what I am saying is, “Since this cannot be restored from backup, we are one human mistake away from destroying all of this infrastructure and taking multiple days or hours to recover.”

When I became the architect of VCF, I saw a system that could not be restored from backup without intensive support from VMware customer support. A typical VCF instance comprises two NSX deployments with their respective databases, an SDDC manager with its database, and several vCenters, each with its internal database. Even worse, upon examining the products, they contain multiple internal databases and configuration files. Products like the supervisor, Operations, and Automation are further dependent on all these systems having the same view of the state to work correctly.

While I was the architect of vCenter, I ensured that backups worked effectively. I reviewed and examined the file-based restore feature. Every feature had to explain what would happen after a restore. I led the effort to make MOIDs stable so that, after a backup, VMs retain their original IDs. I pushed as hard as possible to get a DKVS so that the restore would work without breaking clusters.

What VCF has can be made to work. There is a prodigious amount of research in distributed systems that would allow this to work. However, simply reading the documentation, asking Google, and consulting Reddit will reveal how fragile the current system is.

What astonishes me about Nutanix is that Prism Central is routinely restored from backup by customers. Because it’s so easy to restore from backup, customers will restore from backup rather than try to figure out what broke.

Does that mean Prism Central always works? No. However, there is a qualitative difference between something that cannot work without any painful intervention and something that primarily works.

Why does Prism Central work? Because the team did the hard systems work that takes years to implement.

It is possible to architect a system that can recover from a backup. It is possible to build a system that minimizes data loss. The DKVS was part of such a system.

It’s just hard and takes time.

What astonished me is that Prism Central has such a system.

I joined Nutanix because of great people, inspiring leadership, a solid business model, innovative technology, and a lovely 15-minute commute.

In my early days here, I was a bit puzzled about how we managed to thrive, especially considering the significant energy I had previously devoted to trying to put Nutanix out of business.

Everything changed when I had the chance to meet the CTO Ambassadors. During a conversation, Joe Garvey shared his insights about how Nutanix customers often restored from backup too quickly instead of tackling the challenges they faced with Prism Central.

I was genuinely taken aback.

It was amusing because, from my viewpoint, the ability to restore from backup felt remarkable. The Nutanix team seemed a bit puzzled, unsure why “restoring” from backup was such a significant consideration.

This experience launched me on a fascinating journey of discovery, revealing just how incredibly special Nutanix truly is.

Their management and control plane is a cloud-native application, easily packaged and installed by customers on-premises.

And I have talked about this before (the nutanixist 05 – Nutanix is the RDS of your enterprise. and the nutanixist 08: Is it simple because it’s simple, or because it’s always engineered to be simple? A parable about ARM and Nutanix, or why Paul Graham is right. and the nutaxinist 04: the magical distributed database)

So, what truly defines something as cloud-native? The essential components include scalability, seamless upgrades, no single points of failure, the use of microservices, and an API-centric architecture. Prism Central beautifully incorporates all of these features.

As I delved into the system, I uncovered a very contemporary application architecture that, in some aspects, seemed ahead of its time compared to the Kubernetes platform design. However, the challenge we faced was that the industry ecosystem had evolved alongside Nutanix, making it practical to shift to the standard platform to optimize our investments in the broader ecosystem.

As a result, we recognized the need to replatform our products. In our case, replatforming was a monumental effort, but it wasn’t about rewriting everything. From a business logic perspective, it ended up being mostly transparent.

So, what benefits do we gain?

Today, Prism Central stands as a single-scale-out cloud-native application that can scale to three VMs. With the introduction of the new disaggregated platform, we successfully migrated some Prism Central services to their own Kubernetes cluster, allowing us to scale these services independently.
Just like with Prism Central, our customers don’t need to be platform administrators; we utilize Kubernetes to keep the platform experience seamless and transparent.

What’s even more remarkable is that we didn’t need to entirely shift to Kubernetes to harness all of that value.

It’s truly impressive to think that Nutanix managed to evolve from a modern yet somewhat outdated platform to a newer one in just two years. This transformation speaks volumes about the product architecture and its alignment with future advancements, rather than working against them.

I made an audacious claim about Nutanix: they had the first real SDDC when they added external storage.

Of course, many folks quickly and correctly pointed out the large feature gaps between Nutanix’s offerings and VMware’s. vSphere supports a broader set of storage offerings, NSX has a wider feature set, and vCenter has folders.

Having been the architect behind vCenter (versions 6.5, 7.0, and 8.0, as well as VCF 4.x up to 5.0), I really have a good understanding of the gaps between these products.

My time at VMware was spent trying to build a real SDDC. I have a long list of attempts, code words, and projects I pushed, some with success and some without, within VMware.

I was highlighting another perspective. To me, an SDDC offers a cohesive collection of services that simplifies the physical infrastructure beneath it. This means that, no matter the hardware in use, I can operate consistently and seamlessly. While various hardware might bring a few additional features, my team’s operations remain unchanged and steady.

vSphere’s approach to storage involves creating a new filesystem for each type of storage. vSphere includes VMFS for external storage vSphere features a distinct NFS client that operates differently than VMFS for NFS storage. Additionally, vSphere has vSAN to enhance the performance and capacity of both flash and HDD. Lastly, vSphere introduced another filesystem, VSAN ESA, designed to meet the performance requirements of NVME drives.

Each point product was brilliant, feature-rich, and incompatible, requiring forklift upgrades.

Nutanix took its entire SDDC feature set and ran it on top of external storage. It didn’t create a new file system, data management layer, API, etc.; it just added another storage type.

That simplicity adds business value. What makes it shocking, to me at least, is that they did it in a record amount of time with very few resources.

As a Nutanix customer, incremental new hardware capabilities do not create pools of incompatible infrastructure. It all works the same way; the different hardware just gives you different choices.

There is power and value in that. And that Nutanix could do that with two radically different kinds of storage – HCI and External Storage was incredible.

A while ago, someone asked the question: given that VMware and Nutanix have similar capabilities, what makes Nutanix the only real SDDC?

It was a great question. And to be quite honest, I was stumped. What was it? Was I being a fan of my employer, or was there something there? And I came up with the following answer –

It’s about the vision and the reality.

If you aim to assemble pieces of software, orchestrate them, and deal with their complexity, then you’re trading off hardware complexity for software complexity.

If your goal is a control plane that absorbs different hardware infrastructure, has a simple-to-deploy and operate model, and offers a consistent API and user experience, that’s the SDDC.

And yes – if you are willing to fight with the complexity of the VCF stack and work around its limitations, you can make it do incredible things.

But that wasn’t the SDDC that I envisaged.

That isn’t programmable, dynamic, flexible infrastructure.

Worse, VCF’s intrinsic limitations of the availability of the control plane make it a poor solution when the control’s availability defines the infrastructure’s availability.

SDDC is more than just a bag of features you can script together. It’s an operating model of infrastructure that is programmable and works. Nutanix delivers on that.

At VMware, I passionately argued for such a system. And ultimately, the company had other priorities.

VCF is on a long journey to get there.

Nutanix is on a long journey to add the features.

Very early on in my career, I read this post by Paul Graham, https://www.paulgraham.com/avg.html which argued that software architecture matters. He observed that choosing the right technology could put you on a profoundly different curve. I have always looked for such examples. Those create permanent curves, not transitory ones.

Years ago, while at Juniper, I investigated ARM processors as replacements for x86 processors in high-performance tasks.

I consulted hardware architects and CPU experts, who agreed that ARM consumed less power due to its limited features and said that as its capabilities expanded, power consumption would increase.

This was reassuring until I spoke with a kernel expert who pointed out that the Mac’s power advantage came from macOS’s design, which intentionally reduced power use per task. His analysis was credible.

When Apple released the M1-4 laptops, I tested this hypothesis. Indeed, macOS outperforms Windows PCs and surpasses x86 laptops in power efficiency.

In summary, Apple engineered a system with a fundamentally different power utilization curve than Windows, meaning Microsoft would need significant re-engineering to compete.

VMware customers often wonder about Nutanix’s simplicity. Is the system simple because it is genuinely simple, or is it engineered to be simple even as capability grows?

Many have faced the vSphere complexity curve. In 2015, deploying vSphere was so challenging that VMware engineering spent nine years improving the upgrade and lifecycle on ESX and vCenter. While vSphere can be as simple as Nutanix, VCF is not as simple as NCI.

Is Nutanix simple because it is simple? I thought so, but I was mistaken.

Nutanix’s architecture removes complexity. It features a distributed database per control plane in user space, with separate data, management, and control planes. This database is scoped to a single or multiple clusters (Prism Central).

The database functions as the data path and control plane interface, allowing Nutanix systems to switch data paths without sacrificing control features. Moreover, since the data path works in user space, it can be adapted to Kubernetes.

This is simplicity by design.

The architecture is cloud native because the founders were from Google. A cloud-native application is a distributed database with microservices, precisely what Nutanix is. Adding more management and control plane features can be done as quickly as with any modern application, and by design, the system scales. Porting the system is straightforward. And the proof is in the results.

The team quadrupled Prism Central’s scale in a few years, rebuilt the Networking Control plane, and added support for external storage and Kubernetes.A company a 5th the size of pre-Broadcom VMware to deliver a system with a comparable feature set to VCF.

Nutanix announced external storage support for its platform.

And for those who live in the vSphere bubble, it feels a lot like “whatever” or, more boringly, “Really?”

It’s about perspective. Nutanix has delivered Raghu’s vision of an SDDC in a way VMware and VCF have not.

When Raghu stood before everyone in 2012 to argue for SDDC, he envisioned a software control plane that would allow infrastructure programmability.

Realizing that vision forced Nutanix and VMware to focus on limiting customer choice to provide simplicity. It was a necessary first phase of the market.

VMware and Nutanix adopted a solution in which customers had to back off their choice of hardware to gain the benefits of the SDDC. The software systems imposed various constraints on workloads and on data center construction.

VMware made some progress with VCF, but VCF only fulfills that promise with a very brittle system and a complex set of software choices that only partially realize the potential.

VCF’s answer to the need for customer choice was to provide that choice and simultaneously sacrifice the simplicity and vision of SDDC. You want choice, you get choice, but the integration of the system is sacrificed.

VCF software architecture forces customers to either not use a feature or not get the value of VCF.

The failure of VCF and Nutanix did not mean that the market demand did not exist for an SDDC that delivers programmability of infrastructure, choice, and availability.


And that’s what Nutanix has delivered. The Nutanix system uses Dell Powerflex storage, which gives customers a consistent operating model across SDDCs and hardware choices without sacrificing features.

Nutanix, because the software system is built using a distributed database with microservices that interact on a consistent data model, has—again—taken a better approach.

VCF forces you to choose between the simplicity of vSAN, the hardware choices of VMFS, and the complexity of SDDC manager and NSX while requiring new hardware and software infrastructure to deploy any of these. Furthermore, the feature set of vSphere is artificially constrained in a VCF deployment due to the limitations of the overall solution.

Nutanix offers an elegant model that works fantastically for HCI and now for external storage. Customers now possess a software platform that delivers the SDDC and is not tied to a specific hardware architecture.

External storage with all the availability, simplicity, and advantages of HCI. Because it’s Nutanix, it also offers the availability and recoverability that such a system requires.

Most importantly, Nutanix has delivered an infrastructure that allows applications more choice in the hardware trade-offs they need while retaining a consistent interaction model.

Calling what Nutanix did – “external storage support” – is like calling the telephone a device to exchange pleasantries. It’s both accurate and misses the change.

Since working at Zynga, I have been trying to build a clustered infrastructure control plane for on-prem infrastructure to simplify application development.

As I dug into understanding Prism Element and Prism Central’s architecture, it struck me that my holy grail, building an easy-to-use and configured clustered control plane, was a solved problem. Nutanix had what I was trying to build.

Why was it my holy grail?

A simple principle of software engineering is that it’s easier to write code on top of robust systems than on top of fragile systems. If the system is robust, then it’s forgiving of programming errors. A great example is performance; your code doesn’t have to be as efficient if your hardware is fast enough.

If that intuition doesn’t work, when you write code, you introduce bugs. It’s much easier to debug the new code you just wrote than debug the code you use that you may have no access to.

If that intuition doesn’t work, consider a server running many applications and then installing a new one. It’s much easier to debug the availability of the latest application if the server without the application has predictable availability.

Traditional applications that run in a single VM rely on the OS to be more reliable than their application. So, when their application crashes, they can debug it in isolation.

One of the astonishing engineering outcomes of the last twenty years was the emergence of hypervisors that are more reliable than the OS running the application.

That enabled significant consolidation, engineering efficiency, and operational efficiency.

The challenge with clustered applications is that they have no analog to the hypervisor that manages their infrastructure.

As a result, the application cluster control plane manages the application and the infrastructure.

The recent re-infatuation with bare metal is a side effect of this problem. If the virtualization layer can’t offer a clustered control plane, then the application must. And if the application can do it, why do I need the virtualization layer? Virtualization provides a container on a server. That container and the server are disposable. So why am I paying money and cost?

If your hypervisor doesn’t have a clustered control plane, you may be right.

And since I assumed Nutanix didn’t have one, I planned to build one here.

Imagine my surprise when I discovered that it has been in shipping for 10 years.

Part of the elegance of the Nutanix VDI solution is the Prism Element’s clustered control plane.

Using Nutanix infrastructure, you can rely on the underlying infrastructure control plane to be more reliable than the application control plane. More importantly, you can offload some of the complexity of the application control plane. Most importantly, you can share your infrastructure easily.

Welcome to the future.

I was chatting with one of the many brilliant folks I’ve worked with over the years about what makes Nutanix easy. As we talked, it hit me: If you reduce the operational challenge of managing all of the software to manage your enterprise to just managing a database, the operational challenges are much simpler.

Why is that even theoretically possible?

An enterprise infrastructure system has three elements:

1. The software that interacts with the physical infrastructure (the data path)
2. Meta data that describes that infrastructure.
3. Stateless software that interacts with the metadata to configure and monitor the physical infrastructure (the control and management)

A fundamental problem for any enterprise infrastructure system is protecting metadata from being lost.

At the enterprise scale, the operational challenge is that the more you want to do with software, the more you invest in operationalizing the software, which means the less money and time you have to do stuff.

You spend so much time setting up software that it bottlenecks you from doing more things.

Databases are a great example. Standing up databases and getting them to work, backing them up, and setting up DR get in the way of actually using a database.

RDS addressed that operational challenge, and now, more folks in an enterprise use databases than they would otherwise.

Nutanix solved the really hard problem of building a distributed database. And can now reap what it has sown.

Nutanix, by encapsulating all infrastructure metadata into a reliable, scalable, and transparent database, reduced the most complex operational problem of metadata management to a database operations problem.

The operational challenges of a single database can be solved. It’s not easy, but it can be.

So, when you look at Nutanix, you see very small customers deploying SDNs. Why? Because the deployment’s operational complexity is slight. Why? Because managing the SDN’s metadata database is easy.

It’s easy not because the SDN system has its own database and the rest of the control system has another one but because they are—literally—the same one.

Solving operational challenges for one feature eliminates the need to set up another database and address the operational problems associated with it, as well as the challenges of maintaining two databases in sync.

And so the complexity of deploying the SDN is significantly lower. This makes the consumption of the capability less costly and easier to do.

And the value of an SDN is real.

And if you’re in the business of running a business, shouldn’t you be spending your time using stuff instead of figuring out how to deploy it?

Everyone has a take on their company’s magical mystery sauce. For me, it’s without a doubt the distributed database that underpins Nutanix’s product’s storage, management, and control planes.

At its most reductionist core, a cloud is a distributed database with microservices that implement business logic and an IO path.

The hard problem in the cloud is building a distributed database that is transparent to the microservices’ consumers.

The problem is so complex that the public cloud providers have struggled to bring their platforms to on-prem.

The complexity is along three dimensions: the first is building such a thing, the second is getting it to work, and the third is making it work within a large number of unique customer deployments where the customer does the deployment and life cycle.

The cloud vendors solved the problem by owning the database’s deployment and lifecycle. They could control the hardware and software deployment and, by doing that, achieved astonishing scale and availability. They also have a tremendously sophisticated engineering team that can operate those systems at scale.

Nutanix, because of its origin in HCI, built a clustered database, and you can find out more about it here: nutanixbible.com

That database has then become the basis of their control and management planes.

Because of its origins, that database is—from the management software perspective—infinitely available and infinitely reliable. Its existence is transparent to the customer.

Working at Nutanix, I am struck by the fact that, unlike every other competitor in the space, our system is architected from the ground up to be a control and management plane for the cloud.

That difference delivers real business value. For example, backup and restore of the control and management plane are trivial. Recovery is plausible with a minimum of fuss. That is telling when you compare it to other enterprise-class products, especially other infrastructure products.

Because of how the system is built, you need one backup of one system to get all of the microservices’ states. Restoring is done with a single point-in-time copy, so you don’t need to restore multiple different databases.

Or consider DR and HA; there is precisely one way to do it, and it works for all services.

For systems like backup, DR, and HA, the complexity of the backup/recovery process, the HA process, or the DR process intrinsically affects the system’s availability; Nutanix has a shockingly good system.

In fact, when I joined, I was stunned by how good it was.

It’s the power of that fundamental core architectural building block that makes Nutanix a magical platform.