wrong tool

Very early on in my career, I read this post by Paul Graham, https://www.paulgraham.com/avg.html which argued that software architecture matters. He observed that choosing the right technology could put you on a profoundly different curve. I have always looked for such examples. Those create permanent curves, not transitory ones.

Years ago, while at Juniper, I investigated ARM processors as replacements for x86 processors in high-performance tasks.

I consulted hardware architects and CPU experts, who agreed that ARM consumed less power due to its limited features and said that as its capabilities expanded, power consumption would increase.

This was reassuring until I spoke with a kernel expert who pointed out that the Mac’s power advantage came from macOS’s design, which intentionally reduced power use per task. His analysis was credible.

When Apple released the M1-4 laptops, I tested this hypothesis. Indeed, macOS outperforms Windows PCs and surpasses x86 laptops in power efficiency.

In summary, Apple engineered a system with a fundamentally different power utilization curve than Windows, meaning Microsoft would need significant re-engineering to compete.

VMware customers often wonder about Nutanix’s simplicity. Is the system simple because it is genuinely simple, or is it engineered to be simple even as capability grows?

Many have faced the vSphere complexity curve. In 2015, deploying vSphere was so challenging that VMware engineering spent nine years improving the upgrade and lifecycle on ESX and vCenter. While vSphere can be as simple as Nutanix, VCF is not as simple as NCI.

Is Nutanix simple because it is simple? I thought so, but I was mistaken.

Nutanix’s architecture removes complexity. It features a distributed database per control plane in user space, with separate data, management, and control planes. This database is scoped to a single or multiple clusters (Prism Central).

The database functions as the data path and control plane interface, allowing Nutanix systems to switch data paths without sacrificing control features. Moreover, since the data path works in user space, it can be adapted to Kubernetes.

This is simplicity by design.

The architecture is cloud native because the founders were from Google. A cloud-native application is a distributed database with microservices, precisely what Nutanix is. Adding more management and control plane features can be done as quickly as with any modern application, and by design, the system scales. Porting the system is straightforward. And the proof is in the results.

The team quadrupled Prism Central’s scale in a few years, rebuilt the Networking Control plane, and added support for external storage and Kubernetes.A company a 5th the size of pre-Broadcom VMware to deliver a system with a comparable feature set to VCF.

Nutanix announced external storage support for its platform.

And for those who live in the vSphere bubble, it feels a lot like “whatever” or, more boringly, “Really?”

It’s about perspective. Nutanix has delivered Raghu’s vision of an SDDC in a way VMware and VCF have not.

When Raghu stood before everyone in 2012 to argue for SDDC, he envisioned a software control plane that would allow infrastructure programmability.

Realizing that vision forced Nutanix and VMware to focus on limiting customer choice to provide simplicity. It was a necessary first phase of the market.

VMware and Nutanix adopted a solution in which customers had to back off their choice of hardware to gain the benefits of the SDDC. The software systems imposed various constraints on workloads and on data center construction.

VMware made some progress with VCF, but VCF only fulfills that promise with a very brittle system and a complex set of software choices that only partially realize the potential.

VCF’s answer to the need for customer choice was to provide that choice and simultaneously sacrifice the simplicity and vision of SDDC. You want choice, you get choice, but the integration of the system is sacrificed.

VCF software architecture forces customers to either not use a feature or not get the value of VCF.

The failure of VCF and Nutanix did not mean that the market demand did not exist for an SDDC that delivers programmability of infrastructure, choice, and availability.


And that’s what Nutanix has delivered. The Nutanix system uses Dell Powerflex storage, which gives customers a consistent operating model across SDDCs and hardware choices without sacrificing features.

Nutanix, because the software system is built using a distributed database with microservices that interact on a consistent data model, has—again—taken a better approach.

VCF forces you to choose between the simplicity of vSAN, the hardware choices of VMFS, and the complexity of SDDC manager and NSX while requiring new hardware and software infrastructure to deploy any of these. Furthermore, the feature set of vSphere is artificially constrained in a VCF deployment due to the limitations of the overall solution.

Nutanix offers an elegant model that works fantastically for HCI and now for external storage. Customers now possess a software platform that delivers the SDDC and is not tied to a specific hardware architecture.

External storage with all the availability, simplicity, and advantages of HCI. Because it’s Nutanix, it also offers the availability and recoverability that such a system requires.

Most importantly, Nutanix has delivered an infrastructure that allows applications more choice in the hardware trade-offs they need while retaining a consistent interaction model.

Calling what Nutanix did – “external storage support” – is like calling the telephone a device to exchange pleasantries. It’s both accurate and misses the change.

Since working at Zynga, I have been trying to build a clustered infrastructure control plane for on-prem infrastructure to simplify application development.

As I dug into understanding Prism Element and Prism Central’s architecture, it struck me that my holy grail, building an easy-to-use and configured clustered control plane, was a solved problem. Nutanix had what I was trying to build.

Why was it my holy grail?

A simple principle of software engineering is that it’s easier to write code on top of robust systems than on top of fragile systems. If the system is robust, then it’s forgiving of programming errors. A great example is performance; your code doesn’t have to be as efficient if your hardware is fast enough.

If that intuition doesn’t work, when you write code, you introduce bugs. It’s much easier to debug the new code you just wrote than debug the code you use that you may have no access to.

If that intuition doesn’t work, consider a server running many applications and then installing a new one. It’s much easier to debug the availability of the latest application if the server without the application has predictable availability.

Traditional applications that run in a single VM rely on the OS to be more reliable than their application. So, when their application crashes, they can debug it in isolation.

One of the astonishing engineering outcomes of the last twenty years was the emergence of hypervisors that are more reliable than the OS running the application.

That enabled significant consolidation, engineering efficiency, and operational efficiency.

The challenge with clustered applications is that they have no analog to the hypervisor that manages their infrastructure.

As a result, the application cluster control plane manages the application and the infrastructure.

The recent re-infatuation with bare metal is a side effect of this problem. If the virtualization layer can’t offer a clustered control plane, then the application must. And if the application can do it, why do I need the virtualization layer? Virtualization provides a container on a server. That container and the server are disposable. So why am I paying money and cost?

If your hypervisor doesn’t have a clustered control plane, you may be right.

And since I assumed Nutanix didn’t have one, I planned to build one here.

Imagine my surprise when I discovered that it has been in shipping for 10 years.

Part of the elegance of the Nutanix VDI solution is the Prism Element’s clustered control plane.

Using Nutanix infrastructure, you can rely on the underlying infrastructure control plane to be more reliable than the application control plane. More importantly, you can offload some of the complexity of the application control plane. Most importantly, you can share your infrastructure easily.

Welcome to the future.

I was chatting with one of the many brilliant folks I’ve worked with over the years about what makes Nutanix easy. As we talked, it hit me: If you reduce the operational challenge of managing all of the software to manage your enterprise to just managing a database, the operational challenges are much simpler.

Why is that even theoretically possible?

An enterprise infrastructure system has three elements:

1. The software that interacts with the physical infrastructure (the data path)
2. Meta data that describes that infrastructure.
3. Stateless software that interacts with the metadata to configure and monitor the physical infrastructure (the control and management)

A fundamental problem for any enterprise infrastructure system is protecting metadata from being lost.

At the enterprise scale, the operational challenge is that the more you want to do with software, the more you invest in operationalizing the software, which means the less money and time you have to do stuff.

You spend so much time setting up software that it bottlenecks you from doing more things.

Databases are a great example. Standing up databases and getting them to work, backing them up, and setting up DR get in the way of actually using a database.

RDS addressed that operational challenge, and now, more folks in an enterprise use databases than they would otherwise.

Nutanix solved the really hard problem of building a distributed database. And can now reap what it has sown.

Nutanix, by encapsulating all infrastructure metadata into a reliable, scalable, and transparent database, reduced the most complex operational problem of metadata management to a database operations problem.

The operational challenges of a single database can be solved. It’s not easy, but it can be.

So, when you look at Nutanix, you see very small customers deploying SDNs. Why? Because the deployment’s operational complexity is slight. Why? Because managing the SDN’s metadata database is easy.

It’s easy not because the SDN system has its own database and the rest of the control system has another one but because they are—literally—the same one.

Solving operational challenges for one feature eliminates the need to set up another database and address the operational problems associated with it, as well as the challenges of maintaining two databases in sync.

And so the complexity of deploying the SDN is significantly lower. This makes the consumption of the capability less costly and easier to do.

And the value of an SDN is real.

And if you’re in the business of running a business, shouldn’t you be spending your time using stuff instead of figuring out how to deploy it?

Everyone has a take on their company’s magical mystery sauce. For me, it’s without a doubt the distributed database that underpins Nutanix’s product’s storage, management, and control planes.

At its most reductionist core, a cloud is a distributed database with microservices that implement business logic and an IO path.

The hard problem in the cloud is building a distributed database that is transparent to the microservices’ consumers.

The problem is so complex that the public cloud providers have struggled to bring their platforms to on-prem.

The complexity is along three dimensions: the first is building such a thing, the second is getting it to work, and the third is making it work within a large number of unique customer deployments where the customer does the deployment and life cycle.

The cloud vendors solved the problem by owning the database’s deployment and lifecycle. They could control the hardware and software deployment and, by doing that, achieved astonishing scale and availability. They also have a tremendously sophisticated engineering team that can operate those systems at scale.

Nutanix, because of its origin in HCI, built a clustered database, and you can find out more about it here: nutanixbible.com

That database has then become the basis of their control and management planes.

Because of its origins, that database is—from the management software perspective—infinitely available and infinitely reliable. Its existence is transparent to the customer.

Working at Nutanix, I am struck by the fact that, unlike every other competitor in the space, our system is architected from the ground up to be a control and management plane for the cloud.

That difference delivers real business value. For example, backup and restore of the control and management plane are trivial. Recovery is plausible with a minimum of fuss. That is telling when you compare it to other enterprise-class products, especially other infrastructure products.

Because of how the system is built, you need one backup of one system to get all of the microservices’ states. Restoring is done with a single point-in-time copy, so you don’t need to restore multiple different databases.

Or consider DR and HA; there is precisely one way to do it, and it works for all services.

For systems like backup, DR, and HA, the complexity of the backup/recovery process, the HA process, or the DR process intrinsically affects the system’s availability; Nutanix has a shockingly good system.

In fact, when I joined, I was stunned by how good it was.

It’s the power of that fundamental core architectural building block that makes Nutanix a magical platform.