Over the last 20 years, I have been involved with or lead several rewrites of large systems. And that experience has taught me some basic rules of how to successfully do a rewrite. And they are pretty simple.

1. The business reasons have to make sense to every engineer and the business teams and everyone has to believe them.
2. The business leaders had to be committed to the rewrite. The bigger the scope of the rewrite, the more important the business leader. If you are rewriting the entire product, then the CEO has to be committed.
3. Once the plan was agreed upon, the entire team had to be working on the plan, and we had to deliver business value as soon as possible.
4. Optimize for the right strategic software architecture over the right operational or tactical software architecture.

The first rewrite I was peripherally involved in was a project at SGI called Cellular IRIX. Cellular IRIX was an attempt to build a multi-kernel single memory address space system to support SGI’s new cc-NUMA architecture.

The rewrite failed because there were too many constituencies who opposed the rewrite and too many business leaders who didn’t understand the value leaving too many engineers confused about why they should sign up for this.

When SGI imploded and some key engineering directors got fired, the project died a miserable death.

The second rewrite I was involved in as an individual contributor was the rewrite of the NetCache product and that was a successful rewrite.

The original version of NetCache was a port of software written by Internet Middleware Corporation. The system suffered from some flaws, flaws that were very hard to fix. If my memory serves me well, it was built on a callback-based system that had unclear and uncertain rules about how resources were being managed. There were if-statements littered throughout the code where function writers were expected to understand all paths that could result in a message with the state calling their function. In effect, to understand how to write a leaf function you needed to understand all possible states of the global system.

Although a technical mess, the reasons the rewrite succeeded have little to do with that. The technical mess and the near impossibility of fixing the technical mess were not why we successfully rewrote NetCache; it’s why we wanted to rewrite NetCache.

We were able to successfully rewrite NetCache because the performance and availability and feature velocity were a serious business problem and there were no alternatives on the table, and yet that wasn’t enough.

The real reasons are that

1. Everyone was working on the rewrite.
2. We knew that our new hardware could not be fully utilized using the old architecture
3. There were no constituencies in favor of fixing the old code base. The product managers didn’t want, the engineers didn’t want it and the customers didn’t want it.
4. Everyone from the GM down was committed to the rewrite
5. The code base caused business problems everyone understood.
6.  The investment in the rewrite was 2x the total man years in the original code base.

After the NetCache rewrite, the next failed rewrite was of ONTAP following the Spinnaker acquisition.

NetApp decided to buy a great company with a great engineering organization, called Spinnaker. Spinnaker had a clustered file system and an embedded namespace virtualization and worked. Unfortunately, post-acquisition we failed to properly deliver an integrated product. Instead, we delivered a new version of ONTAP, ONTAP-GX that was not compatible with ONTAP. And then after we delivered that product, started another attempt to re-integrate the Spinnaker technology into ONTAP, an effort that became known as ONTAP 8.0

There were a lot of reasons why this effort failed, and I was in the periphery of the effort. And it’s tempting to point fingers at people I respect a lot. And so I won’t. Because tactical or operational considerations of the failure are of no interest to this blog. What is of interest, is what was the strategic software context that doomed this effort from the get-go?

At the time of the acquisition, the kind of performance and scale Spinnaker offered were of interest to a tiny part of the overall market. More importantly, the kind of virtualization Spinnaker offered – namely global namespaces was of less interest.

By 2004, the entire storage industry had decided that the FC and it’s evil step child iSCSI was the most important protocol on the planet because structured data was the growth business. And structured data lived in databases.

And NetApp’s business problem was to figure out how to address FC and iSCSI not how to insert namespace virtualization. And more importantly – at the time – there was little interest in core storage innovation.  Dave Hitz, the EVP, and founder of NetApp, wrote a future history paper that said as much.

NetApp’s business problem was how do they sell more of what they had to more different customers, not a new storage array.

And every engineering manager, director, and engineer knew this. And so we had a constituency at NetApp develop that said – sure I’ll invest in this new system and at the same time, I will invest in the old system. I know about this alternative plan, because as an architect for the storage management team, I was 100% aligned to wait for the ONTAP rewrite to fail.

The original plan called for all of the engineering team to pivot to the rewrite. And then the pivot didn’t happen. And the pivot didn’t happen because when problems occurred with our core business, investing in a rewrite that wasn’t business critical became a luxury. And so the rewrite got starved of resources.

At the core, if I can use 20/20 hindsight, the mistake at a strategic software architecture perspective was the decision to have two teams, one working on the original product and one working on this rewrite. The core team felt that they were stuck doing sustaining and the rewrite team didn’t have enough resources to compete.

And the core manifestation of this separation was that the new ONTAP had a different build system and source code repository than the original ONTAP.

Much like Cellular IRIX, there were too many people looking outside in, and trying to find ways to cause it to fail.

The company more or less figured this out after ONTAP GX failed in the market. They realized that the only way to go forward was with one ONTAP that the entire company owned making it more cluster aware. And it didn’t hurt that by then; people felt that that the kind of storage system that ONTAP 8.0 was building was something that could be market leading.

The rest, as they say, is history.

The next rewrite I was involved in was the rewrite of Zynga’s backend,  something I lead.

The business goal of the rewrite was quite clear. Zynga wanted to be in the 3rd party market and to do that we need to offer 3rd party games services. We had services, but the API for using them were PHP libraries, and that wouldn’t work. So we had to offer  services that they could consume via APIs  from their apps that didn’ t have to be in our data center.

And so we decided to take all of our systems and export them over a network API to the world. And this became Zynga’s 3rd party platform that lived a short life, but the API infrastructure turned out to be very valuable for something else, and that was mobile gaming. Later on what we built got heavily modified and re-architected, and I like to think that initial effort, a project called Darwin lead the way.

From my lessons of ONTAP and NetCache, I took three key lessons

1. There had to be a compelling business value that was non-technical.
2. The business leaders had to be committed to the project, not the technologists. There could be no way for someone to appeal to some exec higher in the chain to reverse course.
3. Once the plan was decided, we had to go very hard very fast with the entire team and deliver business value as soon as possible.

And that’s more or less what we did. I remember sitting in a room with Cadir, the CTO, and owner of all central engineering, where we made the decision to do this. And I told him I need 18 months, and he told me I had six. And then I remember telling him we needed everyone to work on this or not to bother at all, and he said – let do this. I continue to admire him for that level of commitment.

And to make sure everyone knew he was personally committed to this effort, Cadir lead an all hands where he personally said that this new effort to rearchitect the backend was something that he was committing the entire team to do.

We then had a series of architectural meetings where we figured out how to solve the fundamental problems of making our services available to the internet. At the time, we chose systems that were already in existence over writing new ones. Our goal was to have a working system in a quarter and something we could sell in six months.

Those meetings included every architect and constituency, and ultimately decisions were made that pissed people off. Some that were painful for me later on in my career, as I ran over people when I could have just listened and taken them along for the journey.

We did succeed in an environment where people were quitting post-IPO and then quitting because our business was suffering.

And by success, I don’t mean what we shipped was the right software, I mean we shipped the right software architecture. And the organization was reorganized around the idea of delivering services through APIs that were centrally managed and accounted for.

And that brings me to rule #4 – Optimize for architecture not for software implementation.

Because Cadir forced me to go fast, we couldn’t figure out the right thing to do in all cases. He forced me to prioritize what we needed to accomplish and what we needed to accomplish was get the APIs on the internet fast, not have the best pieces of software to do that.

The point he taught me and made clear is that we can always improve a shipping product, we can’t improve a project that got canceled. And that if the architecture is put together right, then the bits can be enhanced over time.

And because architecture and org structure are the same things, once you reorganize to implement an architecture, you can quit the company. The company will naturally improve the architecture if the business problem remains. And that happened.

 

 

 

 

 

 

 

 

 

 

 

 

My favorite song in Hamilton is the “Room where it happens.” And my favorite part of the song is this part:

I
Wanna be in
The room where it happens
The room where it happens
I
Wanna be in
The room where it happens
The room where it happens

And the reason is that I had a similar experience in my career, which served to drive all of the rest of my professional success.

It was in 2002, at NetApp. Chris Wagner was the CTO of the NetCache product group and called a meeting of all of the senior engineers who worked on NetCache. And I wasn’t invited. And I remember standing outside of that room, looking in and wanting to be there, inside.

And for the next several years, I struggled to figure out how to get into that room. And I succeeded, and only after I succeeded, I realized I didn’t want to be in the room. Because I realized that the room where it happens wasn’t the room I wanted to be in.

What I wanted to be was George Washington.

See when Thomas Jefferson, James Madison, and Alexander Hamilton walk into the room, they are debating options that George Washington was okay with. George Washington had created a strategic framework that they had to operate within.

For example, if George Washington wanted Alexander’s plan to come to fruition, he would have pushed for the plan himself instead of sending his annoying right-hand man to negotiate with James Madison and Jefferson.

Similarly, he didn’t care if the capital was in New York or Virginia. If he had cared, then the topic would have been resolved much earlier, with Washington’s intervention.

In short, George Washington gave the folks in the room where it happens a set of choices that they could make, and he was okay any decision they made.

Strategic Software Architecture done right is about ensuring that any tactical, operational software decision is immaterial and doesn’t affect the long-term strategy allowing individuals to make choices on-their-own that still ultimately produce the right final outcomes. In effect, every decision that gets made is one you are okay with, so you don’t have to be in the room where it happens.

 

 

In 2003, I stopped being a systems software engineer and joined the Manageability team at NetApp. My then boss, Nawaf Bitar, had orchestrated a re-org to absorb that team under him and he saw this as an attractive opportunity for me. The team was very small and understaffed and very talented.

And I was an ambitious, obnoxious, 20-something determined to make my mark which was encouraged to go blow up the current piece of software architecture and build something new.

There were a lot of lessons I learned during that time. And many of them people lessons. And I’ll get to them in time.

But there was one that is particularly relevant to my day today, so I’ll repeat it here.

At the time, the storage management product was called “Data Fabric Manager.” The basic architecture and I am going from memory, was a monitor service that polled the infrastructure, an embedded database, eventing and alarm service that sent out SNMP traps, or emails based on what the monitor service uncovered.

The DFM CLI was a program that executed as a CGI-bin script inside of an Apache web server. The CLI implemented a web-UI and a CLI command set and had an XML input interface.

The problem with the technology was that in 2004, the kind of UI you could build in a web browser was quite limited.

At the time I believed that to build a slick performance monitoring tool, you wanted a thick-client and that a web-UI wasn’t going to cut it.

The team agreed to build a new thick client that would have a new API service, called Acropolis that the thick client would use.

Later on, we built Protection Manager on top of this new architecture, and Protection Manager required a lot of APIs.

And then there was a debate over whether the APIs would be public or not.

As we were building the UI, one of our most talented engineers observed he could be 5x more efficient if there were a private API that the UI engineers could use. His point is that the UI needed a lot of API’s and not all of these APIs were going to be useful to anyone but the UI.

And it was an interesting point. Here I was advocating for a public API at a cost to development at a time when no one was integrating into management systems.

And after some thought, I made the call that all APIs should be public.

And the rationale was the following

1. We had no idea what someone would use the APIs for
2. We had no idea when the APIs would be used
3. It’s practically impossible to justify investing in APIs except when your products need them.

And it became a mantra of mine when you create an API make it public. Because making the API later is very hard to prioritize and get resources for.

Yes, your API is probably not the world’s best API. But then again neither was MS-DOS or the x86 instruction set.

So what happened next…

Later on, when we needed to do integrations, integrations we didn’t anticipate that decision paid some dividends.

The existence of the APIs made it possible to have partnership discussions that centered around extending or improving the API instead of “whether APIs exist”. And because all of the functionality was exposed, the partner could play with the totality of the functionality even if we didn’t have everything they wanted. And more importantly, this allowed us to discuss how we could evolve the whole system to do the right thing.

 

 

 

 

My career has been a lot of fun. And as a career, I hope the best is yet to come.

There are three major pieces of technology that have shaped my view on software architecture.

The first was the delivery of a streaming media cache at NetApp. The streaming media cache was a feature of the NetCache product line. In this project, I was an individual contributor working with some of the best software architects I have had the good fortune of working with. What we built was a system that allowed us to sit between a windows media server or a quick time server or a Real Networks Server and delivery media at a lower cost per $ than having a bunch of servers.

The second was the set of products that I delivered as part of the storage management team at NetApp.  The first was a performance monitoring tool that created the first client-server product NetApp had called Performance Advisor. The second was a radical data protection tool that used policy-driven data management that was too far ahead of its time. And the third was the first solution to integrate storage and virtual machine management.

The third was my time at Zynga. What I did at Zynga, was to create a centralized operations team that delivered a shared back-end with a large of collection of services that reduced the OPEX and CAPEX of managing Web-scale infrastructures while simultaneously improving uptime. Some of the stuff we did was open sourced, like zperfmon and zbase. Our infrastructure and team were so amazing that after a mistake and a bug in a partner product, we were able to restore a 12 million DAU game with several thousand servers in less than 12 hours. The capstone of my time there was the delivery of an API infrastructure that took all of Zynga’s different backend services and put them behind a REST API that made it significantly easier to deliver features and services and enable 3rd party feature development.

Since then, I have had the good fortune to work at VMware, and a lot of what I am doing there I can’t talk about so will not. Although one thing that was accomplished as part of the 6.5 release, was an architectural review board that I chaired that reviewed many vCenter features.

The theme of my career, and what I view as strategic software architecture, is that I didn’t architect a single product. Most of the architects were people who I advised. What I did was to create the context that allowed them to deliver miracles. There were times, of course, where I had to step in, and advice requires technical understanding, but it wasn’t about me writing all or any of the code.

And this is where the confusion happens. I remember sitting in a room with a manager at NetApp. And she was screaming at me, telling me I was good for nothing worthless software architect. That a software architect created a working prototype like her husband. She was pissed that she was forced to work with me.

Given it was early in my career, I freaked out.

And I get her reaction a lot. Every time someone looks at me, and what I do they think – he doesn’t write code, so he can’t be any good. He just talks and talks and talks. And he doesn’t understand what’s really going on.

And then three years later they are like that woman who screamed at me. After we shipped the first version of the product, and more and more of the set of products started to adopt that unifying vision, she realized that I actually added a lot of value. That the whole org of over 200 people was moving in the same direction even though they were working on a variety of products. That what I did was create a context that enabled very large teams to work together well.

All that talking and probing and pushing and getting people aligned and providing the technical depth that allowed people to get unstuck produced a staggering amount of software, more than anyone human being could write in a year.

Because what I do is to figure out what are the strategic lever points, and use those lever points to move the world.

 

 

About a month ago, my wife and I attended a performance of Hamilton in San Francisco.

And afterward, as I was sitting in the car, I realized that there was a story I wanted to tell in defense of software architecture.

More precisely a story that was a  defense of the kind of software architecture I do. Because it’s a kind of software architecture that is extraordinarily valuable, and extremely undervalued: strategic software architecture. What strategic software architecture is and is not will be something this set of essays will attempt to define, describe, characterize and explain. In a nutshell, the central thesis is that for any business the strategic 5 -7-year question of how to marshall people and technology and product to deliver outsized business results is actually a software architecture problem that tries to impose structure and chaos on a fluid situation while providing flexibility in the choice of tactics.

A mouthful indeed.

And in honor of Alexander Hamilton and the Founding Fathers, I decided to write a series of Essays titled the architecturalist papers, a pompous homage to the Federalist papers, that tried to explain and defend strategic software architecture.

The struggle I faced in putting the finger to keyboard, was the daunting task of doing research. After all, shouldn’t I do some survey of the state of the, and show where my thoughts fit into the general understanding of software?

Thankfully, a friend of mine remarked that many researchers in her field are not empiricists. And first I had to look up the word empiricist and discovered that it meant

a person who supports the theory that all knowledge is based on experience derived from the senses.

And it became apparent, that I had a lot of experience in this space, and there was a lot of knowledge to be derived, and some more abstract thinkers could figure out general models that were more valuable.

What clinched the deal, was another exchange with my wife about common sense. Common sense, I had read somewhere was defined as the set of accumulated wisdom from experience. When we say someone lacks common sense, what we mean is they lack the accumulated experience or don’t have access to that experience and make poor decisions.

This set of essays is an attempt to share my collected set of experiences and will allow others and myself to derive knowledge from the entire experience and hopefully share some common sense ideas that have proven to be very useful over the years.

The next essay will be a survey of what things I have been involved in that forms the basis of my experience.