You are finite. Zathras is finite. This is wrong tool.
I intend to write a lot about why you should go and read this https://hackingcapitalism.io/
And then realized I could write very little.
It would be best if you read Kris Nóva’s Hacking Capitalism because it is a highly well-written document by an outsider on how to function and succeed in the alien tech world. It would be best if you read it because Kris Nóva (@nova@hachyderm.io) is a fantastic person who is impressive and distills a lifetime of insight into a tight actionable document.
And that’s the only reason you should read it. And if you need me to tell you why beyond that, here’s my life story.
At NetApp circa 2002, a great manager named Jonathan Crowther took me to lunch to discuss my career. It was the first time any manager had done that.
I was frustrated because my career had stalled out. And I didn’t understand why.
And he smiled and said that the problem was that I only approached people as functional units that needed to solve my problems or I needed to solve theirs. That I never treated people as people.
And I remember staring at him, and then he explained, “when you come in on Monday, you never say – how was your weekend? It’s always – I need this.”
I stared at him. I suspect I am somewhere on the spectrum. And it was a moment of clear revelation.
I spent the entire weekend thinking about this one conversation. A key benefit I had was that I had done a lot of acting, enjoyed D&D, and had an absurdly high executive function. And so I came on Monday morning and acted. I have a youtube video where I talk about this here.
What resonates with Ms. Nóva’s document is that she detaches herself from the reality of understanding why and accepts that things are and that she must devise rules on how to win based on that.
I lack her precision of English, wit, insight, and, to be quite frank, experience. And I wish, when I was 22, she had given me this document.
So who should read it? If you were not born in the United States, you were not a white CIS-gendered man, and you are not privileged, you should read this document.
The United States is an alien country. Its rules are strange. And its culture is derived from 300+ years of interaction between property rights, the evil belief that labor should be enslaved, that extreme Christian Dogmas are fundamental, and a sense of supreme righteousness and contempt.
For example, let’s consider ownership.
A year ago, I had an opportunity to talk to a Chinese woman, from mainland China. And she was talking about ownership and how she was struggling with her career because of that. And I realized she was talking about ownership in a way that sounded like a colorblind person talking about colors.
And then we talked about “what is ownership in the USA.” In the USA, ownership of property is a sacred right. When you own something, you own it. You can do whatever you want to it. And nobody can tell you otherwise. A considerable amount of the conflict in this country is putting boundaries on the ownership of people by others and the limits of ownership of common goods by individuals at the expense of the group. The idea that the country protected property rights from others and the government was eye-opening to her. Her life experience in Xi’s China and property rights was not that. And as we talked about ownership, she said – “oh, so that’s what ownership means.”
I love this country, and as a Greek from a village, a Canadian from Montreal surrounded by more small-town Greeks who fled Greece due to a civil war, a world war, and a junta, the culture of the United States and beliefs frequently leave me perplexed.
Hacking Capitalism is a guide to making this system work for you without understanding the why and just understanding its weak points.
My father is Prof. Charis Roussos. He’s currently 78 years old. And he’s a legendary figure.
To make Prof. Roussos’s impact tangible, let’s compare how set designers imagined the future of hospital medicine.
In 1966, the set designers of the sci-fi show Star Trek imagined the medlab of ~2254 looked like this.
The hospital was a single doctor with a digital chart.
But in 2009, the set designers for the film Star Trek imagined the medlab in 2233 looking like this.
Notice the ventilator, the machines surrounding the hospital bed? Prof. Roussos made them a necessary part of medicine.
But why did this ventilator become so ubiquitous in just 43 years?
Before his research, there was this thing called “shock.” Patients would enter “shock” and die. And the why was a mystery.
But its existence meant that a whole class of things we take for granted are impossible. Surgeries that could save your life but might lead to “shock” were very risky.
But was shock?
He and his teams proved that shock happened when blood pressure dropped.
Let me explain.
Typically you have a specific volume of oxygenated blood pumping through your body. We all know that the heart is a muscle, and we all know about the lungs, but what we didn’t realize was that there were some equally if not more essential muscles, “the lung muscles or the thoracic muscles.”
The body needs oxygen; otherwise, it dies. And without the lung muscles moving, you don’t breathe and die.
But for the lung muscles to move, they need oxygen as well.
Under normal conditions, this is not a problem. There is enough oxygen coming in and enough blood going around for everyone.
But when a patient is under distress, and there is less blood to go around, the brain needs to make some tradeoffs. Well, it can’t stop breathing, so it keeps the lungs working and starves everything else, including itself.
But how much blood do these lungs need? Well, about 25% of the total blood flow.
So what was shock – less blood being pumped through the heart meant less oxygenated blood for everyone, which meant that vital organs started to fail, which led to death. To deal with that, the brain then tells the lung muscles, “WORK HARDER,” and sends even more blood, and the lungs work harder, and vital organs get further starved. . Until they are exhausted, and the brain realize it can do nothing, gives up, and the patient dies.
Shocking.
These lung muscles were the key to keeping people alive!
But how to do that? Well, if the brain only cared about oxygen, what if we fed the patient oxygen or artificially moved the lungs?
And that’s where ventilators come in.
What if we used ventilators to move the lungs artificially, allowing more blood to flow to other parts of the body?
That simple idea saved people who would otherwise have died and enabled surgeries and procedures that in the past were too risky.
You can hear him talk about this here.
In a recent sequence of discussions on availability at work, I realized that there are different flavors.
But first, a history lesson.
Before virtualization became a thing, the physical data center physically existed. And its destruction, although possible, was extremely rare.
The destruction of an entire physical data center was extremely rare, and we referred to it as a “meteor strike or nuclear war.”
For practical purposes, availability was about dealing with physical component failures. For example, if a server had a disk, and the disk died, could the application on the server keep running?
As hardware components became increasingly resilient and the cost of hardware plummeted, the weak link became the software.
Because rearchitecting software to become intrinsically highly available was impossible, new software frameworks came into existence.
One of the most impressive was vSphere-HA because it said the following
vSphere HA and its more boutique solutions more or less solved the availability problem.
And the power of vSphere-HA to systematically solve the HA problem across an entire application fleet greatly expanded the value of VMware’s technology value proposition.
Except, a new availability problem emerged.
In the past, a single software could not destroy a single data center or all data centers. The scope of the blast radius of an operator error was quite limited.
But with virtualized infrastructures and cloud infrastructure, a single button click can destroy an environment in minutes.
For example, at Zynga, an operator error caused by a lousy user-experience design deleted the production Cityville deployment. Cityville had over 10 million Daily Active Users (DAU) and 30 million Monthly Active Users (MAU).
Why was this possible?
Because in the past, you couldn’t delete a server, the applications in the server, the network connections associated with that server, and the data contained in those servers with a single button click in minutes because it would require logging into hundreds of systems. Why? Because the nature of the systems relied on multiple administrators of infrastructure coordinating to either provision or delete infrastructure.
Let’s use an analogy from social media. Before Facebook, I couldn’t share the news with millions of people using a few button clicks to use an analogy. I had to talk to a lot of people. And so, the blast radius of my news was quite limited. Facebook made it easier to share good information like I have a new child very quickly, and sad news like a friend died very quickly. It also made it possible to share fake news and lies very quickly as well. The same technology could be used to do both. And the societal damage of that rapid dissemination of phony information is confirmed.
Virtualization and later cloud moved the entire data center that the application depended on into a single database. The single database allowed agility to grow, shrink, and adjust infrastructure demand to meet the changing application demand. It also made this database the single most significant vulnerability to your environment.
And so software engineers rightfully focused on how to make that database available. But the problem is that no amount of software available can solve operator error, and worse, mistakes that destroy the database in its entirety.
As more and more effort was put into making the central system more robust and available, the overall design became more and more fragile.
But what kinds of errors? Remember, the physical data center rarely goes away in its entirety, but several servers? Those fail all of the time. Worse, as the servers become less available to become cheaper and more disposable because technologies like vSphere HA or cloud-native 12-factor application styles expand, their failure rate increases.
And this cheaper hardware that is more disposable is possible because of the very same centralized databases that when they fail destroy everything. Antifragility makes it clear that the more and more you make everything depend on a single system, the more and more the entire system becomes fragile.
So we have this peculiar phenomenon that the entire virtual infrastructure of the data center resides in a single piece of software that if it fails, everything is gone. And the thing that can cause it to fail is not under the control of software: things like Human error, or data corruptions in parts of the stack that are unknowable, or ransomware, etc.
Without any personal internal knowledge, the Facebook DNS debacle is a great example. The software allowed the network engineering team to make widespread infrastructure changes across Facebook in minutes. And human error combined with technology errors (just another word for human errors) resulted in the entire Facebook infrastructure is unavailable.
IT operators understand this point, as did DevOps teams and the SRE teams. And this is why they refrain from having a single system control their entire infrastructure. The scope of a single error is why those groups like to have more k8s clusters instead of one large one. Or why backup admins have backup software from a different vendor from their primary storage.
Centralizing things allows for unprecedented agility and failure.
What to do?
My take is threefold.
The first is that the era of centralized databases that control the entire infrastructure is closing. Instead, we will have many databases. In the k8s space, the idea of a few large k8s clusters is an anti-pattern. And this is why I am such a huge fan of products like Tanzu Mission Control. They line up with the future.
The second is that those centralized databases will have to be built leveraging blockchain systems that protect against byzantine errors and, in particular, human errors.
The third is that the ability to recover from those database errors in a timely fashion will become a critical differentiator. A once-in-a-decade event is terrible if it happens on a Friday and a company ending event on Black Friday. Software stacks that make that recovery quick and easy will win.
One of the best program managers I have had the chance to work with observed that companies have challenges when transactional-based decision making and arrangement-based decisions come into conflict.
Transactional-based decision-making is what I call strict enumeration and documentation in the form of a contract. Once both sides agree on the document, then even if people change, the decision remains. The trust is in the record, and the change controls surrounding the document. This kind of decision depersonalizes the decision. These decisions tend to be transparent.
Arrangement-based decision-making relies on trust between the two parties. The parties establish trust in various ways, but typically it’s about making sure that the personal goals between both leaders align. Arrangements are remarkably durable and transportable. Meaning that once you establish an arrangement with someone, trust is the basis of subsequent decisions much faster. It also means that decisions without going through a complex process. The problem is that these decisions tend to be opaque.
Both models taken to extremes can be a disaster. I have worked in both. And they both suck.
The law is THE example of a transactional system. Overly transactional environments turn into bureaucratic, legalistic environments.
A pure transactional system can turn into a totalitarian state. A great example is – “the only way to change this spec is to call a meeting of the change control board and submit a request.”
A pure arrangement system can turn into a cult where belief in the great leader is paramount and the phrase “blah said” is used to justify or denounce everything. That disagreement with the great leader is an unforgivable sin. A great example of this is when someone says – “but SR ARCHITECT FOO said”. Or alternatively – an old-boys network – that is impenetrable.
I’ll also observe that both approaches feel natural to different people. My personal experience is being Greek and Canadian and living in the USA. As a Greek, I believe that laws are suggestions. That relationships, and in particular, family relationships, trump everything. As someone who lives in Sunnyvale, CA, I have learned that in the USA, laws matter, but that the laws are structured to satisfy arrangements among the wealthy.
A transactional system feels like a spectacular waste of time because personal relationships trump everything and that, ultimately, transactional systems are a facade for arrangements.
But I have learned that that is naive. The critical flaw of arrangements is the opacity of building trust and the boundaries of trust. Transactional-based decision-making creates a public record of the moment trust was built and describes the trust boundary. Without such a record, trust-building naturally devolves to family, culture, background, and other attributes.
This brings me to how I think about architecture reviews. The purpose of the review is to create a trust boundary between the approver and the author. The more high-level the spec, the more trust that has to exist. That trust is typically built over a series of smaller successes or previous professional successes. The more detailed the spec, the less trust that exists. A key trust moment as an architect is when you are willing to stop being the sole approver of an area.
The key illusion in all of this is that as an architect the spec actually defines what is built. The reality is that unless you are writing the software, some other human being is going to take that document and do what they think the right thing is. My job is to make sure that they are thinking about the problem in a way that aligns with a reasonable solution.
Decrying arrangements because they are old boys networks is wrong. Decrying transactional-based systems because Process is also wrong. Like most everything in life, there is a balance. And like most everything in life, navigating that balance is the art of living and being a software architect.
Over the last 15 years, I’ve been noodling about portability. And the more I noodle about portability, the more I think it’s very poorly defined.
So, let me try.
In the context of computers, there are two flavors of portability, data path portability, and control plane portability. The data path is the bits of code that do real work. The control path is the code that sets up the data path, monitors the data path, and takes corrective actions.
In that weird way that computers work, the control path itself is just another data path. And so – in a bizarre way – there is only data path portability. Simplistically, the control path is instructions that execute on processors, store data in memory and storage.
And although Paris Kanelakis would be pleased to know I learned something in CS051, things are not- quite that simple in the world of commerce that I live in.
Except when they are.
The VM encapsulates a traditional single VM applications control and data path. And the VM, as a result, is a highly portable abstraction because the hypervisor can ignore which instructions are from the control path and which data path; to it, they are all data path instructions. And we know that if you don’t care about performance, any data path can simulate any other data path.
So far, so good.
VMware demonstrated that the portability of a VM is very valuable because you can move it between generations of hardware, move it around to utilize the efficiency of your infrastructure better, and because it’s a convenient way to manage workloads.
And so, the VM abstraction and its portability had another interesting effect. You can hand over the infrastructure operation to another person.
But while I was at NetApp and Zynga, there were all sorts of other portability that people discussed. This data path portability was nice but not sufficient, and it wasn’t enough if you couldn’t encapsulate the entire app into a single VM.
So sure, I could move the VM around, but if I didn’t have the control path, what value was the VM?
And so there was this thought process that said, “VM’s are not that interesting.” 
are
And I looked into all sorts of interesting programming languages and tool chests.
And yet, along the way, something funny happened. If I cared about performance, I cared a lot about the hardware I was running on. And all of a sudden, the behavior of the hypervisor matter a lot.
At Zynga, when Amazon changed their NIC, our costs skyrocketed. Their decision to optimize their business almost tanked ours.
But performance, who cares about performance? Right? Except, here’s the rub, when you don’t have pricing power for a feature, and you sell the hardware that the feature runs on. The only way it can be worth implementing is to implement the feature without meaningfully increasing the hardware costs.
And then the performance of the feature matters a lot. And some features don’t get implemented because there isn’t an efficient way to do that.
But if you can’t charge more money, why implement the feature? Because – here’s the rub – your competition is adding features very fast. And if you don’t stay ahead, the whole SaaS model suggests that eventually, your customers will move. So as a vendor, you have to keep adding features while keeping the cost of the features in check.
Hoom. So – wait, if I want to grow my business and add more services, and can’t meaningfully increase the price, then, erm, the performance of the feature matters a lot.
And yes, you can make code more efficient and thoughtful, but at some point, you start caring about whether the disk is an SSD or whether the NIC has enough buffers, or what exactly is the CPU you are running on.
I mean, if people didn’t care about these things, those things wouldn’t exist.
Okay, so?
Well, once you start getting to the point that you care about performance that much, then the hypervisor details matter a lot. But surely you jest? Hypervisors make choices, and those choices impact the way your applications run. And if you design an application to run on a particular hypervisor, then the portability of your application is the portability of that hypervisor. If the hypervisor only runs on hardware X or Y, or Z, any performance improvement locks you into that vendor.
To put it differently, if the control path of the hardware is in the hypervisor, and you need to control the hardware, you need to control the hypervisor.
So what? It turns out that the hardware vendors are adding more hardware faster than that integrated solution vendor is sharing. And so, being able to have some degree of portability to run on other hardware will matter a lot.
Because why? Because the more it costs to run a feature, the less money you have and the more money the other person has.
Hoom.
Baremetal! BAREMETAL!
I hear you say. I was cheeky. See, every modern OS is a hypervisor of some kind. When you run on a public cloud, you can’t control the hypervisor.
And in some ways, not to be annoying, running your app in a VM on a cloud is like trying to tune your Java code – optimizations can be done, or you can use C or Rust or C++.
So when I hear bare metal, I don’t hear “no hypervisor”; I hear – – run a hypervisor of your choice on hardware.
BUT BAREMETAL IS HARD. I agree.
So?
Well, here’s where things get very interesting. Suppose you had a portable hypervisor that existed, and you didn’t have to do the hard part of managing the hardware but could control said hypervisor to your heart’s content?
Why then, for those applications where the data path was critical, you wouldn’t be tied to the cloud vendor’s hardware choices.
Data path portability is about being able to run your VM anywhere you want and be able to control the hypervisor.
AHA!
So for the essential thing that can move my costs, I need to move my VM to whatever hardware I want and tweak the hypervisor any way I want.
But then why has the public cloud won so much business?
Well, for the same reason, Java has won over C++.
It’s just got a better control plane for application development. Java makes so many things easier.
And really, that portable hypervisor was painful to use compared to the cloud.
So hypervisor portability was nice, but it was uninteresting if you couldn’t run the control plane. And you couldn’t because it took a lot of time and money to build, and the value of doing that work was marginal. Except when it wasn’t like for Dropbox. Or at Zynga, where we designed our control plane to be portable and had options when Amazon decided to optimize their business at our expense.
But the control planes aren’t tied to the hardware platforms as much.
What does that mean?
Well, it means that for some workloads where the costs matter, using the non-proprietary control planes so I can use the portable hypervisor may be a better choice.
And well, a portable control plane is – to a hypervisor – just another data path.
REPATRIATION!
No.
No software service that exists and increases in value hasn’t been optimized and improved over time. The original S3 relied on MySQL databases, for crying out loud, and Facebook used a single NetApp filer.
Thinking about this as “repatriation” is the wrong mental model. The suitable model is that critical services will get optimized, and at some point, the optimizations will care about the hardware. The existence of portable control planes and portable data planes will be fascinating.
But I have to build my data center!
Nope.
See, those portable hypervisors are pretty much available as a service now.
And that leads me to the final round of thinking.
When application control and data plane portability is possible, will the ability to cram a small number of hardware configurations in data centers be that valuable?
Watching how many of my coworkers don’t want to go full-time back to work got me thinking about the physical work environment.
Pre-Covid my home office was a laptop with poor ergonomic characteristics on my kitchen table and my car.
My work office, an 8×12 area, shared with a co-worker, had a 32″ monitor, a desk that could move up and down, and a perfect chair and was just a fundamentally more pleasant place to work.
Post-Covid, my home office is a 40×40 room, with a 32″ and 27″ monitor, a desk that can go up and down, a perfect chair, a nice couch, and is quiet.
The effect of Covid was I invested a lot of time and effort to create a great working area at home that no company in Silicon Valley can afford to replicate.
Being forced to come to work in my old office would be going from a really nice office to a worse office – my car and the workspace in my HQ.
If I wasn’t an extrovert, and loud thus irritating my family, I don’t know if I would want to go back to the office.
I am very fortunate in my workspace at work. Some of my coworkers have much smaller and louder workspaces. And those smaller and louder workspaces were created to pack more people into the office because that was just how we did things.
And this got me thinking further.
Pre-COVID, workspaces were a perk that companies could hand out as a way to reward employees. Now employees, having built much better workspaces at home, are rebelling at the idea that they must trade their home workspaces for the qualitatively worse office space. 
Over the years in my career, I have seen the tension between – “MOAR FEATURES” vs. “MOAR EFFICIENCY’.
The more features camp tends to view everything through the prism of – add more features to find more customers. Therefore, every business problem is a feature problem. And any activity that doesn’t result in more features is a waste of time.
The more efficient camp tends to view everything through the prism of – if the software and the construction of the software were more efficient, we could save money and be more productive.
The pro-features folks tend to argue that growth is the solution to every efficiency problem. If you have more money, you can spend your way out of any problem. And so the goal is always to be making more money. The pro-features camp also argues that there are many ways to be more efficient, including hiring cheaper engineers.
This Manichean view of the universe is how we arrive at Product Manager type CEOs arguing that we shouldn’t invest in anything that detracts from feature velocity.
Strategic software architecture must take a different point of view.
Until a product has achieved product-market fit and viability past 18 months isn’t assured, any investment that pays out after 18 months is of questionable value.
Once a product-market fit has been achieved and viability past 18 months is assured, efficiency investments effectively extend the business’s runway, return the margin to the business, and allow the company to generate more cash per dollar spent.
And the point at which you pivot investment is a delicate one.
The pro-feature camp controls the agenda. And they will push hard against any notion that efficiency investment should trump features.
The pro-efficiency camp being overruled for so long will either not exist or have no ability to drive investment decisions.
Worse, the pro-feature camp, by their nature, tends to view any outcome that is further out than 18 months to be very speculative. And the pro-efficiency camp views the value of their work in a 36 month time horizon.
In this battle, the short-term always wins.
In this struggle and debate, my attitude has always been that efficiency must have its own agenda but must sell it to the pro-feature camp. In effect, every efficiency improvement is really a feature.
But to do that, you have to sacrifice the big bang for small incremental wins along the way. In effect, when the business wants a feature, you write code that also improves the efficiency of the system over time.
And, of course, it’s not that simple.
You also have to pick the right technologies. To deliver the biggest wins, you need to be able to use best-of-breed technologies where there is a competitive market, and you need to be able to swap out technologies easily.
Huh?
Let’s take multi-cloud, for example. One argument is that making your application portable is foolish because it detracts from the core business, which is feature delivery. The argument is typically presented in the form – “private cloud very hard, public cloud easy, single cloud easier.”
And there is some truth to that. And the ease comes at a cost. And debates ensue.
I see it very differently. There is this massive competitive market of infrastructure providers. That infrastructure is a commodity except in the rarest of cases. As a technologist, I enable the business to be more efficient by allowing it to take advantage of that commodity infrastructure through my software choices. Customers of infrastructure will choose because of externalities—things like laws, vendor relationships, and personal taste.
Pre-k8s, this argument was slightly harder to make because there was no obvious API layer that would allow you to decouple your application layer from the common infrastructure, and the market wasn’t very competitive.
In fact, in 2015, at VMware, I argued that such a layer would have to come into existence, or AWS would own the world. So our job at VMware was to stay alive until such a layer emerged. I remember saying to folks – this thing would emerge miraculously, and when it did, we would need to jump all over it. I remember folks looking at me as if I was delusional.
Well, k8s arrived and was exactly that layer, and we jumped all over it. K8s was breathtakingly brilliant because it was the right choice for developers and infrastructure providers.
And the reality is that just picking k8s already makes your SaaS system more portable. And it’s an easy choice and gives you portability from day 0.
But it’s more than just k8s. Picking a managed Postgres over a custom DB makes your application more portable.
Incremental choices that over time add up so that when you want to make bigger bets, the cost of making those bets is small.
And the real bet is that over time because the market is now competitive, the infrastructure vendors will create an ecosystem that makes multi-cloud easier and private cloud easier.
In fact, I believe that a single cloud strategy advocates ignore how k8s is creating an API layer that enables innovation and competition in all layers simultaneously. And it’s not the only layer. Amazon, for example, with i3N metal instances, created a competitive market for hypervisors in AWS.
The anti-multi-cloud folks point to dropbox as a potential failure because of the complexity. On the contrary, I would argue dropbox is an example of how hard multi-cloud is without the right layers creating an opportunity for competition everywhere.
The anti-multi-cloud folks’ argument also turns to – well, java. The argument is that Java attempted to create a sealed abstraction that hid all infrastructure from the applications and failed.
And they have a point. But the goal isn’t the ability to run the application without change on any cloud; the goal is to make the cost of doing that so small that the efficiency gains can be realized. The goal is to make it easy to force the infrastructure vendors to give up some of their margins to the application.
In this picture, you can see what I mean. In the initial state, the application is using only proprietary APIs. Over time, the applications that are being developed have access to standard APIs. The existence of those standard APIs and a competitive market gives choices. The proprietary APIs never go away; they just become a smaller part of the application. And the most important part is that the goal is not to use custom APIs; the goal is not to let custom APIs hold you hostage, to a vendor. And the other point is that it’s multi-cloud; if you want to use custom APIs, use them.
The emergence of that competitive market is creating innovation right now.
And as a result, forward-thinking venture capitalists and forward-thinking architects will choose technologies that will facilitate the emergence of multi-cloud because it is in their selfish interest.
The recent backtracking of Google, Amazon, and Apple on remote work is fascinating. For companies that are supposed to be on the bleeding edge of corporate governance, the decisions that these folks made not to allow remote work was one that makes me want to ask – “show me your work that leads to your decisions.”
In that spirit, let me show mine.
In 1994, Andries Van Dam got in front of an undergraduate computer science class and announced that the idea of all software engineering remaining in the USA was foolish. As software became more about composing components than building everything from scratch, and because any software engineer anywhere can construct a component that any other engineer could use, jobs would be offshored. In effect, he argued that there would be fewer jobs in the USA.
It’s easy to laugh at Andy now. But in 1994, the bulk of software was for military-industrial uses. The worldwide web had not yet come into existence. And thanks to the Peace-Dividend, jobs were being lost everywhere in tech.
Many years later, he and I caught up, and we remarked that he was right; what he didn’t account for was an ever-increasing pie. In other words, both statements could be true; global teams delivering components would happen, and more jobs writing software would exist.
Post-dot-com boom, the fetish in the tech industry was this concept call off-shoring. The thesis was that companies could find good enough talent at lower cost-locales and thus reduce their costs.
I remember being in a meeting where some GMs declared that we would only be hiring in India, convinced me that Andy’s apocalyptic vision was right, and plan for a backup plan for my software career.
Waiting on a Green Card and having no other realistic choice as the nuclear winter of 2001-2009 took hold in the tech space, I had to learn how to work with remote teams.
In 2002, I made my first recruiting trip to India, where we brought up a team whose job was to take on NetCache. It was really a case of training my replacements. Being young and foolish, I don’t think I really understood what was going on, but it was pretty grim.
In 2003, I had another GM show up in a meeting and explained that we needed to offshore more work to India, and my project on storage management was to be sent to India.
But then something funny happened.
We started hiring in the USA again.
And then we acquired other companies in other parts of the USA.
And then, I suddenly found myself leading teams in Bangalore, Palo Alto, North Carolina, and Pittsburgh.
At one point, I was on a project where everyone was remote, and I wondered why the heck I was coming into work?
From about 2004-2009, every meeting had someone dialing in remotely. Every project had a remote engineering team.
Then I moved to Zynga, and I swore I would never work with remote teams again, and yep, I was working with teams all over the world.
Then I arrived at VMware in 2015, where remote work had not been widely adopted across the company.
But then, from 2015 to 2019, my partner in crime and I decided, that talent not location, mattered. So we hired folks anywhere. Our architect for our biggest technical achievement was on an island in Puget Sound, and the other one is in Humboldt county.
The location didn’t matter. We would hire anyone anywhere.
After the Heptio acquisition, I remember talking to some VP, and he said – “well, it’s going to be hard to meet because the Heptio folks aren’t in PA.” And I was like, “oh, if they were, that would suck because the guy they need to talk to is in Humboldt county and a beach house in Washington State.”
If I looked at my team, I had tech leads in Bangalore, Austin, Washington State, Sofia (Bulgaria), and Humboldt county.
We succeeded as a team because we didn’t care where you lived. And we figured out how to work together even if we didn’t have physical proximity.
And so over 17 years, I have been working with remote teams, and those teams have delivered staggering business value across three different companies and three different industries, including a hardware company.
The idea that remote work is worse than local work is a failure of the imagination of business leaders and management leaders.
But why?
Because here’s what I learned, if X units of work need to get done, and individual Y wants to do them in location Z, then X units of work get done if you let them do it from location Z. If you don’t, the X units of work don’t get done until you find individual i in the desired location R.
So the loss of productivity is measured in the following way:
For everyday d that some engineer E is not working, X units per work per day (x/d) do not get done, or P(E) = x/d
If D(i) is the number of days that it takes to find the imaginary i, then the lost productivity is P(i) = x/d*D(i)
If time to market matters, and days are precious, then the value of the lost days is huge.
So when I hear leaders tell me that agility matters, speed matters, and they are unwilling to let people work from where I must admit, I have a hard time believing their math. And like my wife, who is a math teacher, says, I want to say – “show me your work.”
But then I hear startups or small teams say, but small teams.
And that got me thinking.
In 2003, when my team was only 8 engineers, we hired a team in Bangalore.
In 2010, when Zynga was growing, I couldn’t get recruiting to give me the time of day. I had 5 engineers in San Francisco, and the Chief People Officer, Colleen McCreary, asked if I was willing to hire in Seattle. And so we grew that 5 person team very quickly, and they built the infrastructure that Zynga used over the next decade.
In 2015, when hiring was very hard at VMware, I had nobody in Seattle, and when a very senior engineer showed up, and others were wondering how to make him productive, I said – yes.
Every single time, finding talent was the hard part. Dealing with remote locations was easy. And then, because I was willing to go to remote locations, finding more talent became easier.
Talent matters. Location doesn’t.
And that leads me to the following conclusion – if you can’t work with remote teams, then you might want to look very carefully at how your team works.
In 2013, after being pushed out of my role at Zynga, I was noodling on what to do next. And was thinking about working on hybrid cloud technologies, but I figured that with everything Zynga had achieved, this was an area that was well-trodden and not one where a new startup was welcome.
Wrong.
So let me rewind the clock.
Here’s what I had learned at Zynga.
In many ways, it’s the basic tension between using a prototyping environment or language like python versus a more statically typed language like C++.
And that creates a fundamental dilemma, how do you get to the private cloud economics on a public cloud?
Let me start by saying the non-obvious. In 2013, running on the cloud was 1/3 the cost of running AWS with everything priced in.
If Zynga had tried in 2013 when our business was imploding to get to a private cloud, we would have never done it. But in 2009, our prescient CTO Cadir Lee made the strategic bet that we needed to have our own cloud. And by 2012, we had made the transition, allowing us in 2013 to manage our cash very effectively and survive a very long downturn without bleeding cash.
It’s important to realize that in 2011, I was in charge of what we would call DevOps / SRE / AppOps. And I loved AWS. And I was not interested in moving to zCloud. And then we launched CityVille. And the game was crashing, and we had to increase our server count by a factor of three. And when we begged for insight from Amazon, we learned that the problem was that the AWS team had switched NIC’s that had fewer IO buffers, which forced us to get more x86 servers to handle the IO load. That decision ate into our margins decisively, consumed lots of engineering cycles, and convinced me that we had to get off AWS ASAP.
I figured that this knowledge was so obvious that somebody somewhere was working on it as we spoke. And so, I didn’t bother looking for employment in that area.
The other thing I learned at Zynga was about the mobile business and software stack.
I learned at Zynga that vendors like Apple and Google make a lot of money adding software layers on top of Arm instructions to make it hard to write x-platform code.
Games are weird. They are very heavy in terms of graphics and very light in terms of UI. And so, a game can benefit significantly from being written in an x-platform toolkit because most of the time, you are playing on the game board.
And so, I pursued this strategy of trying to enable x-platform development, even to the point where we created our own programming language, playscript.
But it turns out that the x-platform strategy was a losing strategy because if the UI wasn’t native, the experience was marginally worse than some other game. The liquidity of games was so high that it made much sense to use those native features.
And so, the x-platform game strategy wasn’t quite as successful as I would have liked.
But I also learned that whereas with the UX, native integration matters a lot, the value of deep native integration with the back-end systems isn’t.
Customers do not use the back-end systems. Engineers use them. And those engineers – especially the software variety – are surprisingly willing to rebuild or change things.
And a game could be easily transferred from AWS to an on-prem data center if you just organized your system correctly and were prescient.
So how did the system have to be organized?
From my mobile experience, I realized that at the end of the day – every time you used something other than an x86 server, you were paying for someone else’s retirement. And every time you used some service, you were taking a dependency on someone else’s business plan.
Many companies tried to build meta-management layers that spanned the cloud and were a layer above the cloud API. I viewed that as a mistake. The only sane strategy was to have the same x86 server running in the cloud as you had on-prem.
And the only way to achieve that was to use the same kind of hypervisor.
At Zynga, we deliberately chose to run the same kind of Xen Hypervisor that AWS did for that reason. |
When I left Zynga, I was determined to make things like zCloud easy to build. And through a series of convoluted decisions, I ended up at VMware.
Now Kit Colbert talks a lot about multi-cloud in this post, which I urge you to read it.
My personal spin is that ultimately having a common layer that you can reliably deploy anywhere you want is critical. And the companies that last are the ones that can move their workloads and take advantage of the multi-cloud.
In 2013, I had not realized how special the Zynga team was. The guy who built our data centers was Allan Leinwand (currently SVP at Slack). The guy who built our storage system layer was the chief architect at Flipkart, Vinay YS. A key player on the infrastructure side of the house was Nick Tornow (VP of Engineering at Twitter).
The talent we had was staggering.
What VMware is doing is making the ability to run in multiple clouds something that can be done by small teams easily.
It’s a journey and a long one, but one that I am excited about working on.