wrong tool

When I joined VMware, friends asked why? Wasn’t the future public cloud?

And I rejected that hypothesis.

The list below is an abridged version of a lot of deep thinking. These points have served me well over the years as I think about infrastructure.

One: Capex is more efficient spend than op-ex for things that can survive for more than 3 years

It’s taxed more efficiently, frees up cashflow, etc, etc. As power moves from gas/to solar, and spinning media runs on silicon, hardware can last about 1 decade. The systems don’t fail and the cost of running goes to 0.

If you know your capacity, you can literally buy once capitalize for 3 years, and run it for free for 7

Mainframes continue to survive for that reason.

Two: The Turing machine can run any program, and yet we have all kinds of hardware.

Why?

For any given workload, there is optimal hardware that will deliver the desired performance/reliability at an optimal cost.

Cloud doesn’t offer that hardware.

Three: Any prototype of a new system is best done in a typeless scripting language, any understood system is best done in a typed compiled language leveraging hardware

Every python project ever written that required performance or reliability had to be re-written in C/C++

Cloud optimizes for agility, not optimal execution.

Four: Computer systems that are inherently reliable are cheaper to operate than computer systems that are not

The single most important variable in making a system unreliable is how often it changes. A system that never changes, never breaks. The more people that touch a system, the more unreliable it becomes the more costly it is to operate.

Cloud infrastructure is always in-flux, thefore less reliable.

If you read about gaslighting, and you’re a half-decent human being, you may think you never gaslight. But if you are in a position of authority, unless you are careful, you do it all of the time.

As a junior engineer, you can ask a question, but as a senior architect, every issue is loaded. The person on the other side has a very different model for what is going on.

Authority implicitly changes every question.

Let’s take my favorite – “What do you think about my idea, X?”

The person on the receiving end is going to feel gaslit. If the big cheese is asking the question, and you say, “No,” does that end your career? And if you say “Yes,” and it turns out to be a bad idea, does that end your career? And is the big cheese asking your opinion or is he trying to get information about you and your boss?

You are pretending that they have an opinion when they don’t. It’s like asking someone,” Do you think I am fat?”

The other favorite is the” can we do this crazy idea alpha?” The person on the other side has no idea how to respond. If it’s an unfortunate but not damaging idea, the right thing may be to say yes and hope the big boss forgets what you said. If it’s a bad idea and dangerous idea, then you have to argue with the boss. And that sounds fun, but if the boss is committed to their concept, you became a naysayer. And can find yourself trying to keep your role and job.

As a leader, when you propose a solution to the problem, the debate has implicitly ended. And if the idea is a bad one, people are trying to figure out how to understand a bad idea. And thinking about a bad idea feels like your brain is being attacked, that your ability to think is being targetted.

What to do? The right thing to do as a boss is to frame the problem and ask for solutions, not propose them. Or, if you have a solution in mind, phrase it differently.

Instead of” what do you think about idea x,” a person in authority says,” I have thought a lot about this idea that I intend to implement, and I am trying to get a few more perspectives. I would love it if I could run it by you to see if I missed anything and to get your view.”

No longer is the person in authority faking a level of equality that does not exist. Instead, they are telling the truth. And the truth is they don’t care what that other person thinks, but they are interested in knowing if they forgot or missed anything.

But what if the person in authority is frustrated that they can’t get honest feedback? What if they feel that their subordinates are unnecessarily frightened? What if they do want to be challenged and are not?

It must be the spinelessness of their subordinates.

Nope.

It’s not the other’s fault; it’s the boss’ fault. For example, have they created an inclusive environment? If someone objects do they get attacked? Etc.

In a corporate environment gaslighting occurs when the boss pretends your opinion matters, but it doesn’t. Gaslighting occurs when the person in charge acts like they want to hear someone’s opinion, but don’t. Gaslighting occurs when the leader says every idea is on the table, but it isn’t. Gaslighting occurs when the decision-maker says that they will consider every idea reasonably and only attack every suggestion that they don’t agree with.

Learning how not to gaslight, and I am first among equals of those who have to improve, is a critical part of being a technology leader.

One of the tough challenges of acting as a strategic software architect is that it’s not precisely an understood job. Many people ask me what I do, and after several fumbling minutes, I point them to this blog.

The most recent analogy that became somewhat useful is that strategic software architecture is about playing chess while the rest of the world plays checkers. And what I mean is that the world is looking at short time horizons, planning the next step, while the role requires planning two or more steps—and simultaneously making moves during the current phase.

Strategic software architecture is not about the stuff that is shipping now. It’s about the thing that will ship in the future. And the job is not to deliver the current thing. The job is to make sure that the team can deliver the current stuff without you.

As that strategic software architect, when you are working on an immediate deliverable, what it means is you failed your team a long, long time ago. If they need your help, it means that you either didn’t provide them the resources, the people, or strategy that would ensure their success. Fail enough times, and there is a new strategic software architect.

In most software companies, the planning cycle doesn’t extend out longer than 18 months. The world changes too fast for anything more than that. And so nobody is thinking past 18 months.

There is one group that is thinking past 18 months, these strategic thinkers. They are a critical, sufficiently unrecognized group across a large number of business functions, but this is about engineering, so I am focusing on that. As this role doesn’t exist and isn’t recognized, and there are no rewards for long term success and, it begs the question of ‘does it exist’?

There are many people like that at a company. They are the ones who seem to generate magic just when the company needs it. That continues to deliver value, and nobody knows why or how. As engineers, they do it with well-timed code-reviews, speaking whispers to the right people, working on the right project, checking in something that nobody expected. They call meetings to discuss things in private, and thereby create a social network that is impenetrable and built around the respect they have earned and the reputation they have acquired.

And over time, the company strategy is the strategic thinkers’ strategy, even if the company thinks otherwise. For many reasons, beginning with hiring that is shaped through their biases. What is easy to build and hard to make is what they and their social network think is easy and hard. What can be created is controlled by their tastes. What is easy and hard to do is shaped through the myriad of small technical decisions that make change very hard. And their software architecture ossifies their decisions through org charts that can endure long past the code choices that formed them are relevant.

Currently, this entire area is left to chance. We are lucky to hire people who can do the job. And I have seen it in my career. Where there are groups that seemingly out of nowhere, keep doing the right thing. And things keep getting better, but I can’t figure out why. And finally, somewhere someone turns up that has a plan . That plan exists in their heads, or on a piece of paper or a confluence page that nobody reads but that everybody is working on.

I believe that a company that figures out how to do strategic software architecture as a discipline and incorporates it into the 18-month planning has a decisive advantage. I also believe that if they can couple this approach with a rabid focus on immediate delivery, they can’t lose.

There are many reasons to change jobs. Some are better than others. But the best is when your peers or your boss don’t engage you for your best work.

Let me start with a story. In 2009, I was a technical director at NetApp. At many other companies, this role is analogous to a senior technical leader with a pay grade that is equivalent to that of a director-level manager.

At the time, NetApp was engaged in a multiyear effort to converge the operating system of the company they had acquired Spinnaker and their platform OnTAP. After the first unsuccessful attempt with the product known as OnTAP GX, the technical and business leadership of the company rallied around a strategy that eventually became what is now known as OnTAP 8.0.

This effort required vast amounts of synchronization. At one point, I was the architect for the data protection portion of the business. The overall architect for the effort sent me an email with a detailed task breakdown of what the data protection team needed to accomplish over the next two years. As I looked at the list I had some concerns with some of the details and the overall general direction, and so I flippantly responded with a “this is so detailed that I’m not sure what value I’m going to add. Why don’t you just send it directly to the management team in the product managers.” And so the author of the email took my response and forwarded the email with his comment, “you’re right.”

After that, it took very little for me to want to leave NetApp.

Over the years I have wondered why this particular exchange was so critical in my leaving NetApp. NetApp was at the time treating me well. I would’ve made more money at NetApp than I did at Zynga. I had a five-week vacation at NetApp. I was well respected by the then CTO and chief science officer. My boss at the time and I had some differences of opinion, but those differences of opinion were resolvable. And the problem space remained interesting.

So why did I leave?

I left because at the end of the day that other architect did not want to engage with my best self. He was not interested in working with me to come up with the right answer. He just wanted me to do exactly as I was told and to take accountability for his decisions.

In short, what I heard him say is, “I don’t need you to think. I need you to do exactly as you’re told and to make sure that the things I need done are done.”

Over the years, I have seen this pattern play out again and again. Sometimes with me on receiving that message or the author of such a message. What I have concluded is that if you’re engaging someone to solve a critical business problem and you talk to them in a way that demonstrates you do not see value in their abilities, then you’re asking them to leave.

But there is another insidious problem that also can occur. If you’re trying to engage with another team where you happen to know a lot about how that team’s systems are built, it is tempting to bypass the current architect and just tell them to do this and this. The problem is that what you’re doing is not engaging the team to be able to solve your problems. And what this kind of communication does is discourage precisely the people you need. The people who want to have a scope to think and to imagine possible solutions. They decide to not want to work on your problems.

If the problem is small in scope, then this is not necessarily a bad thing. If the problem is significant in scope, this is a calamitous decision. You’ve traded off the delivery of a small set of critical capabilities at the expense of a deeper understanding of a hard, complex problem. You have reduced a team that could potentially add value by understanding the problem and thinking about it hard and long for a team that will do exactly as they’re told and no more.

“But that wasn’t the goal,” I have said on more than one occasion.

No, it wasn’t. But that is precisely what I have achieved in the past. Because the people who could think realized that there was room for them to think, only to do. So instead of looking at a problem and trying to solve it, they saw a list of activities that could be done by someone who couldn’t think as deeply as they could. At the end of the day knowledge workers have a tremendous amount of freedom to choose what problems they work on. They also have a tremendous amount of freedom to decide how long and how hard they want to work on an issue. The single most calamitous decision you can make as an architect is to engage with another architect by treating them as less than an equal.

Over the years, I have made this mistake. To those that I treated poorly, my loss is massive but nothing as compared to how poorly I treated you. All I can say is that life is about getting better and learning more new things. And maybe this goes a little distance as an apology.

Over the last years, I have gotten into a series of protracted debates about multi-tenancy.

What I have begun to understand is that it is essential to define the objectives of multi-tenancy before one starts to talk about it.

And even before we get to that need to define what is multitenant.

Consider a piece of hardware, say a server with four sockets. An individual owns the server. Another individual owns the building in which the server resides.

In effect, when there are two actors Mary and Tom, that have access to a system, that system is said to be multitenant if Mary and Tom do not trust each other.

But how much do they trust each other? The trust goes to how much the system must protect Mary from Tom and vice versa. For example, suppose Mary trusts Tom. Then Mary doesn’t care that Tom has physical access to the hardware. And Mary takes no actions to protect her data or her applications running on that server. In effect, Mary and Tom are the same people; they have different roles.

But suppose Mary trusts Tom, but Tom doesn’t want to damage Mary’s system accidentally. Identities and roles play a factor. What Mary would like to do is have a role that Tom can use that allows him to do the things he needs to do to Mary’s server and no more.

And so this is where things get complicated. There are two basic approaches; the first is to bake into the system the set of controls that Tom has access to and to use some role-based access system integrated with some identity system that determines what Tom can do. The problem with such an approach is that if Tom needs to do something that is not in the system, he has no way to do it and has to ask Mary. Now, if Mary is okay with that, all good, however, Mary may not want to do the task and may wish to allow Tom to do the job. But if the system has no way for her to do that, then she is forced to give him access to more controls than he is capable of using.

The second approach is to use layering. You create a net new interface that interacts with Mary system through some APIs, and that net new interface is what Tom uses. Thus when Mary wants to enable Tom to do something new, she will, Tom, can extend his tool to do that. The problem with this approach is that Tom now has access to a whole bunch of operations he shouldn’t have. The only thing preventing Tom from using those operations is his adherence to procedure and the fact that at the end of the day, Tom isn’t malicious. He’s a good guy.

My observation is that approach one doesn’t work. The reason it doesn’t work is the set of operations that Tom needs to perform is ever-evolving. Worse, the collection of activities that Mary wishes Tom to do is ever-expanding. And as a result, they end up using the second approach.

Okay, so what?

The problem is that too many people attempt to build the first model. For example, suppose I have an interface for interacting with the system. That interface allows me to create objects delete objects, or modify objects. Then what happens is that somebody decides that the hierarchy of those objects should reflect some authorization scheme. Then what happens is that Tom and Mary can’t do their jobs because the hierarchy or the complexity of configuring and setting up the hierarchy and setting up authorization is not expressible by the system. In effect, the hierarchy and system that allows you to create edit and manipulate objects for one task is not the same hierarchy you would use for another.

And so, ultimately, what you do is you create a tool that has a specific set of operations that Tom needs. Mary and Tom configure the tool so that it only does what it needs to do.

But, the advocates of the first system point out that the second approach is less secure. And they are right. Or I’ll take them at their word.

They ask, what if Coke and Pepsi want to run their software on the same physical servers. I always found that to be an absurd question. Even if we could assume that the system was entirely secure, there is human error. I thought that Coke and Pepsi would always buy their servers. What is interesting is that the market seems to be doing that even in the public cloud. The Nitro hardware that Amazon has produced mainly provides physical instances on a shared server. And this was before we discovered that there are architectural holes in our systems that allow data to leak between programs running on the same physical server that belong to different tenants.

And so, my assumption has always been that if you care about security, air gaps are about the only thing you should trust.

What does this mean?

Consider the server. With no software, it can do anything you could imagine. The minute you start running software, the set of things you can do becomes increasingly more limited. It turns out that there is a whole slew of user interfaces that are a lot more useful than just starting with the hardware. Over time, a set of interfaces for using a system and controlling access to that system have emerged. And we have figured out over many years how to make them address both Tom and Mary’s needs. A great example is the use of root and less privileged users on most operating systems.

A user interface for the system is handy to both Mary and Tom is incredibly powerful. And therefore, whenever a new way of interacting with servers emerges, there is a temptation to try to figure out what the boundary between the two tenants should be. The reality is that such interfaces developed after years of hard work and experience in operational practicality. Therefore in a new system, you are most likely to draw the boundary in the wrong place. And thus, in my mind, how you access a system should be independent of how you control access.

Okay, I’m dangerously close to talking about security, and when it comes to security, I know that I know nothing.

The problem is that even if you don’t care about security, another critical use case of multi-tenancy is to reduce the cost for the infrastructure provider, Indigo. What Indigo wants to be able to do is assign quotas to Mary and Jane and Tom. And what they want is to ensure that Mary, Jane, and Tom don’t ever exceed their quotas.

Amazon’s solution was to create an infinite supply of servers and bill Mary Jane and Tom for their usage.

The limitation of such a system is that you can only buy the set of servers that Amazon has decided to make available. The other limitation is that it assumes an infinite infrastructure.

If, however, Indigo does not have access to an infinite infrastructure or it’s inappropriate for their use case, what to do?

In my opinion, they should choose approach number two. What does this mean? There are a set of objects that Mary Jane and Tom use to do their jobs. Indigo has a set of quotas that they assigned to Mary, Jane, and Tom. Mary, Tom, and Jane’s need to refer to the quotas and use them transparently. And so there is a temptation to encode them in the objects. But instead of having quota enforcement done at every access to the objects, it should be done lazily unless you exceed some threshold.

And if you look at what Amazon does, they do the same thing.

If you want to use one more server, they will give it to you. If you’re going to employ 10,000 more servers, that involves a phone call. They have their quotas that are lazily enforced and, at some point in time, block access.

In effect, Amazon has decoupled secure isolation from quota enforcement.

And so, when we talk about multi-tenancy, what we need to do is ask, are we trying to solve for secure isolation, or are we trying to solve for quota enforcement? The requirements for security depend on the customer, the trust, the legal requirements, etc. How you do quotas is independent of all of those security restrictions and should be treated as such.

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

Engineering managers have releases they need to deliver to now. They have engineers who have problems now. The managers have budget squabbles now. The teams have debates with product management now.

An architect has visions of the future, and those visions are often years away from delivery, and worse, even more years away from solving any of the immediate problems engineering management has.

And so the VP of engineering begins to see the architect as a distraction. The architect distracts in two ways. The first is that they are always complaining that the team is not investing in the future. The second is that the things they want are unimplementable. And worse, the architect is typically unwilling or unable to go figure anything out. Powerpoint gets produced, confluence pages get edited, wiki pages get updated, and in some cases, small prototypes get written, but no useful code gets changed.

And thus, the VP tries to manage that distraction. There are two fundamental approaches. The first is to keep them away from anything that matters and fire them at some opportune moment. The second is to focus them on a small problem. A small problem keeps the architect from distracting other people and makes it easy to measure their delivery of value. In other words, force the powerpoint jockey to write code, and either they figure out how to write code, or they get fired.

And you know what, the VP of engineering is right.

The thing about being a software architect is that you always have to be right. What I mean is that you are guiding a team, and the direction has to be correct. If it’s not correct, then the team is heading towards a catastrophe. You can change course over time, but it always has to be the right course.

In addition to always being right, you also have to be sufficiently high-level, so everyone is doing the right thing. Pulling this off is another tricky thing to master. If the correct answer is not inclusive of the entire organization, then somebody is doing something wrong. Furthermore, if it’s too low-level, then there are no white-spaces for engineers to innovate.

Another customer for the long term architecture is the CEO/GM and Product Management team. They have to see enough value that they can talk about it to their customers.

In short, you need to be right, and high-level enough that you can’t be wrong, but if that’s where you end, you fail.

The software architect must also add value right now. What does that mean? It means if the head of product management chooses to fund X features, all X are stepping stones to your architectural vision. If the engineers have to design something, it should be evident from the high-level architecture what they should do. If the managers have to figure out how to trade off long term vs. short-term execution, they should make that decision in the context of the global vision.

But it’s more than that. Managers operate in terms of things that need to get done with some set of resources. And so it’s vital that the architecture, be broken down into discrete tasks.

And so my mantra,

• I must always be right.
• I must be sufficiently high-level to be always right.
• I must be right, right now.