wrong tool

Hans Christian Anderson wrote a great story – “The Emperor’s New Clothes.”

The central tenant of the story is that great men can be fooled by yes men. That eventually, the truth wins out, and when it does, it destroys great men.

There is a staggering amount of truth to that.

Except, another author, Neil Gaiman, in his masterpiece, the Sandman, made another observation:

It it is the prerogative of the fool to say the emperor has no clothes. But in the end the emperor is still emperor and the fool is still a fool.

I’ve spent a lot of time in my career thinking about this sentence.

Much of my career, I saw the fool as the person of power. The fool pointed out what was wrong and humiliated the Emperor. The fool was the source of truth, the Emperor a joke.

Much of my career has been motivated by this story. I wanted to be the fool by fool talking truth to power, and an Emperor that no fool can ridicule.

Such was my life until I discovered that there was a part of the story that nobody talks about.

It turns out that the Emperor arrives in his palace. And he’s furious and humiliated. And he thinks, “I’ve lost a measure of my power. My advisors will demand more control over the state finances and decisions or, worse, my resignation.” But then he remembers that he is still the Emperor.

“What an extraordinary opportunity, this fool has presented to me,” he thinks.

A meeting of all of the advisors is quickly called.

The Emperor, solemnly, announces, “We have erred. We erred in which advisors we trusted. And that faith has cost the Empire the trust of the people in the Emperor. And as Emperor, we must protect the Empire.”

The advisors who let this happen are delighted, the Emperor has been knocked down a peg.

The Emperor then announces a set of advisors who are to be taken to the dungeon and then publically executed.

Everyone is shocked. And then they look at the names, and those advisors slated to be killed were all enemies of the Emperor. Faction leaders who were untouchable and thorns in his side are now dead men.

And everyone realizes the crowd will roar with approval and the Emperor will be even more powerful and more beloved.

A few days later, the Emperor has a meeting with his head of secret police.

“We need to create a fool-astro-turfing program. When I need to get rid of some enemy, we will have a planted agent attack my policies. I will then use that to get rid of the enemy and change the policy and gain more power.”

“And what about that fool who mocked you?”

“I should hang him, but let’s make him a knight of the realm. And then we can kill him when he is forgotten.”

Meanwhile the people celebrate their Emperor who acknowledges his mistakes, rewards those who speak the truth and punishes his bad advisors!

The thing is, and I have learned this late in my career, speaking your mouth off and saying the truth is hazardous. If you do it without a plan, you have no idea what the outcome is. The folks in power will use it to their advantage to do whatever they want. And at least once in my career, speaking the truth cost me my job.

So was Aaron Burr right, “Talk Less, Smile More?”

Yes and no.

When you do speak truth to power, be aware of what the set of possible consequences are. Make sure that you do it in a way that maximizes your leverage to get what you want. And most importantly of all, be willing to live with the consequences.

At one of my former employers, there was a titanic attempt to build a new form of storage replication. And the team was under excruciating time pressure. One of the architects, we will call him Jim, objected,  but could not sufficiently articulate his position. The other architect, John, pressed for time, came up with what appeared to be a reasonable solution.

100 million dollars of NRE later, we had to throw out the solution because it didn’t work.

Why?

Because it turns out Joe was right. The approach didn’t correctly account for a particular property of how WAFL worked that was unique to file-systems.

Joe was saying that if someone were to take advantage of that property in the future, the proposed scheme would not work. At the time Joe was making that case, no one could imagine such a thing.  By the time John’s proposal was about to ship, the thing had been invented.

In short, John’s proposal worked in all cases but 1, and that 1 case turned out to be a massively important piece of technology.

So what went wrong?

Engineers are told, “you have to be good at communicating.” In other words, you have to be good at explaining your idea, selling your idea, and selling yourself. And there is a lot of merit to that.

But – and it’s a big but – I believe that there are moments where we have insight but cannot communicate. The bigger skill is listening, especially when the person providing the feedback can’t explain their intuition.

That experience of that project has been seminal in my life. And since then, whenever I try something on an old codebase, and those with intuition and experience look at me funny and can’t explain their intuition, I refuse to make progress until I can explain their intuition to myself and others. And only then do I move forward.

Most recently, this happened at VMware. One of the trickiest and thorniest issues in virtual machine management is the identity of a virtual machine. When I joined, I started an effort to come up with a globally unique and durable identifier. A senior engineer warned me that this wouldn’t work, and it would make some other systems impossible to implement. So we spent multi-hour days going over his intuition, the implication of his intuition, and the details around that intuition.

And he was right, and we decided to pursue a different strategy that gave us durability and locally unique identifiers.

Why do I bring this up? Because I sat in a meeting today, where thank goodness I had listened to him all that time ago. If I had not, we could not implement the new thing we wanted to.

Communication is a great weapon.  But listening is a great tool. Are they both equally important? I think you can get to a certain level of professional growth through great communication and a modest ability to listen.  But to get to the pinnacle of professional leadership requires listening and understanding even when others are struggling to communicate.

So recently someone said – “huh? You’re a fan of D&I? I hear it from VPs and managers and not tech-leads.”

And it got me thinking, really? I mean, I have been talking about this since 2013 – which is about 15 years too late.

First, I am a fan because it’s the right moral thing to do. And that’s enough for me. I can not stress that enough. In fact, I was tempted to end the email here.

—- but if the right thing isn’t enough —-

Second: BECAUSE I LIKE TO WIN.

When I was a young boy, I was told that all point guards need to be short as a matter of faith. Then this guy called Magic Johnson showed up.

Then later, I was told that a shooting guard could never win a championship. Then this guy Michael Jordan won 6.

Later, I was told that human beings that were over 5’10” could not win a sprint.

https://wrongtool.kostadis.com/moneyballing-recruiting-engineers-or-looking-for-usain-bolt/

If you read the post, you’ll understand the point.

But the short version is that victory happens when you avoid GroupThink.

Third: My dad

So my dad showed up in the USA in 1973 and spoke with this horrible Greek accent. And looked like a goon. And so he’s at a research meeting, and Sol Permutt is presenting on apnea. And my dad says – you should use a CPAP.

Sol Permutt looks at him and wants to escape from this smelly Greek (his words). Then he walks out and comes back in and says, “Wait, that’s Genius.”

Sol was willing to look past the Greek smelly guy and hear him.

If you sleep with a CPAP, it’s because Sol Permutt was willing to look past this Greek. He found to be unpleasant and listen to the idea. And because some admission team decided to bring him from Greece.

Fourth: Zynga

Zynga created a multi-billion dollar business because we weren’t gamers. We created games that gamers loathed. And along the way, it generated several billions of dollars of shareholder value and entertained millions.

——

I could go on and on and on and on.

I want a diverse set of people at the table where decisions are being made.

Because if there isn’t, we miss a perspective, and someone else who is listening will be making a fortune.

Photo by Robert Hrovat on Unsplash

Several years ago, in a post on Quora, I wrote that the only thing I feared as a software engineer was running out of money.

It was an odd realization on my part, and it has been the central guiding principle of my approach to software architecture.

So, I think it’s worth expanding on why I feel that way and how I use it to drive decision-making. At NetApp and Zynga, I had seen how financial decisions killed technology. I spent four years of my life working on a product called the net cash. The last thing I built was a multithreaded system that offered unique safety properties. It was the most intellectually satisfying thing I ever made. And it sits in a code repository collecting dust. Here’s the patent https://patents.justia.com/patent/7373640 for those that care.

At Zynga, I watched how a brilliant team that had built up a brilliant set of tools for operationalizing private and public clouds got destroyed in the space of four months because we couldn’t pay the bills.

My one personal regret is that I did not appreciate how much value there was in offering servers on demand.

What the experience taught me is that unless you are making money every quarter, you are going out of business. Any plan has to be making money now, or the organization will get destroyed.

So? I take a very pragmatic you towards software architecture. There are two critical questions.

1. What is the right answer to the problem?

As computer science is a science, we can reason about correctness. For a given problem and a given a set of constraints on computation and a bunch of desired outcomes, it is possible to articulate a correct answer that is independent of staffing or resourcing or timelines. Without knowledge of the right answer, it is impossible to know whether the problem has a solution.

So what?

Without the knowledge of a correct answer, it is elementary to spend a lot of time building wrong things. I am not talking about technical debt, but something so wrong that the only path forward is a complete and utter rewrite.

Let me give an example. Suppose you have a system where two entities must synchronize before the system converges to correct behavior. If the system assumed a human operator would observe and take action if the convergence doesn’t happen, and you wanted it to work without a human being, you need a new system.

Another example I like to use is the fallacy that it is possible to build a distributed system from a monolithic system by just adding RPC’s. The reality is that because of how RPC’s are different from a function call in a shared memory address. This approach doesn’t work. When failure modes get introduced in new places or timings of functions change, the system starts to break in all sorts of wonderfully wondrous ways. In my career, I have had to stop four different projects at four companies where this was the proposed direction.

Understanding whether the problem you’re trying to solve has a correct answer and that the proposal you’re making is correct is a critical element to any architecture.

In many ways, software architecture is proof of overall system correctness. For those who are knowledgeable in the field and understand the system well, it is possible to understand by reading the architecture spec whether the system is or is not correct.

So this brings me to the second important question.

2. What is the right answer, right now?

Money is what pays the bills. Often, I have found myself in debates over “the long-term answer” versus the “short term answer.” On the one hand are architects who are upset that we aren’t taking shortcuts to build the correct answer. On the other hand, some are frustrated that there is the immediate business value that could be derived if we were willing to make some sensible shortcuts.

I believe that any architecture that cannot deliver value right now is of no value. But wait, I hear you say you can’t deliver something in six months if it takes two years to build. True. But if you know what the long-term answer is, you can make better short-term trade-offs that move you along the long-term trajectory.

In effect, I believe that knowing what the correct answer is, allows you to evaluate whether the trade-offs of the short term answer are worth it.

Furthermore, understanding what the correct answer is, allows you to look at a short term answer and determine what set of use cases it will work for and what set of use cases it will not.

Finally, that understanding allows you to determine the business value of the short term answer. For example, if the short term answer is for 70% of the use cases, and that represents 97% of the revenue, then it may be okay. However, if it addresses 97% of the revenue that is shrinking rapidly and 0% of the revenue that is exploding, it may be a waste of time.

In conclusion, the notion that there is a trade-off between the long-term and the short-term is a fallacy. The long-term is always changing, and the short-term is what pays the bills, and software is a very malleable substance. Understanding the long-term allows you to make the right trade-offs in the short term such that the correct long-term answer is always within reach and that every step of the way, the exchanges are being made explicitly and not blindly. And software can always be changed. The question is how much.

In my professional career, one thing that troubled me is the statement from folks that “titles and money” don’t matter. That if you do good work, both will come, and that fixating on them was a bad thing.

My professional career has taught me the exact opposite – titles matter a lot and money matters. The only thing I learned over the years is the – how much – depends.

Let me offer a personal perspective.

[edit – forgot one more piece of the narrative]

In 2005, I was ready to quit NetApp. As a Senior Engineer at NetApp, I was never in the room where it happened. And so all of my ideas went nowhere. I had good intuition on what the storage management team needed to do, but my role made it impossible to move the ball forward.

After my wedding, and realizing that I could spend the next few years frustrated, I called an old friend and former boss and said – “I’m out because I can’t impact the company the way I want to.”

He took it upon himself to get me the role and title I wanted. And I spent the next four years at NetApp doing some amazing stuff.

Until the accumulated frustration and powerlessness to affect product strategy pushed me out again. I didn’t have the role and title to get myself heard about what the company wanted to do.

In 2009, I was looking for a job. I thought that NetApp’s strategy was wrong. I will observe that ten years later, the company seems to be pursuing a better strategy than what they were doing then.

When I went looking for a job, I made the decision that money be damned, I wasn’t taking a step back in the role. Companies that could not offer me a similar position were just not interesting. Zynga’s org structure at the time, had precisely the work I wanted, that of a CTO of a team with a large amount of operational freedom.

The cash money was lousy, and the equity was good.

In 2013, after Zynga changed its CEO to Don Mattrick, the company had to choose who to make the CTO. There were three excellent choices, and Don made a fantastic choice in picking Nick Tornow. I was disappointed it wasn’t me, and at the same time, I know Nick was a better choice. In fact, he is such a better choice, I spent several years trying to recruit him.

After that, I quit.

Why? Because the title mattered. Why did it matter? It mattered because it was a recognition and validation of the blood sweat toil and tears I had put into the company. And the successes I had been part of. It was a public statement of my accomplishments that my new boss had to acknowledge. When he didn’t, it was a personal statement about me. He disagreed with my contributions, and more importantly, didn’t see me the way I saw myself.

The money wasn’t that important. In fact, if I look back at the money I walked away from at Zynga, it was more than the money I made after Zynga until I lucked into a job at VMware.

After Zynga – I had to find a new job.

At the time, I was stuck between a rock and a hard place. Thanks to Zynga, I had made some money, and for a large but not insurmountable amount more, I could have had enough of a nest egg, that when I turned 55-ish, I could think of retiring as long as I didn’t dip into my savings.

The painful experience of that time was that I mismanaged my career. I came up with this cutesy title, “Chief Engineer,” instead of a title like GM or VP of Engineering. As a result, when I went looking for a job, and recruiters applied their ML algorithms to look for people, no one looked at me. I spent more time explaining why I had that title than what I did. And when I explained to them that I was a GM, but had no GM title, you could imagine the credibility gap.

I went looking.

And I had a CTO dream job with an old friend, but the guaranteed money of Juniper mattered more. Because of what was a non-trivial amount of money, my ability to fund my retirement, and my kid’s college education – it meant I had to take the job that paid more.

The role at Juniper was weird, I basically was working for a GM whose job it was turn around a company. I had no experience in security or networking, but I knew a lot about how to motivate teams and build software. The thesis was I would help with the team and software, and we would surround me with networking and security experts.

The money was excellent. And it solved a particular personal problem. If I could stick it out for three years.

Long story short, thanks to a hedge fund, a new CEO was hired, and then the new CEO made a series of unbelievably boneheaded decisions that lead to my layoff in about 1 year.

Because of the way the deal was structured, the three years of money was obtained in one year.

Having gotten that money, I was interested in what kind of job and impact and title.

In 2015 – VMware and nimble made competing offers. Nimble’s VP of Engineering created a great offer.  But the then GM at VMware, who was looking to hire me, made an excellent point – that being a VP of engineering matters. And that having that title on my resume from a company like VMware mattered.

He pointed out that as a VP, you have access to information, and you are at tables that you are not invited to as a non-VP.  And lastly, he pointed out how it will help my next job.

And so when I weighed the opportunity, I chose VMware. I believed at VMware I could do great things. But also I think I could have done great things at Nimble. The title VMware offered made it clear how much more the scope of impact was at VMware.

What I have learned from my experience and continue to learn from that experience is that titles matter. Maybe not for your current job, but for the next one. And money matters, because it’s how you choose what to do next.

And most importantly of all, titles are given to people who make certain decisions. And those decisions drive strategy.

Every career decision is very personal and context-dependent. There are times when I felt that someone made a horrible decision to pursue a title. But I had no idea what is going on with their lives, and so I respected their decision even though I didn’t understand it. My lack of comprehension was more about me not being them.

When I joined VMware, friends asked why? Wasn’t the future public cloud?

And I rejected that hypothesis.

The list below is an abridged version of a lot of deep thinking. These points have served me well over the years as I think about infrastructure.

One: Capex is more efficient spend than op-ex for things that can survive for more than 3 years

It’s taxed more efficiently, frees up cashflow, etc, etc. As power moves from gas/to solar, and spinning media runs on silicon, hardware can last about 1 decade. The systems don’t fail and the cost of running goes to 0.

If you know your capacity, you can literally buy once capitalize for 3 years, and run it for free for 7

Mainframes continue to survive for that reason.

Two: The Turing machine can run any program, and yet we have all kinds of hardware.

Why?

For any given workload, there is optimal hardware that will deliver the desired performance/reliability at an optimal cost.

Cloud doesn’t offer that hardware.

Three: Any prototype of a new system is best done in a typeless scripting language, any understood system is best done in a typed compiled language leveraging hardware

Every python project ever written that required performance or reliability had to be re-written in C/C++

Cloud optimizes for agility, not optimal execution.

Four: Computer systems that are inherently reliable are cheaper to operate than computer systems that are not

The single most important variable in making a system unreliable is how often it changes. A system that never changes, never breaks. The more people that touch a system, the more unreliable it becomes the more costly it is to operate.

Cloud infrastructure is always in-flux, thefore less reliable.

If you read about gaslighting, and you’re a half-decent human being, you may think you never gaslight. But if you are in a position of authority, unless you are careful, you do it all of the time.

As a junior engineer, you can ask a question, but as a senior architect, every issue is loaded. The person on the other side has a very different model for what is going on.

Authority implicitly changes every question.

Let’s take my favorite – “What do you think about my idea, X?”

The person on the receiving end is going to feel gaslit. If the big cheese is asking the question, and you say, “No,” does that end your career? And if you say “Yes,” and it turns out to be a bad idea, does that end your career? And is the big cheese asking your opinion or is he trying to get information about you and your boss?

You are pretending that they have an opinion when they don’t. It’s like asking someone,” Do you think I am fat?”

The other favorite is the” can we do this crazy idea alpha?” The person on the other side has no idea how to respond. If it’s an unfortunate but not damaging idea, the right thing may be to say yes and hope the big boss forgets what you said. If it’s a bad idea and dangerous idea, then you have to argue with the boss. And that sounds fun, but if the boss is committed to their concept, you became a naysayer. And can find yourself trying to keep your role and job.

As a leader, when you propose a solution to the problem, the debate has implicitly ended. And if the idea is a bad one, people are trying to figure out how to understand a bad idea. And thinking about a bad idea feels like your brain is being attacked, that your ability to think is being targetted.

What to do? The right thing to do as a boss is to frame the problem and ask for solutions, not propose them. Or, if you have a solution in mind, phrase it differently.

Instead of” what do you think about idea x,” a person in authority says,” I have thought a lot about this idea that I intend to implement, and I am trying to get a few more perspectives. I would love it if I could run it by you to see if I missed anything and to get your view.”

No longer is the person in authority faking a level of equality that does not exist. Instead, they are telling the truth. And the truth is they don’t care what that other person thinks, but they are interested in knowing if they forgot or missed anything.

But what if the person in authority is frustrated that they can’t get honest feedback? What if they feel that their subordinates are unnecessarily frightened? What if they do want to be challenged and are not?

It must be the spinelessness of their subordinates.

Nope.

It’s not the other’s fault; it’s the boss’ fault. For example, have they created an inclusive environment? If someone objects do they get attacked? Etc.

In a corporate environment gaslighting occurs when the boss pretends your opinion matters, but it doesn’t. Gaslighting occurs when the person in charge acts like they want to hear someone’s opinion, but don’t. Gaslighting occurs when the leader says every idea is on the table, but it isn’t. Gaslighting occurs when the decision-maker says that they will consider every idea reasonably and only attack every suggestion that they don’t agree with.

Learning how not to gaslight, and I am first among equals of those who have to improve, is a critical part of being a technology leader.