wrong tool

One of the trickiest parts of the job is how to start. If it’s a brand new system, at a brand new company, I am not sure I have any useful advice, and more practically, that’s not the kind of role this blog series is trying to describe.

After all multi-product system architecture of a single product with a single engineer is putting the cart before the horse.

Over the years, this process of how to become productive has tortured me through a series of jobs. In fact, in 2009, I almost got pushed out of Zynga because I hadn’t figured it out. And in 2008 I got my worst professional review because I hadn’t figured it out.

The crux of the problem is that when someone hires a very senior technical person, what they are hiring is an entire leadership org chart. They don’t realize that at the time. Worse, they don’t even realize how many new people they will have to hire. And worse the woman being hired has no clue as to how many people she will have to hire or fire.

At Zynga we had a high infant mortality rate with senior folks who would quit the company relatively quickly because we had a really bad on-boarding process. We paid them like they were senior, but they didn’t operate as senior folks because they lacked context and we didn’t help them get the context.

When I arrived at Zynga, Michael Luxton whom I mentored in the past, helped me. He basically told me the following: On the day I arrive I am an MTS 2. And should be given MTS 2 tasks. In about a year, I will be doing the job he hired me to do. It turned out to be a little bit faster, not because I was particularly capable, but because circumstances and luck made it go faster.

It was a bitter pill. And I almost didn’t swallow it. In fact, I was so frustrated that I almost walked out of the company one fine evening, frustrated that I was failing. But another great engineer talked me out of it.

At Juniper, and later at VMware, my bosses who were great VP’s of engineering provided a lot of structure in my onboarding and basically followed the same process.

But before I get to the process, let’s talk a little bit about what the problem is. The problem is that you don’t know where a company is on the technology curves. Is it a bleeding edge company that needs you to go invent something radically new? Or is it a laggard that needs you to fill feature gaps? Can your team write code but can’t architect? Or can it architect but not write code?

Put pedantically: can they write a block of code correctly? Can they write a function correctly? A module? A system? A product? Depending on where they are, the problem is different.

Is the problem not the code being written, and the architectures being proposed the release process? The build system? The schedules?

Is the problem the set of features the product team wants?

Does the team have the people who can build the features the product team wants? If you want to build a distributed system where you need world-class distributed systems engineers, and you don’t have them, you have the wrong team or wrong product ideda.

And they only way to figure that out is to get down and dirty.

So here’s the process I used for myself and I use for people who join the team. This is a process that my bosses codified when I joined VMware and Juniper and Michael Luxton intuited and mentored me through.

An architect is an imaginary architect – meaning she’s done nothing for you. And an imaginary architect can’t possibly be trusted to do anything. So step one is to make the imaginary architect do something that is not imaginary, for example, deliver code. The next step is to deliver a feature. The next step is to architect something. The next step is to sell something big they want to build. And to remind them every step of the way that they are imaginary until they accomplish all of the above. 

In general, my perspective is every new architect should spend a month fixing bugs in the core product they are working. They should fix a bug, check it in. Ideally several bugs. They should attend the scrum’s, the etc. And they shouldn’t pontificate but they should prove to their team that they can fix the bugs and do the work that an engineer can do. 

In fact at VMware, I only started to be taken seriously when I showed folks I could use a debugger. I remember one of the architects looking at me: WOW you know how to use a debugger. And I almost exploded in frustration because of course I know how to use one. But then realized that maybe if I had done that on day one, I could have saved myself some grief.

Once the architect has crossed that threshold, then and only then can we expand their scope.

Meanwhile we are feeding her the fire-hose of everything they need to do.

Let’s get even more granular.

After the architect has been hired, settled into their desk, you tell them: you don’t have any credibility with anyone on my team. Worse, you have anti-credibility because it means someone who thought they had the job no longer has it.

To get that credibility you need to go fix some bugs on product X, Y, Z. You need to own a feature in that product. You need to prove to people you can ship shit.

Until she does that people won’t take her seriously. 

She need to demonstrate you understand more about the system than they do. In pedantic detail.

The next step in my mentoring process is to tell the imaginary architect that she needs to get the team to buy into building something new. I tell her, “I don’t care what it is.” As far as I am concerned, the technical team must want to build the next thing you are selling.

Thirdly you need to sell to product management the thing you are building. 

And if they can’t sell to product management, tell them “I won’t overrule product management .” And if they say: What if I can’t? Like George Washington told Alexander Hamilton the only correct answer response: Well I suppose we’ll have to find an architect who can. Go figure it out!

And at that point, you no longer have an imaginary architect, you have a real one. And guess what during that entire process, the architect has been delivering value, gaining credibility with the team, and learning the corporate values and when they tell the team go left, the team is willing to follow them.

Recently one of my co-workers asked about how do I value innovation. And so I started with a formulation I described before (here: https://wrongtool.kostadis.com/a-relativistic-theory-of-innovation/).

And in our discussion, and because many years have passed since I first wrote about it, I realized I was using a more sophisticated model.

The most significant bit from that earlier post is that there are regions of innovations. And within an area, you can innovate, and the next thing to do is reasonably apparent to those who are in that region and may appear as magical if you are further behind.

When I think about innovation and products, I think about these four regions:

1. The edge of human knowledge
2. Innovative commercial products
3. Mainstream products
4. Laggard products

Most academic research is in the first region and tends to be 5-10 years away, at least, from being part of some innovative product. And most research is intellectually exciting but of marginal commercial value.

Being able to predict which of the research technologies will win is impossible, and the surface area of opportunity is huge, and the investments that are required to turn those ideas into products is vast.

The biggest commercial wins of all time are when someone can take a research idea and turn it into a commercial product. Examples of such things include – server virtualization and network virtualization, the search of the web, cloud computing.

However, as a strategic software architect, the job isn’t to find that disruptive new technology, the position is to keep a sizeable successful software business successful and relevant.

Searching for that next disruptive idea is only part of the equation.

The equally important part is managing your investments in delivering commercially viable innovations, ensuring that the mainstream market is satisfied with the product and that you never fall behind the mainstream requirements and become a laggard – primarily avoid four at all costs, always be on 3, and do some of 2 while looking for 1.

A challenge is that although a strategic software architect may be thinking about all four regions at the same time, most teams are in one place and are focused on their backlog for their region.

If your goal as a software architect is to move a team downstream from being pure research or upstream from being a laggard, you will effectively deprioritize the obvious set of next work with something different. Something more risky because it’s not the obvious next thing.

And if you are a laggard, the product will be suffering from a lot of code rot problems. And just cleaning it up may feel like the top priority. 

And that would be the wrong thing. 

You are a laggard because you are missing features. 

If your code sucks, and your build times are horrible, etc., it’s tempting to think that you are a laggard because of that. In fact, that is irrelevant. You are a laggard because in the market you have a feature gap between the laggard and mainstream market products.

The real objective is to own most of the mainstream market and to offer innovative features that capture premium value while hoping that you discover some groundbreaking innovation.

And because you are a laggard, the market doesn’t look for you to innovate, they look for you to deliver the things that they want in the way that they want. The mainstream has defined its requirements, and you must fit into those requirements. A lot of times the temptation is to argue with the mainstream market, but that’s the wrong strategy. The mainstream isn’t interested in innovations, it has a particular point of view, and you need to satisfy it.

There may be innovative ways to deliver those solutions, but they have to meet the requirements of that market even if those requirements are – for lack of a better word – wrong.

What this means is that you have to add – sometimes – even more technical debt to get to the mainstream market so that you can have the revenue dollars and credibility to innovate.

In any new situation, the first job of a strategic software architect is to validate if the product meets the mainstream requirements and if not, all energy has to be focused on that.

Warning signs that you are not meeting mainstream requirements include declining market share, the field losing trust, emergence of direct competitors offering the same kind of product that is bought and deployed in the same way, decaying investment in the core product.

You have to direct your first set of investments and energy at getting out of the laggard space.

At NetApp when I joined the storage management team and was made their architect, this was precisely the situation we were in. The product, DataFabric Manager (aka DFM) had had marginal investment, and suddenly storage management had become a thing that customers cared about. Our field referred to DFM as Doesn’t F*king Matter and our customers were livid.

The first thing we had to do was identify the set of feature gaps that we needed to close to make the product not suck. And to be fair, the product didn’t suck given the investment. What had happened was that the market requirements had shifted on the team and the company had not made the investments to meet those requirements.

Once we identified those feature gaps, we had to code like mad. I remember sitting in a room with the head of engineering, and he said: Kostadis, I need you to disappear for the next few months. We need to code like mad. And I can’t have you distract anyone with your great next set of ideas.” The engineering head was dyslexic, and he never wrote anything. When he wanted to emphasize things he would take a pen and write it down, and he wrote “Code like mad!!” And he underlined it twice. I wish I still had that paper.

And so we coded like mad and got out of the laggard space.

I like to joke this is the easy part. The requirements are understood, the team top-to-bottom understands what it needs to build, the only problem is you have to go very very very fast and have to be very very very disciplined. The hard part is that a laggard product has a lot of internal challenges – a lack of resources, talent and brain drain, and a lot of internal corporate vultures. The internal corporate vultures are the worst. Instead of fixing the laggard product they decide that the right answer is to buy another product, or to do a rewrite or to fire everyone or to do anything but fix the feature gaps.

It’s mind-boggling. In fact, if you walk into a situation where everyone wants to re-write the core product, it probably is indicative of a laggard product.

And the right answer isn’t to do a rewrite but to fix the critical feature gaps.

So when we delivered that next DFM release, we didn’t advance the architecture, we didn’t solve the underlying technical problems, and we may have added some tech debt, but we got to the mainstream, and that set us up for trying to get to the innovative product space.

And we did.

A year ago, in a promotion meeting, a senior technical leader warned us about promoting someone too soon. And he and I rarely agreed on anything, and I always learned something from our discussion.

His comment is that to be a successful technologist you need to have an area under the curve not just cross the threshold. And that time and experience were how you accumulated that area under the graph.

At NetApp, in my 20’s, I was determined to become a technical director as fast as possible. And I crossed the threshold, and they promoted me, and then it took me one more decade to learn the actual job. I was 33 years old, and I was the second youngest technical director.

And I had to have a lot of failures, and experiences before I actually could do the job properly. And I had to learn a lot from people.

For example, one thing I had to learn and then re-learn, is that in a new market all of your instincts are wrong. And I also had to learn that in all markets some things are always true, there are no shortcuts.

What I realized is that the sentences and verbs never change, but the nouns and adjectives do.

A lot of the job of strategic software architecture is not about technology but pattern matching.

And the job is to understand the details so you can map the right sentence. And that was a lesson I had to learn the hard way.

For example, when I went to Zynga, I sat in meetings, and people were using words that I had never heard before used in ways that made no sense. In my first meeting with the executive team, they said: “We need new IP”. And I was flabbergasted – why would a game company need a new internet protocol? Except they meant “Intellectual property” which meant “new game franchise” which meant “new product” and that the real discussion was about how do we launch a new line of business?

The sentence, in this case, was “How do we launch a new line of business,” the noun was “new IP.” And once you realize that a game company spends all of its time launching new games, it makes you rethink what the purpose of software is.

The solution, again from experience, was that before I could do the job they hired me for, I had to learn about the technology. And so I spent several months taking notes on words and asking questions, and within a few months, I had begun to match patterns.

When I went to Juniper, after Zynga, I took the same process. First, learn the nouns, and adjectives, then do the pattern matching and then start figuring out what needs to be done.

Being able to do that job, the pattern matching, you need the ability and the experience, and so getting promoted fast is great, but it takes time to be able to do the job.

The most intriguing part of the tech industry is the interplay between Moore’s law and opportunity.

My rule of thumb is the 8-year rule. The rule says the following; market disruptions happen every eight years because the incumbent software stack can’t be adapted to the new hardware.

As some examples…

My current employer took over the world because of the shift to multi-core servers. Apple took over the world courtesy of mobile processors becoming fast enough to run most useful applications. Microsoft and Linux took over the world when  Pentium closed the performance gap between x86 processors and RISC systems.

But why 8, because we human beings, don’t understand exponentially growing curves.

For small enough values of x, y=x*x is equivalent to y=2x. The two start to diverge when x = 3  or when y = 8.

So?

Let’s assume big tech titan has 80% of the market in year 0. Then in year 2, the new hardware emerges, but it’s not appreciably faster, except for some small use cases, so instead of selling to 8/10 customers, it’s now selling to 8/12. Then in year 4, it’s 8/14, and then in year six it’s 8/18, and you go from being 80% of the market to less than 50%, and there is probably some other tech-titan growing much faster than you were and you are the has-been tech-titan?

Many books and articles cover this topic.

What does this have to do with anything?

When you consider strategic software architecture, the tricky bit is to navigate across that 8-year transition.  And what makes it particularly tricky is that you have to assume that the software you had is going to be your boat-anchor, and simultaneously, your source of funding.

The challenge is that a mature software architecture that’s tuned for one market takes about 8 years. And the reason it takes about 8 years, is that any market takes 8 years to mature. And the reason it takes 8 years, is that it takes 8 years for the hardware to become capable enough to grow the market. And if your software is tuned for one market, it is not tuned for the next one.

How do you solve this?

The short answer is that you have to find the 8-year curve after the current one.

The strategy is the following:

1.  Grow with the first 8-year cycle
2.  Preserve market share during the 8-year cycle you missed and figure out what you will do next. Use this time to re-architect your system for the next grow opportunity.
3.  Grow with the next 8-year cycle.

The tricky bit in my mind is to understand when you are in (2) and to realize that your goal is not to continue to improve what you had but to go build something new that builds on top of the assets you have in place.

Where I have failed in the past is not understanding the importance of (2). That it’s tempting to see some market you missed and try to attack it and repeatedly fail, instead of admitting you blew it and then trying to find the next thing.

So how do you set yourself up for (3)?

You have to think about where the puck is moving to, and then do everything while you are in phase (2) such that it lines up with where you think phase (3) will be.

And that’s the trick, to do the major re-architecting nominally for phase (2), but really for phase (3). And as a strategic software architect, that’s the hardest job, selling the future as an improvement on the present when the entire company is obsessed with a market they already lost, and with a market, they could win at not yet visible.

 

 

 

 

 

One of the enduring myths about software architecture and in general technology leadership is the degree of control an architect has.

Our advocates believe we walk into a room, draw a picture, everyone listens to us and then code materializes.

And I have worked in places like that. Teams willingly were lead off a cliff.

I remember a time at NetApp where a team just wanted me to draw the picture. And I did and then projects got spun up and engineers got assigned. And then I left because that’s not the way I work.

The best teams don’t work that way.

The best teams draw the picture for you and you evaluate if the picture makes sense.

What you want is people to be passionate and believe in their solution. And only very rarely is what they are proposing wrong. Most of the time it’s good enough.

And so my job is to find out how to nudge them away from potential disasters.

Sometimes it can be exasperating because it’s not the picture you want drawn. And sometimes the picture is a compromise between organizations and not the best software. And there is always someone smart enough to point out the better picture that no one had any passion for.

And then they look at me as a failure, because isn’t my job to build the best possible system and force it down my teams throat?

And the answer is almost never.

The job is to have product that can evolve to be the best product it can be. And to do that you need a committed team. And a committed great team will always produce great software even if the picture isn’t exactly what you would have drawn.

Because drawing the picture – ironically – is almost never the job.