wrong tool

You are finite. Zathras is finite. This is wrong tool.

Rage quitting

by Leave a Comment

The only rule in tech is you never, ever, ever, rage quit.

What’s rage quitting?

It’s getting so pissed off, that you walk out the door and take whatever next job you find.

Now, let me point out there are circumstances where rage quitting is the right thing to do, and that’s where there are moral and ethical lines being crossed that you can not abide. Or where someone does something that is truly horrible. I have no experience in those kinds of circumstances, and I offer no useful advice. All I can say is – do what you need to do to be safe, and talk to a labor attorney.

But if you aren’t in one of those really bad places, why is rage quitting a bad plan?

First, it’s a bad plan because your next job is probably not going to be the next job you actually want. And so you’ll be looking for another job pretty soon after that.

Second. it’s a bad plan, because the person who pushed you to rage quit, is forcing you to leave on their terms, not yours.

Third, because it puts you in a vulnerable position. A lot of very bad people can tell if you are the kind of person who will rage quit. They will exploit that, to get you to quit. They will push your buttons, and get you so angry that you will quit.

And they don’t get to run your life, you get to run your life, and surrendering that much control to them is never a good idea.

So you are angry, and furious and pissed off and want to quit, what do you do?

First, take some time off. You can take time off at work, you can take time off at home, you can find the mental space to not be angry. As Joe Beda said, don’t rage quit, rage vacation.

Once you are done with the vacation, then figure out what the right next move is. Be honest with yourself, and determine what you want. In some cases, it’s going to find another job. In other cases, its talk to your boss or his boss about the fact you wanted to rage quit, and see if a change is possible.

But figure that out.

And then go get it.

They is quitting, and what to do about it

by Leave a Comment

Over the years my teams have gone through waves of attrition. And typically those waves of attrition hit me like a punch to the gut, because every time they have happened, I was asked to do a dive and catch.

I was asked to try and convince someone to not take a job that they had already accepted. And that’s a tough place to be.

And it sometimes worked, but most often did not.

And even it did, they probably quit within 6 months.

And especially for senior people, their departures at critical phases of a project or business was very disruptive.

So if you get int that situation, what can you offer?

1. Offer more money

2. Offer another title

3. Offer time off

4. Offer a new project.

But the problem with these offers is that you are just buying yourself some time. Whatever underlying issues caused the person to quit will resurface after 6 months.

And what really upset me was that I was always felt they were leaving, not going. That we failed to show them the opportunity and deal with their real legitimate grievances.

So why bother? Because you get six months to figure out, on your terms, what the transition plan should be.

And that brings me to what you actually have to do.

From painful personal experience, I learned that 1×1’s are the best path to ensuring that folks are feeling connected to the organization.

The more 1×1’s you have across the org, the more people are connected to the team, the more you can deal with problems earlier rather than later.

Sometimes people leave, and that’s okay. Change is not a bad thing. But you owe it to your team for them to feel that there was a better opportunity somewhere else, and that they are not fleeing a bad place.

And so in my latest gig we mandated that our teams and leads do 1×1 and report the number of 1×1’s. And our attrition rate dropped to historic lows. Yes, part of it was the awesomeness of the Project Pacific, and our stock price, and part of it was that folks were feeling very connected to the mission as a whole.

And I am doing a lot fewer dive and catches these days….

We live in a magical era.

by 5 Comments

Last night, my father-in-law and I were discussing the notion of truth, and in particular, how the anti-Vax movement exists and why is the truth is so malleable.

And what I observed, and thought may be worth sharing, is that the human-made world is as magical as the natural world. And that has profound implications for us as human beings.

What do I mean by magical?

If we consider something like math, there are fundamental axioms that allow us to create or discover theorems. Those axioms make it possible to understand the world more deeply. Those axioms are not provable.
As mathematics is layered, a typical mathematical proof will have some reference to some theorem that has been proven elsewhere, that the paper hinges on.

And mathematicians assume that a theorem is correct. It is right because it both can be used to derive more math, and because it has resisted the assault on its correctness from other mathematicians.

Every so often, however, a theorem is proven to be wrong, or unproven. When that happens, a whole sub-branch of mathematics that depended on that theorem dissolves into nothingness.

In short, mathematicians rely on the assumption that the other parts of mathematics are correct.

They believe the theorems are correct, without necessarily understanding the entire depth of their correctness. Mathematicians also simultaneously understand that at some fundamental level, their truth depends on unprovable but useful axioms.

And that faith is magical thinking. We believe that the system works; therefore, it works, while realizing it may not.

And is this fundamentally different from the Ancient Greek perspective on Helios? After all, they believed that a titan dragged the Sun was dragged across the sky. And that explanation, although absurd, had sufficient predictive power that the farmers of the time could make other inferences and assumptions. And it was a perfectly workable model of the Sun. Absurd, but workable.

In short, the mathematicians’ faith in a theorem is the same faith of the Ancient Greek farmer in a titan who drags the Sun across the sky. Both could be proven to be absurdly incorrect, and both useful.

My intent is not to suggest that mathematics is not rigorous, but to illustrate how the most stringent of intellectual disciplines is also based somewhat on faith.

But why do I say that the world is magical?

Because, in the past, the human-made world was not magical. A person could understand everything about his house. They could understand how to build it, and they could understand how to make the tools they used, he could understand why some tools worked better than others. The natural world was magical, but the person-made world was rational.

The triumph of modern science is a person-made world that is as magical as the natural world.

The rationalist project to reduce the natural world to the well understood human-made world failed. We know that we don’t know and must make assumptions.

And if it only ended at science, the rest of us could live in a perfectly rational and understood world.

But the engineers created a magical world that defies understanding. Consider the micro-processor. It represents several hundred thousand person-years of engineering and science (if not millions). No one can understand every piece of it, because there is not enough time in a human life-time to understand it.

Or consider the reader of this blog, that the set of technologies and science to make reading this blog possible, make the Apollo project look like a ten-person startup.

So engineers must rely on faith. Things work, so we continue to use them. And we build stuff on top of them.

But unfortunately, this faith has destroyed the truth. Since no one can understand the human-made world, based on a natural world we don’t understand, then everything is based on faith.

And if we base everything on faith, then everything is magical. We believe things to be true, because they work and because no one has disproved them or because they are useful. And if some truths are useful, then others are also useful. And if all truths are contingent and based on assumptions, then …

Why then it becomes a natural slippery slope to anti-Vaxers. Their truth is just as valid as anyone else’s in their minds; they have a different faith. They have a particular faith that science is wrong. They have a particular belief that big-pharma is evil. They have a particular faith that they have been lied to.

Because it turns out that everything about our world is magical. Everything we touch works because we believe other parts work, and as long as all of our assumptions hold, things hold. But, deep down, we all know that these assumptions are fragile things, like the mathematician and their axioms. And we know that something we don’t understand could up-end them, like the Ancient Greek farmer and his belief in Helios.

18 architecturalist papers: As was foretold.

by Leave a Comment

Recently, a coworker of mine approached me and said, “All you have to do is figure out what the final right answer is.”

And I stared at him, and I was surprised at how ridiculous the comment sounded.

I turned to him and I said, “anyone can do that!”

In fact, it was at that point in time that I realized how little I understood what the nature of system architecture is. Figuring out how to build the correct answer is the least interesting part. Figuring out how to build the next part, while giving you optionality to build the correct solution later is the real job.

The job is to see what is being done today, understand where you want to go, and course correct efforts that are going in the wrong direction.

System architecture lives between the now, and perfect future, an area of complete grey. And the challenge is in that gray area; there are no correct answers.

The Minbari Grey Council is a perfect metaphor. Their job was to stand before the now, and the future that they knew was coming and making the hard choices. That space between the now and the future was unclear and uncertain. And they chose, when confronted with the future, to not make a choice.

The job of the system architect, is to know when the right thing to do is to break the Grey Council and when it is not.

The challenge for the system architect is that when you see a project that is going off the rails, you need to understand how much you need to get involved. Is this a project that if it succeeds will take the company in the new wrong direction? Or is this an effort that will open new opportunities that currently don’t exist? The height of hubris is to assume you know the answer. But to do nothing, is to say yes to everything.

Ultimately system architecture is a reflection of the taste of the architect. And like fake turning machines, not all taste is correct all problems.

The challenge, is to understand when you were taste is getting in the way of new opportunity and when your taste is telling you that something is going wrong.

17 architecturalist papers: go fast and build things

by Leave a Comment

Over the last ten years, I have struggled with a Facebook motto: Go Fast and Break Things.

It reminds me too much of the Great Gatsby quote:

They were careless people, Tom and Daisy—they smashed up things and . . . then retreated back into their money . . . and let other people clean up the mess they had made.

The process “go fast and break things” does not describe a method for creating value; it represents a process for an adrenaline high, an excuse to do whatever you want.

Let me ground this into a real-world example. Suppose I have a system, and I want to radically re-imagine what the system can do.

There are two paths.

The first is to create a brand new system, that is entirely incompatible and breaks everything. What do I mean by everything?  Any system is part of an ecosystem of tools and operations, and people that interact with the system. When you break the system, you are breaking that web of relationships and interactions. The net outcome is a radical change of that web.

So why do it? Well, because the cost of that change is borne by the people who use the system, not by the people who built the system. The more powerful the market position, the easier it is for the entrenched system to break things.

For example, Facebook used to break APIs all of the time. And that was okay for Facebook, because their captive audience, had no choice but to change to use the new APIs. The consumer-owned the full cost of the disruption.

The second approach is to evolve the system in a way that doesn’t break anything. In this model, instead of forcing the world to adapt to your system, you figure out how to integrate into their world.

Intel is an excellent poster-child for both of those. The first time was when they delivered the Pentium. At the time of the 486, there was a bunch of RISC processors like MIPS, Alpha, SPARC, that those of us in the field thought had a legitimate chance of dethroning Intel because the CISC core was intrinsically slower than RISC systems. But getting off of Intel meant breaking things. Instead, Intel did something that was a surprise to the casual followers of the industry, they embedded a RISC-like core into their processor, preserving the CISC instructions. By choosing not to break things, they won the CPU core wars.

Ironically, a little later, Intel pursued a foolish – in retrospect – strategy of the Itanium. The thing about the Itanium was that at the time there was no 64 bit Intel processor. Intel planned to move away from the x86 instruction set to go to a new kind of instruction set. The switch was highly disruptive. And AMD delivered what the market wanted, a 64-bit x86 processor and achieved a huge market opportunity.

In both cases, the winner deliberately chose to break as little as possible and add value in a way that did not disrupt the consumers of the technology.

To me, that is the best kind of engineering.

16 architecturalist papers: you work for the future GM

by Leave a Comment

One of the most challenging parts of the job of strategic software architect is that your job is to think about the future, and the GM’s job is to think about the present. And what’s worse, your planning horizon is typically beyond the planning horizon of the current GM.

Why is that a problem? Because we hate our future selves.

There is a lot of behavioral research that suggests we hate our future selves. We will do things that optimize for current happiness vs future happiness. Explains so many things about our choices. 

This, for example, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5611653/ or a more accessible and possibly more useful version here: https://www.anderson.ucla.edu/faculty-and-research/anderson-review/future-self-health

And this leads to my favorite story about the conflict between GM’s and their Strategic Software Architects.

In 2006 I sat in a room with Guy Churchward while at NetApp.
He was the new GM, and it was our first 1:1. And I told him: Look, Guy, you’re going to be gone in 18 months. And I’m going to be here for 3-5. My job is to make sure you don’t screw this technology up. If you do something I think is stupid, I will assure you it will not happen. Because I will make sure the dumbest, worst engineers are working on it. If you want to do something smart, I will make sure it succeeds. So you need to get me on board. And even if you get the smart guys to work on it, I will undermine their success, because it’s my job to make sure there is a technology that the next guy sitting across me has to go to market with.
It was a breathtakingly arrogant comment. But it was a true comment. Guy looked at me wondering if Technical Directors at NetApp could be fired.
Unfortunately for NetApp, he left way too soon from his job. And I left shortly thereafter.

He left not because he failed, but the nature of the GM job tenure is less than the tenure of the strategic software architect, and that is by design. We want the GM to be more short term focused and we want the strategic software architect to take the longer view.

The challenge is that we are working for the next GM. And the current GM is not interested in helping the future GM who is probably going to be somebody different. 
So the architect – in some sense – is that person who gets in the way of the current GM plan’s to help the future GM, someone the current GM hates (even if it’s him in 2 years). 
So then what? 
As architects, we are constantly fighting our current boss.
How does this manifest itself concretely:
If I am a GM and I have a product, to hit my numbers, I only need junior engineers. But if none of those turn into senior engineers in 1-2 years then product will have problems in 4. And if none of them turn into architects in 3, the product is dead in 7.
We can argue about the dates and ranges but the story holds true. If you don’t have senior technologists thinking about the future, then you miss the future.
So now what?
As strategic software architects, our job is to make the current GM think that the future GM is working for him.
Here’s how I always try to do that.
1. Make sure that the Strategic Software Architecture is something that the current GM will profit from. A GM has to make sales to companies who want to believe the company has a future, he has to attract technologists to build the current stuff, having a compelling technology strategy is very useful for both.
2. Make sure that the Strategic Software Architecture adds value all of the time to the current GM. This means that future pieces deliver value now.
3. Hire for the people you need to build the future, and have them build the present. This is this weird thing. You bring in someone to build your future, and have them work on the immediate problem. While you are doing that you wonder if you are making a mistake. I think of it as a twofer. The new gal learns something new AND gains credibility AND will build the future thing better.

4. Be flexible in planning. Every new GM will have new priorities, so be willing to change what you recommend to be built.

Is the great cloud shell game over?

by 3 Comments

With the success of VMC on AWS, it’s time for us to admit that the Cloud Native programming model as the dominant and only programming model for the cloud is dead.

The Cloud Native programming model was a godsend for IT and software engineers. The move from CAPEX to OPEX forced all of the pre-existing software that ran on premises to be completely re-written for the cloud.

Jobs, careers, and consulting firms exploded as everybody tried to go on this cloud journey.

It was like the great y2k rewrite, which was followed by the C++ rewrite, which was in turn followed by the great Java rewrite …

This rewrite was forced because On-Prem software assumed infrastructure that did not exist in the cloud and could not work.

On-premises, you have very reliable and robust networks, storage that offers 5 9’s reliability, and a virtualization infrastructure that provides automatic restart-ability of virtual machines.

Furthermore, on-prem you had the opportunity to right-size your VM to your workload instead of playing whack-a-mole with the bin-packing strategy known as “which cloud instance do I buy today?”

The problem with on-prem was that you had to buy the hardware, and worse you had to learn how to operate the hardware.

The operating environment for the cloud where networks are unreliable, storage is unreliable, and applications must be HA aware and integrate with HA aware systems required a full-rewrite of perfectly fine working software.

What motivated this mass migration and mass rewrite?

The motivation was that new software could legitimately be written faster on the EC2 PaaS. Furthermore, companies didn’t want to own the hardware and wanted to rent their infrastructure.

The two factors pushed companies to look at Cloud Native not as a way to augment their existing software assets, but a once-in-the-lifetime opportunity to rewrite their software assets.

But it turns out that is hard. And it also turns that the pre-existing operating model on premises is kind-of-valuable. Instead of every application having to figure out how to deal with infrastructure that’s flaky, it’s just simpler to have a more robust infrastructure.

And now that the cost and agility advantage of the cloud has been in-part neutralized, what I hope we might see is a collective pause in the software industry as we ask ourselves – not whether it’s on-prem or cloud-native, but what is the appropriate programming model for the task at hand.

15 architecturalist papers: The Turing Machine Fallacy

by Leave a Comment

One of the most enduring mistakes software architects make is what I call the Turing Machine Fallacy. The argument goes like this.

1. My system solves this fantastic class of problems

2. This system will address all problems.

3. Therefore, there is no room for any other system

The most recent example of this fallacy was the cloud. Four years ago, when I joined VMware, everyone assumed that the public cloud was the future. The assumption was that a small set of public cloud providers would provide the infrastructure that everyone would consume. That computing infrastructure was fundamentally undifferentiated and, therefore, not something that would be worth investing in.

I didn’t, don’t, and never will agree.

I believed when VMware’s stock price was in the mid-’50s that this was a ridiculous proposition.

In short, my point is the following. If you believe that a single computing infrastructure will meet all computing needs, you also believe that all software will run on a single computing platform and that that computing platform is X.

We have a name for X, it’s called a Turing Machine, and last I checked, the guys selling paper were not making a lot of money.

I believe that the minute the industry has coalesced around yet another fake Turing Machine, I need to start looking for the thing that will replace it.

But why do I believe this?

Because fundamentally, the software is an approximation of the real world. The software is a model of the real world. It is not the real world. And the real-world changes. And when the world changes, the software approximations become increasingly ill-fitting until they no longer fit. And changing software is very, very hard.

Changing software is so hard that new software fits into the gaps between the current software and the real world.

The strategic system software architect’s role is to recognize that your job is to see where the approximations are ill-fitting and cause investment to happen in solutions that will fit into those areas.

And those investments are software that is – by definition – different than the current winning system architecture. And those investments will drive hardware investments to support that software. And the hardware architecture, which is, in turn, an approximation of reality, will change. And as the hardware architecture changes to adapt to the change in software architecture, the winning system stops being the final answer to software system architecture.

We realize that the system architecture is not a universal Turing Machine but a computer.

Hardware evolves to support software, which is continuously evolving to support a changing world. Performance, form factor, power consumption, and legal requirements are in constant flux, and therefore the needs change as well.

And so to end on a proof point, in 2015, everyone assumed that Amazon would own The Cloud. And yet, here we are, and what is clear is that there will be a plethora of clouds – private, public, authoritarian, IoT, etc. Each with their optimizations that deal with their specific requirements.

The universal Turing Machine is an excellent mathematical abstraction, but there is no such thing in the real world where I live and work.

14 architecturalist papers: the imaginary architect

by 4 Comments

One of the trickiest parts of the job is how to start. If it’s a brand new system, at a brand new company, I am not sure I have any useful advice, and more practically, that’s not the kind of role this blog series is trying to describe.

After all multi-product system architecture of a single product with a single engineer is putting the cart before the horse.

Over the years, this process of how to become productive has tortured me through a series of jobs. In fact, in 2009, I almost got pushed out of Zynga because I hadn’t figured it out. And in 2008 I got my worst professional review because I hadn’t figured it out.

The crux of the problem is that when someone hires a very senior technical person, what they are hiring is an entire leadership org chart. They don’t realize that at the time. Worse, they don’t even realize how many new people they will have to hire. And worse the woman being hired has no clue as to how many people she will have to hire or fire.

At Zynga we had a high infant mortality rate with senior folks who would quit the company relatively quickly because we had a really bad on-boarding process. We paid them like they were senior, but they didn’t operate as senior folks because they lacked context and we didn’t help them get the context.

When I arrived at Zynga, Michael Luxton whom I mentored in the past, helped me. He basically told me the following: On the day I arrive I am an MTS 2. And should be given MTS 2 tasks. In about a year, I will be doing the job he hired me to do. It turned out to be a little bit faster, not because I was particularly capable, but because circumstances and luck made it go faster.

It was a bitter pill. And I almost didn’t swallow it. In fact, I was so frustrated that I almost walked out of the company one fine evening, frustrated that I was failing. But another great engineer talked me out of it.

At Juniper, and later at VMware, my bosses who were great VP’s of engineering provided a lot of structure in my onboarding and basically followed the same process.

But before I get to the process, let’s talk a little bit about what the problem is. The problem is that you don’t know where a company is on the technology curves. Is it a bleeding edge company that needs you to go invent something radically new? Or is it a laggard that needs you to fill feature gaps? Can your team write code but can’t architect? Or can it architect but not write code?

Put pedantically: can they write a block of code correctly? Can they write a function correctly? A module? A system? A product? Depending on where they are, the problem is different.

Is the problem not the code being written, and the architectures being proposed the release process? The build system? The schedules?

Is the problem the set of features the product team wants?

Does the team have the people who can build the features the product team wants? If you want to build a distributed system where you need world-class distributed systems engineers, and you don’t have them, you have the wrong team or wrong product ideda.

And they only way to figure that out is to get down and dirty.

So here’s the process I used for myself and I use for people who join the team. This is a process that my bosses codified when I joined VMware and Juniper and Michael Luxton intuited and mentored me through.

An architect is an imaginary architect – meaning she’s done nothing for you. And an imaginary architect can’t possibly be trusted to do anything. So step one is to make the imaginary architect do something that is not imaginary, for example, deliver code. The next step is to deliver a feature. The next step is to architect something. The next step is to sell something big they want to build. And to remind them every step of the way that they are imaginary until they accomplish all of the above. 

In general, my perspective is every new architect should spend a month fixing bugs in the core product they are working. They should fix a bug, check it in. Ideally several bugs. They should attend the scrum’s, the etc. And they shouldn’t pontificate but they should prove to their team that they can fix the bugs and do the work that an engineer can do. 

In fact at VMware, I only started to be taken seriously when I showed folks I could use a debugger. I remember one of the architects looking at me: WOW you know how to use a debugger. And I almost exploded in frustration because of course I know how to use one. But then realized that maybe if I had done that on day one, I could have saved myself some grief.

Once the architect has crossed that threshold, then and only then can we expand their scope.

Meanwhile we are feeding her the fire-hose of everything they need to do.

Let’s get even more granular.

After the architect has been hired, settled into their desk, you tell them: you don’t have any credibility with anyone on my team. Worse, you have anti-credibility because it means someone who thought they had the job no longer has it.

To get that credibility you need to go fix some bugs on product X, Y, Z. You need to own a feature in that product. You need to prove to people you can ship shit.

Until she does that people won’t take her seriously. 

She need to demonstrate you understand more about the system than they do. In pedantic detail.

The next step in my mentoring process is to tell the imaginary architect that she needs to get the team to buy into building something new. I tell her, “I don’t care what it is.” As far as I am concerned, the technical team must want to build the next thing you are selling.

Thirdly you need to sell to product management the thing you are building. 

And if they can’t sell to product management, tell them “I won’t overrule product management .” And if they say: What if I can’t? Like George Washington told Alexander Hamilton the only correct answer response: Well I suppose we’ll have to find an architect who can. Go figure it out!

And at that point, you no longer have an imaginary architect, you have a real one. And guess what during that entire process, the architect has been delivering value, gaining credibility with the team, and learning the corporate values and when they tell the team go left, the team is willing to follow them.

13 architecturalist papers: getting to mainstream

by Leave a Comment

Recently one of my co-workers asked about how do I value innovation. And so I started with a formulation I described before (here: https://wrongtool.kostadis.com/a-relativistic-theory-of-innovation/).

And in our discussion, and because many years have passed since I first wrote about it, I realized I was using a more sophisticated model.

The most significant bit from that earlier post is that there are regions of innovations. And within an area, you can innovate, and the next thing to do is reasonably apparent to those who are in that region and may appear as magical if you are further behind.

When I think about innovation and products, I think about these four regions:

  1. The edge of human knowledge
  2. Innovative commercial products
  3. Mainstream products
  4. Laggard products
I also think of these regions as waves, where the innovations in the bleeding edge wage move downstream.

Most academic research is in the first region and tends to be 5-10 years away, at least, from being part of some innovative product. And most research is intellectually exciting but of marginal commercial value.

Being able to predict which of the research technologies will win is impossible, and the surface area of opportunity is huge, and the investments that are required to turn those ideas into products is vast.

The biggest commercial wins of all time are when someone can take a research idea and turn it into a commercial product. Examples of such things include – server virtualization and network virtualization, the search of the web, cloud computing.

However, as a strategic software architect, the job isn’t to find that disruptive new technology, the position is to keep a sizeable successful software business successful and relevant.

Searching for that next disruptive idea is only part of the equation.

The equally important part is managing your investments in delivering commercially viable innovations, ensuring that the mainstream market is satisfied with the product and that you never fall behind the mainstream requirements and become a laggard – primarily avoid four at all costs, always be on 3, and do some of 2 while looking for 1.

A challenge is that although a strategic software architect may be thinking about all four regions at the same time, most teams are in one place and are focused on their backlog for their region.

If your goal as a software architect is to move a team downstream from being pure research or upstream from being a laggard, you will effectively deprioritize the obvious set of next work with something different. Something more risky because it’s not the obvious next thing.

And if you are a laggard, the product will be suffering from a lot of code rot problems. And just cleaning it up may feel like the top priority. 

And that would be the wrong thing. 

You are a laggard because you are missing features. 

If your code sucks, and your build times are horrible, etc., it’s tempting to think that you are a laggard because of that. In fact, that is irrelevant. You are a laggard because in the market you have a feature gap between the laggard and mainstream market products.

The real objective is to own most of the mainstream market and to offer innovative features that capture premium value while hoping that you discover some groundbreaking innovation.

And because you are a laggard, the market doesn’t look for you to innovate, they look for you to deliver the things that they want in the way that they want. The mainstream has defined its requirements, and you must fit into those requirements. A lot of times the temptation is to argue with the mainstream market, but that’s the wrong strategy. The mainstream isn’t interested in innovations, it has a particular point of view, and you need to satisfy it.

There may be innovative ways to deliver those solutions, but they have to meet the requirements of that market even if those requirements are – for lack of a better word – wrong.

What this means is that you have to add – sometimes – even more technical debt to get to the mainstream market so that you can have the revenue dollars and credibility to innovate.

In any new situation, the first job of a strategic software architect is to validate if the product meets the mainstream requirements and if not, all energy has to be focused on that.

Warning signs that you are not meeting mainstream requirements include declining market share, the field losing trust, emergence of direct competitors offering the same kind of product that is bought and deployed in the same way, decaying investment in the core product.

You have to direct your first set of investments and energy at getting out of the laggard space.

At NetApp when I joined the storage management team and was made their architect, this was precisely the situation we were in. The product, DataFabric Manager (aka DFM) had had marginal investment, and suddenly storage management had become a thing that customers cared about. Our field referred to DFM as Doesn’t F*king Matter and our customers were livid.

The first thing we had to do was identify the set of feature gaps that we needed to close to make the product not suck. And to be fair, the product didn’t suck given the investment. What had happened was that the market requirements had shifted on the team and the company had not made the investments to meet those requirements.

Once we identified those feature gaps, we had to code like mad. I remember sitting in a room with the head of engineering, and he said: Kostadis, I need you to disappear for the next few months. We need to code like mad. And I can’t have you distract anyone with your great next set of ideas.” The engineering head was dyslexic, and he never wrote anything. When he wanted to emphasize things he would take a pen and write it down, and he wrote “Code like mad!!” And he underlined it twice. I wish I still had that paper.

And so we coded like mad and got out of the laggard space.

I like to joke this is the easy part. The requirements are understood, the team top-to-bottom understands what it needs to build, the only problem is you have to go very very very fast and have to be very very very disciplined. The hard part is that a laggard product has a lot of internal challenges – a lack of resources, talent and brain drain, and a lot of internal corporate vultures. The internal corporate vultures are the worst. Instead of fixing the laggard product they decide that the right answer is to buy another product, or to do a rewrite or to fire everyone or to do anything but fix the feature gaps.

It’s mind-boggling. In fact, if you walk into a situation where everyone wants to re-write the core product, it probably is indicative of a laggard product.

And the right answer isn’t to do a rewrite but to fix the critical feature gaps.

So when we delivered that next DFM release, we didn’t advance the architecture, we didn’t solve the underlying technical problems, and we may have added some tech debt, but we got to the mainstream, and that set us up for trying to get to the innovative product space.

And we did.