Photo by Biegun Wschodni on Unsplash

A friend of mine and a colleague sent me a note today that he feels that his job feels more like a shepherd’s than anything else. He now had to look at what people were doing and keep telling them “not good enough” with the hope that it gets good enough at some point.
This contrasted with the model in his head of the software architect as gatekeeper.

Having worked with him for many years and seeing his growth as a person and a software architect, he articulated what I have been noodling about for many years.

The traditional mental model of a software architect is what several open-source projects implement. Every change is ultimately approved by one or more contributors who judge every change for clarity and quality.

That model makes the architect the gatekeeper of every coding decision.

That model might work well, but it does not work for me.

My model is more of that of a shepherd. I define in my head what is unacceptable. Unacceptable is anything I don’t understand. And that is anything that will break the customer’s deployment of the current product.

A breaking change need not break the current deployment of the existing product. It is always possible to transition to a new version of a system. A truly breaking change is – in my mind – a different product.

What about things I don’t understand? Understanding doesn’t mean agreement. And most importantly, it doesn’t mean code. It means the person sharing the technical data has given me enough details that I am confident I can understand what is being proposed and what it might manifest as, and I am okay with that.

To borrow from Gandalf, until I have understanding and believe this won’t break customers, “You shall not pass.”

Then the next set of questions are about the confidence that what you are building is being built well. My basic strategy is to hire great engineers who build great software. I assume the great engineers are going to produce great code. The real tricky bit is to make sure that the engineers believe that what they are building is the right thing for the right reasons. Like all of us, it’s often easier to say “Yes” to the wrong answer than to communicate “No.” Not every battle is worth fighting over.

My view is that as a software architect, I am required to say, “No.” And that my job is to discover if the engineers think what they are building is correct. And if it is not, be the fulcrum and lever that they need to get the space and time to figure out the right answer.

The other point is that once we know the correct answer, we can then agree to do the wrong thing because of time-to-market reasons. And that is an explicit tradeoff and not a consequence of never even considering the correct answer. And the ability to choose to do the wrong thing allowed flexibility and enabled the right long-term thing to be done. Without that ability, I have seen standoffs between engineering management, product management, and architects. The architects don’t trust the managers and the PM to do the right thing, so they only propose the right long-term solution, pushing out timelines, etc.

Again it’s about shepherding people through a process, not telling them what to do, and not reviewing everything they produce.

Why am I okay with this?

Look, I have 10 fingers, and I do more check-ins than any person in the company. Because I have a team of hundreds that does hundreds of check-ins per day, and they can operate independently, looking at the correct data. By leveraging their collective intelligence, we can go faster and innovate faster than any other leadership model.

Is there a risk that things will go badly? Yes. But I prefer to deal with the failures and then understand how to mitigate them going forward. To innovate is to fail. To do new things is to fail. To encourage people is to fail. I would instead fall that way than any other way. Because when you innovate, do new things, and encourage individuals and teams, greatness happens.

[so I tweaked an earlier post to be more inclusive and more relevant]

VMware has – possibly – the coolest skunkworks system at any company. Skunkworks projects are shared at a three-day internal conference known as RADIO.

During the year, employees across the company work on projects and produce papers based on those projects whose only purpose is to share them at RADIO and possibly get funded later.

Andrew Lambeth is a Fellow and all-around amazing person who gave an excellent talk titled P-zero or die. The point of the talk was how to take any big idea that you had and get it funded.

What does P-zero mean? Well, the P means priority. Everything in a company has a priority to decide what the team works on first, and the number zero is the highest priority level.  Why zero? Because, well, it turns out, first you start with the most important things having priority 1. And then the teams realize that there are too many P1’s. So instead of moving everything down to P2, a new priority level is created, priority zero. And then, of course, there are too many priority zero’s, and someone says, “let’s create p -1”. Then everyone realizes how many scripts and tools will break and that this makes them look ridiculous. So the org finally does the hard work to figure out how to reprioritize. But why didn’t this happen when p-zero was created? Well, because computer scientists start counting at 0, not 1. So 0 is the lowest number, not 1.

The fundamental principle of the talk is that if you don’t make your big idea a p0, it will get deprioritized for other stuff. And the reason it got deprioritized is that it’s big, and its value proposition was unclear, and people didn’t understand what you were trying to accomplish.

And so here’s the checklist of things you need to do to make your big idea someone else’s p0.

1. 1. Describe it effectively in 5 minutes.
   2. Success is easy to measure.
   3. Listeners must understand, not agree.
   4. Have a document to share, not go over.
   5. Describe it on any media.
   6.  Pitch it at every opportunity, relentlessly.

And the most crucial thing is this:

7. If you get no traction, then move on to the next big idea.

Sometimes a big idea’s time has not come, and you just need to let it go.

I liked the talk so much that I decided to make a t-shirt.

So my dad is one of those legendary figures in modern science. His H-index is 70+. His research transformed what we think of as hospital medicine, from a series of narrowly scoped disciplines to a systems theory of the human body where the center was critical care of key systems. And along the way, he transformed medicine in at least two countries, Canada and Greece, with his tireless advocacy and work to improve the medical systems either directly or through the people and institutions he built.

Whenever I need humility, I talk to him.

I mean, I have done a lot of big things in my life, but no, I didn’t change the fundamental way we think about how we die (body first, brain last).

In oh-so-many meetings, I stand in front of a large number of engineers and product managers and engineering managers and say, “see this impossible hill; we will climb it.”

And there is a natural inclination, “how?” And “how fast?” And “we have so much insanely hard work to do!”

For most of my professional life, I view that as an affront. That the person asking the question was dismissing the solution because it was too hard. As if there was some easier path that I had deliberately chosen to ignore. I thought they were saying you came up with the wrong answer. And I would get angry, and pissed off, and frustrated.

Recently I took this EQi test. And a key element of that test was how much you used reality to make judgments. And, well, I scored poorly.

And that got me thinking, why? Because I tend to look at reality not as immutable but as mutable.

As someone who has seen how the future changed because of what individuals like my dad did, and more modestly I did, my opinion of the relentless forces of nature and history is that I will always be willing to use my living hand to challenge any invisible or dead hand.

So once again, I was in a meeting where someone said, “this is the moral equivalent of scaling the North Face in a snowstorm.”

And I was like – “Man, you and I are so different, in a world that is 70% water, I look at an empty glass and think how easy it is to fill it.”

And, of course, that went down poorly. Because that person felt I was dismissing their observation. And, to be honest, I was.

And so I told this to my dad. My dad goes, “blah. Filling a glass with water is easy. Go to a lake or the sea. Finding a glass is easy. Go to any store. But make the right kind of glass that fits in the right place. That’s what’s hard.”

And he then looked at me with that look, “Good you can do hard things, I am proud of you.” My dad was never one for recognition, etc. H was always about the next hill to climb. And then we spent the rest of the time talking about how filling that glass is so damned hard.

And it got me thinking a lot about how strategic software architecture is about finding that right glass. And that is hard. And when I find that glass, I am excited, and all I can see is how we could achieve miracles if we just filled it.

But the work has barely begun for everyone else.

And filling that glass with water is hard.

And being dismissive of the challenge and not recognizing that the work has barely begun is critical. And when people tell you, “dude, I am not climbing the North Face in a snowstorm unless you break this down a little bit more,” I shouldn’t get angry; I should be delighted. Any sane person would be like – “good luck.”

Next time, I’ll not stand waiting for the applause and get annoyed that everyone isn’t admiring my achievement of identifying the right glass.

So let me adjust my thinking.

When I find a glass, I know we can fill it with water. The effort to fill it with water in a timely fashion will make the finding of the glass feel like a trivial subtask.

And to ask people to be happy for me for finding the glass is kind of like the tenor asking for applause for clearing his voice. I’m asking them to trust me that I know where the safe path up the North Face is. Maybe, instead of asking them to applaud, I should start preparing for the climb and be grateful that they might follow me.

One of the most problematic practices in engineering organizations that I have joined is the rejection of complex solutions to complex problems because they are difficult.

Any problem at a large scale of customers is complex. The software has to deal with a comprehensive and unknowable set of use-cases and deployments. The software has to cater to a large number of other systems and users that interact with it.

Any change to the system can have unknowable impacts.

Suppose we have a piece of software, let’s say Acme Management Center (AM Center), and we need to make a significant architectural shift.

For example, if you need to change a system from:

1. being single-threaded to multi-threaded;
2. being a single point of failure to tolerate multiple-failures;
3. using a single database, and using multiple federated databases;
4. being a closed a system being open;

the size of the effort depends on the complexity of the software being change.

If AMC has a small number of users or is small, the cost of the change is tiny.

If the system, let’s call it BigVirtuCo Center (BVCC), has many users and has been around for 20 years, the effort to make the identical architectural change is enormous.

We intuitively understand the complexity of the two efforts as being different.

But then let’s turn it around a little bit. Why was the change of BVC Center’s architecture so much harder than AM Center? It’s because BVC Center supports 20 years’ worth of unique features more than AM Center.

Now BVC Center is worth 10 billion dollars because of how many people use it. AM Center is worth 1 million dollars because of the number of people who use it.

AM Center is a delicate product, but it simply doesn’t have the feature set that BVC Center has and thus can’t solve the problem that BVC Center solves.

A typical fallacy that the BVC Center competitors engage in is to say, “well, we can make something that has 80% of the value of BVC Center and kill it in 1/10 the time by focusing on 20% of the features”.

They are saying that there is a more straightforward solution to the complex problem that BVC Center solved.

And the competitors of BVC Center typically fail.

Oh, but what about the innovator’s dilemma?

Simpler products can indeed displace BVC Center. But they don’t replace BVC Center when they are simple; what happens is that they find a big complex problem that BVC Center ignored and become entrenched in that space.

BVC Center doesn’t get displaced by a simpler product; it gets displaced by a complex product solving a complex problem of higher value than the one BVC Center solved. Most of the time, the thing replacing BVC Center is more complicated than BVC Center ever was.

The problem with complex problems is that they take a lot of time and resources to solve.

As a software architect, it’s essential to understand what the correct solution to the problem is. And to not shy away from the complexity.

At the same time, we need to ship software. And this brings us to the tricky problem:  not the discovery of the correct solution but the delineation of an approximation that adds value and moves you in the direction of solving the problem.

For example, making a program multi-threaded that had a single thread is a massive undertaking. Incrementally adding threads and parallelizing pieces of the system allows you to take advantage of more processing power.

Similarly, a customer may have a complex problem that will take years to solve. The customer is willing to live with a partial solution if that leads to a complete solution.

To revisit the beginning of this thread, frequently, folks say – “this is too complex.” And perhaps it is. Or perhaps what they are saying is that the cost of the solution is too high.

As an engineer, it’s essential to understand what the correct answer is, and at the same time, what is the slice of the right solution that we need to do today and never to confuse the cut as the correct answer.

In the tech industry, there are as many ways to buy products as there are companies.

Today there are two big variants. The first variant is what we call “Packaged Software,” and the other variant is what we call “SaaS.”

And the corporate transition from selling one variant to the other can feel like the ocean hitting land. And the transition is made worse because organizations insist that the problem is culture and mindset and not how the software is built. No amount of mindset can overcome software architecture. And because software architecture is hard, and because the two teams are screaming at each other, the real problems never get addressed.

But, as I stared at the problem recently, I realized that we architects must provide meaningful guidance. Saying “but the architecture” is as useful as “but culture.”

So let me try.

Packaged Software is when you go to a store, buy the bits, and install them.

The other variant is SaaS.

Now the word SaaS translates to “Software as a Service,” but like any FLA (Four letter acronym), it’s so devoid of specificity to mean nothing.

I claimed that you were a SaaS service many years ago, not if you run in the cloud or on-prem but if you satisfied one criterion.

1. The provider of software decided when software gets upgraded

This definition was interesting, but big enough customers can demand special upgrade cycles even for SaaS.

So I started adding all sorts of other criteria, “easy to use, consumable, freemium, cloud delivery, DevOps,” and every single time, it floundered because I could trivially point to how packaged software offered the same features.

But SaaS was different. And then it hit me; SaaS is different because of another important criterion.

2. If someone other than the provider wants to extend the software what’s important is how to connect, not where the Software is.

The significance of this – is that the software consumer assumes the software is always running and that the consumer and integrator doesn’t have to care about how it’s deployed, how many resources the SaaS system requires, etc.

Integration doesn’t involve step 0; install the software.

Elements 1 and 2 starts to separate SaaS and non-SaaS.

But there is in my mind this third criterion:

3. The producer of the SaaS software chooses where it can run, not the consumer.

Now, this is a weird one. Almost every piece of software written has some minimal hardware and/or software requirements. So, at some level, why is this different from non-SaaS? The answer is that in non-SaaS, the actual location of where the software runs is opaque to the producer.

So now we have – in my mind – a minimal test for a team that wants to be SaaS.

1. Does the team owning the software decide when it gets upgraded and consequently what features the software has?
2. Does the team/customers that use the software just care about how to connect to the software?
3. Does the team that owns the software get to decide where it physically runs?

These are necessary conditions, although not sufficient.

But they are necessary for service ownership to become a thing.

Service Ownership?

In any SaaS company, service ownership means you have a pager (or someone/somewhere on your team) has a pager.

When a company first goes to SaaS, they typically install much software from a bunch of teams into some cloud and offer that as a cloud offering.

The team that installed the software then finds itself triaging bugs in all of the software installed.

The team that did the installation then screams at the other teams: “You need a culture change. You don’t care about customers!”

The teams that produced the software look at the cloud team and said: “How dare you! We respond to every bug!”

The cloud team says: “We are a service.”

The other team says: “we produce software.”

And the screaming continues.

The truth is the following. The cloud team built a service out of packaged software. They control the version; they control how you connect to the service; they control where the software runs.

From the other team’s perspective, the cloud team is just another unhappy customer, running the wrong version of the software, has done all sorts of weird things that don’t quite match other customer usage patterns, etc.

And like every other customer, they just want people. And the product team has mastered the fine art of saying no.

But why does it feel to the product team that they are just another customer? Because the product team doesn’t have any control over when a bug gets fixed and deployed, the cloud team does. Because the product team doesn’t control how you connect to the software, the cloud team does. And finally, because the product team doesn’t even have any say in where the software runs, the cloud team does.

And the SaaS standoff occurs.

The SaaS standoff ends when the cloud team demands a specific service, not support, not cultural shifts, but a service. And that service has to be well-architected and engineered to provide both teams with what they want.

And the minimal requirements of that service are the three things I enumerated.

1. Does the team owning the software decide when it gets upgraded and consequently what features the software has?
2. Does the team/customers that use the software just care about how to connect to the software?
3. Does the team that owns the software get to decide where it physically runs?

About 15 years ago, I met an Inktomi architect who said that the future of all computing was the cell phone. It was before the iPhone.

And I stared at him and his dorky phone, and I said to myself, “nope.”

For 15 years, that was THE worst bet of all time.

But COVID changed all of that.

Mobility ground to a halt. And more importantly, the emergence of a whole slew of technologies called into question the value of physical mobility.

And then got me thinking about the future of mobile computing.

Mobile computing intends to allow the person to go anywhere with their computing device.

That was the paradigm.

But what if the paradigm shifts a little bit, that mobile computing is about letting the person be present anywhere without physically being there.

I like to follow mobile apps as proof points of a thesis.

For example, before Covid, Doordash’s desktop features lagged their mobile devices. It has since changed.

The simplistic answer is that people are at home more and in front of a computer more.

But that’s not satisfying.

What if, instead, more people had more computers? And had created places to work at home that allowed them to use their computers more?

In that case, more people would stop carrying their phones with them and use them to do work stuff. Instead, they would go to their workspace and do their work stuff there.

So?

In that case, technologies that improve our ability to be present are more important than technologies that allow us to carry our computers with us.

And that mobile computing and battery life, the dominant trend and a perceived inevitable future, was just technology that accommodated the previous paradigm, well.

My point in this discussion wasn’t to focus on COVID. My point was to focus on how paradigm shifts can occur.

And that assumptions about the inevitability of anything are usually wrong.

Tying this back to strategic architecture, this falls into my general thesis that the role of a technologist who owns a portfolio of technologies is to recognize that things fall out of fashion. That when they do, it is tempting to call them dead. It is also tempting to ignore things that are in fashion and call them a fad. Instead, we need to keep our eye out for trends that change and then think through all of the implications.

And that if you are leading a broad enough portfolio of products, the strategy of how to navigate that shift is very important.