wrong tool

 

In the ongoing debates over SaaS and SaaS transformation is the belief that software engineers must do dev-ops to acquire empathy.

It sounds so good in principle. The problem is that the engineers don’t understand the problem because they don’t care!  So let’s make them do the job so that they can understand the problem and care.

No one ever asks – what if the software engineer sees the whole experience as a tax on his life that adds very little value. That their brain doesn’t work in a way where doing is the best form of learning. And why wouldn’t a well-crafted document suffice? What if the engineer is the kind of person that doesn’t function well in a high-stakes operations crisis but can deliver complex systems?

And then there is the emotional blackmail of the use of the word empathy. It’s not that  I don’t understand the problem because you haven’t explained it to me. It’s because I am callous.

My other favorite is that you must feel the pain so that you do what is necessary.

In short, the argument sounds like this to me:

• No pain, no gain.
• To grow, you must suffer.

It’s like a certain superhero’s origin story.

You don’t use your superpowers (in this case, writing code) because you don’t understand the good you could do (by doing more operational software).

I preferred the original – “With great power comes great responsibility.”

Enough.

When software lacks features, it’s because the engineers weren’t given prioritized requirements; it’s not because they lacked empathy.

I feel like the last 20+ years of interaction design have passed the entire world by.

And having owned operations at scale and done product development at scale, I feel like I am uniquely qualified to say “No” when people tell me it’s about empathy.

And as someone who has struggled with empathy because of how my brain works, the statement – “you lack empathy, and that is why you can’t build the right software” – when I have – is plain infuriating.

About 20+ years ago, the science of human-computer interaction emerged. And the idea was that software could be better if, instead of forcing human beings to change, the software met us halfway.

A book called About Face articulated that well. I still remember the aha.

About Face argued that software engineers did a terrible job of building software for their customers because they didn’t understand them.

And that there was a systematic approach to understanding the customers that could allow software engineers to build the right software.

That methodology became the art of software design.

And it has produced a much better user experience for software than ever existed.

Operations of software in a SaaS environment has a new persona, the operator. And the operator is not the software engineer.

And having the software engineer build UXs for the operator without the same kind of design discipline that we have for other products produces user-experiences from the 1990s.

In fact, the whole empathy argument is in many ways a rejection of over 20+ years of insight into the process of software design.

What the process did was to translate the user goals and aspirations into something that software could solve. And when you combine brilliant designers and brilliant software engineers, great things happen.

So the next time you are told you need empathy, inform the person screaming about empathy that moral arguments didn’t produce great software design. Great designers produced great software designs. And have the design team look at the problem with you.

If doing the job is a great way for you to learn, then do the job. If reading a document is a great way for you to learn, do that. If listening to a talk is a great way for you to learn, do that.

But learning what to build doesn’t require empathy; it requires a definition of what needs to be built.

 

 

 

Over the years, I have struggled with multi-tenancy. The definition of it feels fluid and imprecise.

And so, in a desperate attempt, I wrote some incoherent thoughts down. I’ll take another stab at being precise.

Multi-tenancy has two elements; the first is that there are a set of users who have access to a set of resources. And the users and their rights to those resources are a technical problem. What is the hierarchy of users? What is the hierarchy of permissions? What rights do they have on the resources?

There is a separate problem about the unit of isolation of resources. If two tenants have access to resources, the immediate follow-up question is how much isolation does the resource provide between tenants?

For example, a physical server is a unit of isolation. A virtual machine is a unit of isolation. And so on.

Let’s dig into the notion of isolation a bit. Isolation is used to provide security, performance, and availability isolation. A great example is a black site. A black site is completely isolated for reasons of security from the rest of the world. A dedicated server running a single application is used to isolate that application from any other application that may affect it.

At one extreme level of isolation, each computer system is isolated from another.

There is no multi-tenancy at all.

This model is costly.

And so consumers ask – can I trade off some security, performance, and availability for cost optimization?

In short, can I group a set of applications on a set of resources where the isolation isn’t as strong to get some sharing?

Time-sharing systems were, in many ways, the first such system when they replaced batch systems.

And that’s where things got confusing.

Because the only real isolation is hardware isolation, the minute the hardware is shared, two applications share something.

And although hardware vendors and software vendors have gone to heroic lengths to ensure isolation, the reality is if you are sharing something, you are not isolated.

Now let’s introduce a second person, the hardware resource provider. The hardware resource provider wants to offer resources of varying degrees of isolation. And wants to bill accordingly. The resource provider’s challenge is that the consumer may not be willing to pay for resources if they are too expensive. In other words, the consumer of the resources is willing to trade off some isolation for cost savings.

And now we introduce a third player, the manager of the resources that the consumer of resources uses.

The resource provider gives up some resources, the manager divides them up amongst consumers, and the consumers consume them.

The resource provider provides some resources that are isolated. The manager associates those resources into something that consumers can use, and the consumers consume them.

This simple model assumes that a resource that is isolated can be controlled and offered to a consumer and is at the right granularity.

But what if it isn’t?

And thus, we have the fourth player in our dance, a person who takes isolated resources from resource providers that they then offer to a manager of resources that offers them to consumers. Let’s refer to these resources as “virtual physical resources” (VRP).

And that’s fine, except in one very annoying use-case.

What if two different sets of virtual resources are sharing the same underlying physical resources?

If the VRP’s assume that is not the case, then the entire system falls apart.

Why? Because the guarantees of isolation assume that the underlying hardware has some degree of isolation (remember, the VRP is built on isolated resources).

Huh?

Imagine VRP-1 assumes that a socket is isolated and it controls the resource. VRP-1 can make assumptions about security and performance. If VRP-2 is also using that resource, VRP-1’s is being lied to. And that’s fine until VRP-1 and VRP-2 have conflicting objectives.

At that point, the thing that actually controls the socket has to arbitrate between VRP-1 and VRP-2.

And we’re still good. What if VRP-1 assumed it had control of the socket and gave out virtual sockets, and VRP-2 assumed that it had control of the socket and was giving out a full-time slice of the socket.

The problem is that the isolation unit was a socket, and there was no way to arbitrate between the two VRP’s.

So?

The real challenge in multi-tenant systems is the proliferation of VRP’s. Some VRP’s can not be layered on top of each other because they rely on isolation primitives at a lower level that loses critical information about how the VRP wants the resource to be used.

An obvious solution is to create a new VRP, called VRP-12, the union of VRP-1 and VRP-2 and can effectively share a system between the two users. I worked on such a system at SGI. It was a batch scheduler called Miser.

The problem with that approach is that as the VRP’s proliferate, there is a cost in pushing them down a layer.

What’s the cost?

There are three costs. The first is the system resources for the control plane itself. The second is the complexity of using the control plane. And the third is that as the control plane becomes more complex and supports more resource isolation, the control plane needs isolation.

At some point, it becomes simpler to have different VRP’s operating independently rather than on a single shared system.

But wait, there’s more.

Because a VRP only requires a layer that isolates hardware, and every VRP isolates hardware, VRP’s proliferate and get layered on top of each other. There is no final VRP layer.

So?

What it boils down to, in my mind, that multi-tenancy really means three things:

1. A set of physical resources that need to be shared
2. Some VRP system being used to share the physical resources (1) that uses mechanisms to isolate the hardware and provides isolated resources coupled with a mechanism to assign those resources to an individual group that can use them.
3. Some mechanism for grouping users into hierarchies’.

And what makes multi-tenancy even more complicated is that in any system, there are multiple VRPs.

And so the challenge of multi-tenancy is really about how you enable this ever-expanding hierarchy of VRPs such that they don’t require their own isolated hardware, and each consumer, resource provider, and resource manager can effectively do their job. And understand what guarantees they are getting and providing.

To make this a little bit more concrete, let’s consider k8s.

Some developer wants to deploy a k8s application and request a certain amount of CPU cycles.

To do that, there must be physical CPUs.

Those physical resources are partitioned by some hypervisor that hands out virtual CPUs to an OS.

The OS, in turn, hands out threads that run on the vCPUs.

The k8s system in turns groups threads into things called Pods.

The k8s system is deployed in such a way that each tenant has his own namespace.

Each layer, the hypervisor, the OS, and k8s is providing virtual resource pools. Each layer has a distinct manager.

A multi-tenant system is the combination of those virtual resource pools and the physical hardware resources.

 

 

My father is Charalampos Roussos. His H-Index is 76. He was a key player in creating modern hospital medicine. He did that as a side effect of his research on lung behavior that among other things determined that a human being’s will to live ends after the body dies, not the other way around. While still at the top of his game, he went to Greece in 1986.

And if you read this thread, https://twitter.com/greekanalyst/status/1367818017581768704?s=21, you will begin to understands the challenges associated with his decision to return to Greece.

And my dad saw them all. And yet, he managed to create. Why? He believed that the only way to resist the forces trying to destroy was by choosing to do high-quality work at all times.

His point-of-view was summarized with a saying, the title of my blog, that translates in intent if not precisely “the resistance of high-quality work.”

He is writing his memoirs. And he wrote a longer version. And I want to share it because it summarizes my view on so many different things.

Όσο δύσκολη και να είναι οι καιροί,
όσο ελλιπείς και αν είναι οι υλικοί και ανθρώπινοι πόροι,
όσο σοβαρή και αν είναι η αρρώστια,
όσο και αν ενδημεί η ζήλεια, η συκοφαντία και ο φθόνος,
πάντα οι ελεύθεροι και οι δημιουργοί μπορούν:
να αναπτύσσουν το χώρο της ευθύνης τους,
να προβάλουν αντίσταση με την ποιότητα της εργασίας τους,
να πιστεύουν στην αριστεία,
μια αρετή που δεν χαρίζεται,
μια αρετή που μόνο κατακτάται.

no matter how scarce material and human resources are,
no matter how serious the illness,
as much as jealousy, slander and envy are endemic,
freelancers and creators can always:
develop their area of responsibility,
resist with the quality of their work,
believe in excellence,
a virtue that is not bestowed,
a virtue that is only conquered.

 

 

When I started my career in software engineering, I didn’t know how to interview. But I did know that the basic paradigm was engineered to keep people like me out.

Let me explain. I have ADHD. And I can’t read quickly. And I was taught at a young age that speed measured fluency in a task, not the ability to perform it automatically. And I had marginal dyslexia, so my handwriting was poor.

Put me on a whiteboard, and ask me to write code, and it looked messy. I would forget semi-colons. Error conditions would be skipped over. Subtle details about the problem would be missed.

In short, the test would be an excellent evaluation of my ADHD/Dyslexia and a poor evaluation of my ability to be a software engineer.

SGI offered a program on how to interview, and so I decided to take it.

And the key insight the person who was doing the class had was that past success was a measure of future success.

And so I changed my strategy. I stopped asking stupid technical puzzles and instead asked them about what they did.

The problem with that strategy is that what you did is a good predictor of what you will do in a similar circumstance, but will it predict what you will do in a new environment on a new problem?

Also, it results in overemphasis on skills rather than aptitude. And the specific skills you may need change over time. And I saw this play out at a former employer where we switched to mobile, and the needs of backend software dropped. Engineers quit or were asked to leave because they lacked the skills and either lacked the interest or aptitude to acquire new ones.

At Zynga, we had this level that was just above MTS, called Principal Engineer. I discovered that a lot of those engineers who were hired into those roles flamed out. After I dug into it, I discovered that we had created a role for people who had mastered some technology. When we hired them at Zynga, they had to master a new thing, and not all of them succeeded at that. I will observe a larger part of it that we didn’t set them up for success, but it was interesting.

Especially because the level just above Principal Engineer was Architect, an architect demonstrated that they had mastered multiple technologies.

Architects almost always succeeded at Zynga.

This then leads to my next observation, that a good predictor of a senior engineer is someone who has mastered multiple different technologies and made an impact on them in multiple environments.

And then I went to VMware and realized that that wasn’t a perfect definition either.

Some companies need to have world-class experts in a specific technology. For example, VMware has world-class experts in compute virtualization. Those individuals are undoubtedly senior engineers who have mastery and depth that is astonishing.

And then, I learned about how some people don’t get an opportunity because of their gender, orientation, and skin color. And that reminded me of the story of Devil and the King. The King asked – “Who was the General that could have saved us?” and the Devil pointed to the King’s slave, “he was.” And the King said indignantly- “but he’s a slave!” And the Devil said, “yep, and you lost the war and your country because that is all you saw him as. Funny how that works.”

Finding a senior engineer is about finding a needle in a haystack. You have to look hard, and wide and deep and in places, you wouldn’t. You have to look at objective criteria. And you have to keep your ears and eyes open to see people you don’t normally see.

So? So, hiring a senior engineer is a multi-faceted process.  It would help if you looked at a bunch of criteria, your team composition, the problem you are solving for in their hiring, and in the end, make a judgment call.

Anyone who says that they can evaluate a senior engineer by themselves in 45 minutes is – I hate to admit this about myself and people I have judged – lying to themselves.

Over the last year, I have been thinking a lot about the nature of software and software systems requirements. When I started my career, I was at the tail end of the era that only cared about performance. The fundamental thesis of that era was that CPU cycles were not plentiful and that writing efficient code was critical.

It was 1996 when I left Brown University. At that time, the forward-thinking software engineers advocated for higher-level programming languages to alleviate some of the tedium of writing assembly.

Getting a computer to work, though, was more important than it working for very long. Having a machine that did a computation was the hard problem. As we got better at building computers, a lot of energy was expended to make those computers more robust.

It’s important to realize, in 1996, writing efficient code on computer systems that were assumed to work was the only mental model and constraint you had when designing system software.

In 2004, when Steve Jobs introduced the iMac with colors, the industry was aghast. The design of the user experience was no longer an afterthought but the centerpiece of software. I remember doing my master’s degree at Stanford and wondering why human-computer interaction was a thing. After 2004, we all had to get religion very quickly.

In 2009, the next profoundly disruptive requirement was availability delivered through clustering. Before 2009, the mental model for availability was to buy bigger and bigger machines that were robust and reliable. But AWS happened. And although the debates about clustering had indeed begun in the late 90s, they took center stage when AWS released infrastructure where availability was very poor and required native clustering to deliver uptime.

At about the same time as AWS in the cloud happened, the dev-ops movement emerged. Rather than fixate on what dev-ops is and is not, let us look at what it was a response to. In the 1990s, when you wrote software, the expectation was that you would buy new hardware when the new version came out. Upgrades happened when there were critical bugs.

What AWS enabled was a completely novel form of software delivery. Instead of delivering software with hardware, it became possible to only deliver the software. In short, the software went about from being something that you bought once to a machine that continuously was being maintained. This need for continuous maintenance created a new set of requirements.to deliver software continuously

In the early 90s, the serviceability of the hardware was not an important consideration of hardware design. As customers bought more and more computers, and the time to repair each computer took more and more time, the ability to service the computer became more and more important. So much like hardware in the 90s suddenly started caring about how fast and how easy it was to replace the disk drive, software engineers had to start caring about how easy it was to operate the software. Why? Because it was expected that the software would always have to go through some regular maintenance.

Let me try this differently. In the early 90s, when hardware failed, the computer went down. Through a series of hardware innovations, it became possible to replace most hardware components when they failed if you are willing to pay the money. In 2009 we adopted software architectures that enabled a similar kind of upgrade of components and replacement components while the system was continuously running. Once that became possible, it became possible to add new value to the software continuously. In short, the software became just like the hardware, something whose serviceability became critical.

To give a flavor of this, in 1996, SGI released the Octane. The Octane was a fabulous system. But serviceability was never considered. You couldn’t rack it. It generated too much heat for an office.  And so, it wasn’t that useful if you needed to buy 50 for a rendering farm.

In 2013 Edward Snowden happened. After he betrayed his country and fled to Russia, information security which had been a sideshow, became front and center. The difference in how computer system architects considered security before 2013 and after 2013 is night and day.

The next trend to emerge in software at large-scale companies will be the discipline of strategic system software. Most software companies have exactly one product. Large companies have many different products that must all work together and be independent. Software architecture has historically only considered how you architect a single software system with a single market in a single business. Strategic system software architecture considers how a collection of products, each with their own markets and business, fundamentally integrate to deliver more business value than the individual components by themselves without turning them into a single product.

Because of this discipline’s boutique nature: you have to work at a large company, be fairly senior, and the executive team has to see the value of that; there are very few practitioners of this art.

But as the arrow of time moves forward, I expect more of us to emerge.

Now those of you wondering what’s next, I believe it is the intersection of the law and of software. As more and more software gets embedded more and more deeply into people’s lives, the need to understand the law, the law’s intent, and technology will be a new architectural discipline that software systems will need to incorporate. To date, software systems and their architects view regulatory constraints as an inconvenience, not a necessity. Software system architects, who wish to define a new era, are advised to get a legal degree and understand the complex regulatory legal and its technical nexus that will drive the next important technical innovations.

When I started my career, system architecture was essentially about making the data path go fast. 20+ years later, it’s much, much, much, much more than that. And the experience of these last 20 years is that the set of constraints will only increase over time.

What does this mean? I don’t know.