Photo by Markus Winkler on Unsplash

Recently, I have been noodling about the qualitative difference between Ransomware and other forms of cyber attacks.

Ransomware is fundamentally a form of reversible sabotage. When someone encrypts your data, the data is there, it’s just destroyed. When you pay them, the data becomes undestroyed.

If you have a backup that has some of the data that is unencrypted and safe, then the question becomes:

_ How much data will I lose?
_ How much will that affect my business?
_ How long will it take to recover my business given the lost data?

Because the current active data is destroyed, the first step is to determine how much data you still have. To do that, you need to restore your data from a backup.

The problem with doing a restore is that the bad guys will have penetrated several of your systems, and so if you blindly restore a backup, the bad guys will destroy that copy.

So you need to restore the backup in a safe environment.

And so a critical question in any ransomware event is – “What is the last known good backup?”

And once you have that backup, the next question is – “How long will it take to rebuild the information that has been lost?”

Having, once again, entered the enterprise storage space, what I find interesting is how DR planning and preparation doesn’t include DR preparation around “recovery from backup.”

Not from yesterday’s backup but from a backup that is 30, 60, or even 90 days old.

If you routinely wargame that problem, you’ll uncover databases that are not backed up. Yes, I know a huge company that 6 years ago had super critical production databases in a DR configuration but no backup.

But more importantly, when a ransomware event occurs, you can quickly determine whether it’s worth paying the fee.

Testing backup recovery of old backups isn’t a nice-to-have in today’s world; it’s a necessity.

Or, as I like to say it, the IT teams that practice recovery from backup will be the only ones that are currently employed.

As to why I am so invested in this topic, at Zynga, two weeks before our IPO, a game of ours got destroyed because of an operator error.

Thankfully, we were able to recover from backup in less than 12 hours.

That experience taught me to think about backup differently.

And it’s why when I say Single Point of Failure, I think about recovery from backup not a server crashing.

When I talk to IT professionals and technical leaders, this need for backup to work is not understood.

Over the years of writing and architecting software systems, I saw the emergence of automatically generated layers of software that were nominally supposed to eliminate a class of software development or even software developers as an employment category.

For example, in the 1990s, higher-level programming languages and dynamic compilation of code were supposed to eliminate the need for low-level programming. I remember taking classes in the early 1990s where garbage collection and object-oriented programming would unburden software developers from resource management.

At the same time, we had the emergence of frameworks that promised further productivity gains, like the Distributed Computing Environment and CORBA for distributed systems. As for building Graphical User Interfaces, we had the proliferation of toolkits like OpenView, X-Motif, Borland’s Application Framework, Java Swing, Microsoft .NET Framework, etc.

Over the next 30 years, the story stayed more or less the same. Every year there would be some new technology that would, once again, unburden the software developer from toil.

And, of course, if you happen to be the person paying the software developer, make it possible to have fewer of those developers working for you.

I recall in 1994, a Professor at Brown University remarked to the students in computer science his concern that there would not be enough high-paying jobs in the field. The emergence of OO programming, and the ability to have people work remotely, would reduce the value of a degree in computer science over the long haul.

And he was right, but not quite in the way he anticipated.

All those technologies did reduce the need for highly skilled technology experts to build much of the world’s software. In 2003, I had to write my own time-series database for a product. In 2022, I would spend more time deciding which one to use.

The available software building blocks are more powerful, flexible, and compelling than at any point in the history of computer science.

And that has enabled more people to write software that I could not have conceived of in shocking time frames.

That point was brought home to me in 2009 when a small team of engineers at Zynga leveraging AWS, PhP, and a home-grown NoSQL database built a game on a distributed systems architecture with over 2000 servers. And none of them had a Computer Science degree.

And yet we still invest in software, building new products, and hiring new engineers.

Why?

Competition. It turns out that when everyone can make a scalable game without a CS degree, the magic is not in making the scalable game but in building a scalable game. And when there is a large pool of competing choices, and success only goes to the best game, marginal and incremental improvements result in huge wins.

This brings me to my somewhat non-obvious conclusion: software products’ success is a function of how easy it makes the lives of people who do something useful. The problem is that when anyone can do something, it has no commercial value.

For example, when everyone can build a scalable game, the value of a scalable one is zero. It’s like with special effects; when everyone can have great special effects, the value of special effects in a movie is zero.

So what did the framework and toolkits make easier? They made the expected stuff easier, so you can work on the element differentiating your product or game. You still need to invent a new game mechanic, and that new game mechanic, because it is new, requires new code. And, here’s where it gets fun; eventually, you find new game mechanics that require new hardware, which requires new frameworks, which require…

And if the demand for new games is significant, then the need for new game mechanics is large, and the demand for new software is powerful. We all wonder why we are not more productive while playing games that are more beautiful, engaging, and deeper than ever before.

We live in a world where the amount of software is growing. And more people are writing software than ever before.

And that’s a good thing. But the need for new innovative software remains. And that innovative software is not generated; it is created. And that new software must be created when the boundary between the digital world and the human or physical world changes.

In my post yesterday, I attempted a rebuttal of Mr. Jassey’s arguments about the superiority of in-place work.

My meta-argument was that a diverse workforce has a diverse set of needs. And that what works for some does not work for others.

A great meta-question is – so what? Does it matter to an organization if some folks are excluded, and some are not? Is cultural homogeneity an asset or a disadvantage?

When a company asserts specific lived experiences are valid and others are not, they explicitly articulate that specific human experiences suit their culture and others are wrong.

Those whose experiences fit into the wrong category self-select against that company. And over time, if it is successful, the company views its choices as correct because those who don’t fit in never show up.

They never bothered to come and talk to you because they saw that you thought they were terrible and that rejection of their existence meant there was no space for them there. But it’s just working from home? Right? Except it isn’t. Where does that end when you assert your lived experience is correct and the other person is wrong? Why should it stop at how we work together? Why not how we dress? How do we address each other?

“We chose to do it this way, and we are successful, therefore we were right.”

What was a choice, a preference, becomes because of the group’s success criteria for articulating what is generally correct and generally incorrect for all human beings.

And again, so what?

Diversity and diverse perspectives matter because otherwise, you suffer from groupthink. If everyone does it one way, and all successful people do it that way, and no one objects to how we are doing it, then it must be the right way to do things. And what happens, invariably, is that someone else figures out a better way to do it, and then the group collapses as the new way triumphs over the old course.

Several years ago, I wrote about Usain Bolt. Usain Bolt was too tall for a sprinter. That was the orthodoxy. But a coach decided to ignore orthodoxy, and guess what? Usain became the greatest sprinter of all time.

Or Ervin Johnson was too tall to be a point guard. But another coach saw his passing and skills and decided that the advantages he brought to the game because of his vision and height and reaching from the point position outweighed the benefits he got to his team as a front-court player.

History is littered with companies, teams, and organizations convinced they were doing things the right way. By asserting that their lived experience was correct over other people’s lived experiences, they drove talent away. That talent then thrived elsewhere.

One of the best program managers I have had the chance to work with observed that companies have challenges when transactional-based decision making and arrangement-based decisions come into conflict.

Transactional-based decision-making is what I call strict enumeration and documentation in the form of a contract. Once both sides agree on the document, then even if people change, the decision remains. The trust is in the record, and the change controls surrounding the document. This kind of decision depersonalizes the decision. These decisions tend to be transparent.

Arrangement-based decision-making relies on trust between the two parties. The parties establish trust in various ways, but typically it’s about making sure that the personal goals between both leaders align. Arrangements are remarkably durable and transportable. Meaning that once you establish an arrangement with someone, trust is the basis of subsequent decisions much faster. It also means that decisions without going through a complex process. The problem is that these decisions tend to be opaque.

Both models taken to extremes can be a disaster. I have worked in both. And they both suck.

The law is THE example of a transactional system. Overly transactional environments turn into bureaucratic, legalistic environments.

A pure transactional system can turn into a totalitarian state. A great example is – “the only way to change this spec is to call a meeting of the change control board and submit a request.”

A pure arrangement system can turn into a cult where belief in the great leader is paramount and the phrase “blah said” is used to justify or denounce everything. That disagreement with the great leader is an unforgivable sin. A great example of this is when someone says – “but SR ARCHITECT FOO said”. Or alternatively – an old-boys network – that is impenetrable.

I’ll also observe that both approaches feel natural to different people. My personal experience is being Greek and Canadian and living in the USA. As a Greek, I believe that laws are suggestions. That relationships, and in particular, family relationships, trump everything. As someone who lives in Sunnyvale, CA, I have learned that in the USA, laws matter, but that the laws are structured to satisfy arrangements among the wealthy.

A transactional system feels like a spectacular waste of time because personal relationships trump everything and that, ultimately, transactional systems are a facade for arrangements.

But I have learned that that is naive. The critical flaw of arrangements is the opacity of building trust and the boundaries of trust. Transactional-based decision-making creates a public record of the moment trust was built and describes the trust boundary. Without such a record, trust-building naturally devolves to family, culture, background, and other attributes.

This brings me to how I think about architecture reviews. The purpose of the review is to create a trust boundary between the approver and the author. The more high-level the spec, the more trust that has to exist. That trust is typically built over a series of smaller successes or previous professional successes. The more detailed the spec, the less trust that exists. A key trust moment as an architect is when you are willing to stop being the sole approver of an area.

The key illusion in all of this is that as an architect the spec actually defines what is built. The reality is that unless you are writing the software, some other human being is going to take that document and do what they think the right thing is. My job is to make sure that they are thinking about the problem in a way that aligns with a reasonable solution.

Decrying arrangements because they are old boys networks is wrong. Decrying transactional-based systems because Process is also wrong. Like most everything in life, there is a balance. And like most everything in life, navigating that balance is the art of living and being a software architect.

Over the last 15 years, I’ve been noodling about portability. And the more I noodle about portability, the more I think it’s very poorly defined.

So, let me try.

In the context of computers, there are two flavors of portability, data path portability, and control plane portability. The data path is the bits of code that do real work. The control path is the code that sets up the data path, monitors the data path, and takes corrective actions.

In that weird way that computers work, the control path itself is just another data path. And so – in a bizarre way – there is only data path portability. Simplistically, the control path is instructions that execute on processors, store data in memory and storage.

And although Paris Kanelakis would be pleased to know I learned something in CS051, things are not- quite that simple in the world of commerce that I live in.

Except when they are.

The VM encapsulates a traditional single VM applications control and data path. And the VM, as a result, is a highly portable abstraction because the hypervisor can ignore which instructions are from the control path and which data path; to it, they are all data path instructions. And we know that if you don’t care about performance, any data path can simulate any other data path.

So far, so good.

VMware demonstrated that the portability of a VM is very valuable because you can move it between generations of hardware, move it around to utilize the efficiency of your infrastructure better, and because it’s a convenient way to manage workloads.

And so, the VM abstraction and its portability had another interesting effect. You can hand over the infrastructure operation to another person.

But while I was at NetApp and Zynga, there were all sorts of other portability that people discussed. This data path portability was nice but not sufficient, and it wasn’t enough if you couldn’t encapsulate the entire app into a single VM.

So sure, I could move the VM around, but if I didn’t have the control path, what value was the VM?

And so there was this thought process that said, “VM’s are not that interesting.” are

And I looked into all sorts of interesting programming languages and tool chests.

And yet, along the way, something funny happened. If I cared about performance, I cared a lot about the hardware I was running on. And all of a sudden, the behavior of the hypervisor matter a lot.

At Zynga, when Amazon changed their NIC, our costs skyrocketed. Their decision to optimize their business almost tanked ours.

But performance, who cares about performance? Right? Except, here’s the rub, when you don’t have pricing power for a feature, and you sell the hardware that the feature runs on. The only way it can be worth implementing is to implement the feature without meaningfully increasing the hardware costs.

And then the performance of the feature matters a lot. And some features don’t get implemented because there isn’t an efficient way to do that.

But if you can’t charge more money, why implement the feature? Because – here’s the rub – your competition is adding features very fast. And if you don’t stay ahead, the whole SaaS model suggests that eventually, your customers will move. So as a vendor, you have to keep adding features while keeping the cost of the features in check.

Hoom. So – wait, if I want to grow my business and add more services, and can’t meaningfully increase the price, then, erm, the performance of the feature matters a lot.

And yes, you can make code more efficient and thoughtful, but at some point, you start caring about whether the disk is an SSD or whether the NIC has enough buffers, or what exactly is the CPU you are running on.

I mean, if people didn’t care about these things, those things wouldn’t exist.

Okay, so?

Well, once you start getting to the point that you care about performance that much, then the hypervisor details matter a lot. But surely you jest? Hypervisors make choices, and those choices impact the way your applications run. And if you design an application to run on a particular hypervisor, then the portability of your application is the portability of that hypervisor. If the hypervisor only runs on hardware X or Y, or Z, any performance improvement locks you into that vendor.

To put it differently, if the control path of the hardware is in the hypervisor, and you need to control the hardware, you need to control the hypervisor.

So what? It turns out that the hardware vendors are adding more hardware faster than that integrated solution vendor is sharing. And so, being able to have some degree of portability to run on other hardware will matter a lot.

Because why? Because the more it costs to run a feature, the less money you have and the more money the other person has.

Hoom.

Baremetal! BAREMETAL!

I hear you say. I was cheeky. See, every modern OS is a hypervisor of some kind. When you run on a public cloud, you can’t control the hypervisor.

And in some ways, not to be annoying, running your app in a VM on a cloud is like trying to tune your Java code – optimizations can be done, or you can use C or Rust or C++.

So when I hear bare metal, I don’t hear “no hypervisor”; I hear – – run a hypervisor of your choice on hardware.

BUT BAREMETAL IS HARD. I agree.

So?

Well, here’s where things get very interesting. Suppose you had a portable hypervisor that existed, and you didn’t have to do the hard part of managing the hardware but could control said hypervisor to your heart’s content?

Why then, for those applications where the data path was critical, you wouldn’t be tied to the cloud vendor’s hardware choices.

Data path portability is about being able to run your VM anywhere you want and be able to control the hypervisor.

AHA!

So for the essential thing that can move my costs, I need to move my VM to whatever hardware I want and tweak the hypervisor any way I want.

But then why has the public cloud won so much business?

Well, for the same reason, Java has won over C++.

It’s just got a better control plane for application development. Java makes so many things easier.

And really, that portable hypervisor was painful to use compared to the cloud.

So hypervisor portability was nice, but it was uninteresting if you couldn’t run the control plane. And you couldn’t because it took a lot of time and money to build, and the value of doing that work was marginal. Except when it wasn’t like for Dropbox. Or at Zynga, where we designed our control plane to be portable and had options when Amazon decided to optimize their business at our expense.

But the control planes aren’t tied to the hardware platforms as much.

What does that mean?

Well, it means that for some workloads where the costs matter, using the non-proprietary control planes so I can use the portable hypervisor may be a better choice.

And well, a portable control plane is – to a hypervisor – just another data path.

REPATRIATION!

No.

No software service that exists and increases in value hasn’t been optimized and improved over time. The original S3 relied on MySQL databases, for crying out loud, and Facebook used a single NetApp filer.

Thinking about this as “repatriation” is the wrong mental model. The suitable model is that critical services will get optimized, and at some point, the optimizations will care about the hardware. The existence of portable control planes and portable data planes will be fascinating.

But I have to build my data center!

Nope.

See, those portable hypervisors are pretty much available as a service now.

And that leads me to the final round of thinking.

When application control and data plane portability is possible, will the ability to cram a small number of hardware configurations in data centers be that valuable?