Photo by Markus Winkler on Unsplash

Recently, I have been noodling about the qualitative difference between Ransomware and other forms of cyber attacks.

Ransomware is fundamentally a form of reversible sabotage. When someone encrypts your data, the data is there, it’s just destroyed. When you pay them, the data becomes undestroyed.

If you have a backup that has some of the data that is unencrypted and safe, then the question becomes:

_ How much data will I lose?
_ How much will that affect my business?
_ How long will it take to recover my business given the lost data?

Because the current active data is destroyed, the first step is to determine how much data you still have. To do that, you need to restore your data from a backup.

The problem with doing a restore is that the bad guys will have penetrated several of your systems, and so if you blindly restore a backup, the bad guys will destroy that copy.

So you need to restore the backup in a safe environment.

And so a critical question in any ransomware event is – “What is the last known good backup?”

And once you have that backup, the next question is – “How long will it take to rebuild the information that has been lost?”

Having, once again, entered the enterprise storage space, what I find interesting is how DR planning and preparation doesn’t include DR preparation around “recovery from backup.”

Not from yesterday’s backup but from a backup that is 30, 60, or even 90 days old.

If you routinely wargame that problem, you’ll uncover databases that are not backed up. Yes, I know a huge company that 6 years ago had super critical production databases in a DR configuration but no backup.

But more importantly, when a ransomware event occurs, you can quickly determine whether it’s worth paying the fee.

Testing backup recovery of old backups isn’t a nice-to-have in today’s world; it’s a necessity.

Or, as I like to say it, the IT teams that practice recovery from backup will be the only ones that are currently employed.

As to why I am so invested in this topic, at Zynga, two weeks before our IPO, a game of ours got destroyed because of an operator error.

Thankfully, we were able to recover from backup in less than 12 hours.

That experience taught me to think about backup differently.

And it’s why when I say Single Point of Failure, I think about recovery from backup not a server crashing.

When I talk to IT professionals and technical leaders, this need for backup to work is not understood.

Over the years of writing and architecting software systems, I saw the emergence of automatically generated layers of software that were nominally supposed to eliminate a class of software development or even software developers as an employment category.

For example, in the 1990s, higher-level programming languages and dynamic compilation of code were supposed to eliminate the need for low-level programming. I remember taking classes in the early 1990s where garbage collection and object-oriented programming would unburden software developers from resource management.

At the same time, we had the emergence of frameworks that promised further productivity gains, like the Distributed Computing Environment and CORBA for distributed systems. As for building Graphical User Interfaces, we had the proliferation of toolkits like OpenView, X-Motif, Borland’s Application Framework, Java Swing, Microsoft .NET Framework, etc.

Over the next 30 years, the story stayed more or less the same. Every year there would be some new technology that would, once again, unburden the software developer from toil.

And, of course, if you happen to be the person paying the software developer, make it possible to have fewer of those developers working for you.

I recall in 1994, a Professor at Brown University remarked to the students in computer science his concern that there would not be enough high-paying jobs in the field. The emergence of OO programming, and the ability to have people work remotely, would reduce the value of a degree in computer science over the long haul.

And he was right, but not quite in the way he anticipated.

All those technologies did reduce the need for highly skilled technology experts to build much of the world’s software. In 2003, I had to write my own time-series database for a product. In 2022, I would spend more time deciding which one to use.

The available software building blocks are more powerful, flexible, and compelling than at any point in the history of computer science.

And that has enabled more people to write software that I could not have conceived of in shocking time frames.

That point was brought home to me in 2009 when a small team of engineers at Zynga leveraging AWS, PhP, and a home-grown NoSQL database built a game on a distributed systems architecture with over 2000 servers. And none of them had a Computer Science degree.

And yet we still invest in software, building new products, and hiring new engineers.

Why?

Competition. It turns out that when everyone can make a scalable game without a CS degree, the magic is not in making the scalable game but in building a scalable game. And when there is a large pool of competing choices, and success only goes to the best game, marginal and incremental improvements result in huge wins.

This brings me to my somewhat non-obvious conclusion: software products’ success is a function of how easy it makes the lives of people who do something useful. The problem is that when anyone can do something, it has no commercial value.

For example, when everyone can build a scalable game, the value of a scalable one is zero. It’s like with special effects; when everyone can have great special effects, the value of special effects in a movie is zero.

So what did the framework and toolkits make easier? They made the expected stuff easier, so you can work on the element differentiating your product or game. You still need to invent a new game mechanic, and that new game mechanic, because it is new, requires new code. And, here’s where it gets fun; eventually, you find new game mechanics that require new hardware, which requires new frameworks, which require…

And if the demand for new games is significant, then the need for new game mechanics is large, and the demand for new software is powerful. We all wonder why we are not more productive while playing games that are more beautiful, engaging, and deeper than ever before.

We live in a world where the amount of software is growing. And more people are writing software than ever before.

And that’s a good thing. But the need for new innovative software remains. And that innovative software is not generated; it is created. And that new software must be created when the boundary between the digital world and the human or physical world changes.

In my post yesterday, I attempted a rebuttal of Mr. Jassey’s arguments about the superiority of in-place work.

My meta-argument was that a diverse workforce has a diverse set of needs. And that what works for some does not work for others.

A great meta-question is – so what? Does it matter to an organization if some folks are excluded, and some are not? Is cultural homogeneity an asset or a disadvantage?

When a company asserts specific lived experiences are valid and others are not, they explicitly articulate that specific human experiences suit their culture and others are wrong.

Those whose experiences fit into the wrong category self-select against that company. And over time, if it is successful, the company views its choices as correct because those who don’t fit in never show up.

They never bothered to come and talk to you because they saw that you thought they were terrible and that rejection of their existence meant there was no space for them there. But it’s just working from home? Right? Except it isn’t. Where does that end when you assert your lived experience is correct and the other person is wrong? Why should it stop at how we work together? Why not how we dress? How do we address each other?

“We chose to do it this way, and we are successful, therefore we were right.”

What was a choice, a preference, becomes because of the group’s success criteria for articulating what is generally correct and generally incorrect for all human beings.

And again, so what?

Diversity and diverse perspectives matter because otherwise, you suffer from groupthink. If everyone does it one way, and all successful people do it that way, and no one objects to how we are doing it, then it must be the right way to do things. And what happens, invariably, is that someone else figures out a better way to do it, and then the group collapses as the new way triumphs over the old course.

Several years ago, I wrote about Usain Bolt. Usain Bolt was too tall for a sprinter. That was the orthodoxy. But a coach decided to ignore orthodoxy, and guess what? Usain became the greatest sprinter of all time.

Or Ervin Johnson was too tall to be a point guard. But another coach saw his passing and skills and decided that the advantages he brought to the game because of his vision and height and reaching from the point position outweighed the benefits he got to his team as a front-court player.

History is littered with companies, teams, and organizations convinced they were doing things the right way. By asserting that their lived experience was correct over other people’s lived experiences, they drove talent away. That talent then thrived elsewhere.

One of the best program managers I have had the chance to work with observed that companies have challenges when transactional-based decision making and arrangement-based decisions come into conflict.

Transactional-based decision-making is what I call strict enumeration and documentation in the form of a contract. Once both sides agree on the document, then even if people change, the decision remains. The trust is in the record, and the change controls surrounding the document. This kind of decision depersonalizes the decision. These decisions tend to be transparent.

Arrangement-based decision-making relies on trust between the two parties. The parties establish trust in various ways, but typically it’s about making sure that the personal goals between both leaders align. Arrangements are remarkably durable and transportable. Meaning that once you establish an arrangement with someone, trust is the basis of subsequent decisions much faster. It also means that decisions without going through a complex process. The problem is that these decisions tend to be opaque.

Both models taken to extremes can be a disaster. I have worked in both. And they both suck.

The law is THE example of a transactional system. Overly transactional environments turn into bureaucratic, legalistic environments.

A pure transactional system can turn into a totalitarian state. A great example is – “the only way to change this spec is to call a meeting of the change control board and submit a request.”

A pure arrangement system can turn into a cult where belief in the great leader is paramount and the phrase “blah said” is used to justify or denounce everything. That disagreement with the great leader is an unforgivable sin. A great example of this is when someone says – “but SR ARCHITECT FOO said”. Or alternatively – an old-boys network – that is impenetrable.

I’ll also observe that both approaches feel natural to different people. My personal experience is being Greek and Canadian and living in the USA. As a Greek, I believe that laws are suggestions. That relationships, and in particular, family relationships, trump everything. As someone who lives in Sunnyvale, CA, I have learned that in the USA, laws matter, but that the laws are structured to satisfy arrangements among the wealthy.

A transactional system feels like a spectacular waste of time because personal relationships trump everything and that, ultimately, transactional systems are a facade for arrangements.

But I have learned that that is naive. The critical flaw of arrangements is the opacity of building trust and the boundaries of trust. Transactional-based decision-making creates a public record of the moment trust was built and describes the trust boundary. Without such a record, trust-building naturally devolves to family, culture, background, and other attributes.

This brings me to how I think about architecture reviews. The purpose of the review is to create a trust boundary between the approver and the author. The more high-level the spec, the more trust that has to exist. That trust is typically built over a series of smaller successes or previous professional successes. The more detailed the spec, the less trust that exists. A key trust moment as an architect is when you are willing to stop being the sole approver of an area.

The key illusion in all of this is that as an architect the spec actually defines what is built. The reality is that unless you are writing the software, some other human being is going to take that document and do what they think the right thing is. My job is to make sure that they are thinking about the problem in a way that aligns with a reasonable solution.

Decrying arrangements because they are old boys networks is wrong. Decrying transactional-based systems because Process is also wrong. Like most everything in life, there is a balance. And like most everything in life, navigating that balance is the art of living and being a software architect.