wrong tool

As I write this post during take off, I am reminded of the power of compelling and amazing products.

For the longest time, the FAA had this silly rule about electronics. Until the kindle showed up it was a minor nuisance. You learned to read your book during flights. And heck the size of the seats made it impractical to use a laptop except for business travelers.

Then the kindle showed up and all of a sudden the rule became a source of intense aggravation but heck it only affected those of us that wanted to read.

For the first time the rule became irritating.

But when the iPhone showed up the rule became unbearable.

You couldn’t play a game of words with friends, your kid couldn’t keep watching a movie and you couldn’t read a book. So we started breaking the rule all of the time.

There were moments like these you could never record.

Your son clutching his toy helicopter and airplane, eyes as big as saucers staring out of airplane all because of bad bad science.

I don’t know if Amazon or Apple ever lobbied on our behalf, but I doubt it.

What took down the stupid rule was that Steve and Jeff created a product that consumers demanded to use all of the time making the rule unenforceable.

Great products can transform the world by creating a demand to overturn something stupid. Something to think about as we build stuff.

Every so often, I get pulled into a discussion about how do you identify a great systems programmer. Mostly because I hang out with other systems programmers and we’re evaluating a candidate for a job opening. And then the usual discussion about interview questions and projects emerge. In fact there is a quora question that I answered on the topic. The discussion usually devolves into a discussion of the ability to understand things like hardware software interface, kernel internals, asynchronous behavior etc.

Just recently, at a meeting in Juniper, it struck me that we never talk about the truly rare skill of systems architecture. And more importantly, how do you find and recognize that skill?

So what is systems architecture?

Systems architecture is the ability to understand the abstract system architecture of the problem, understand what kind of hardware options exist and then define a software architecture that is able to exploit the hardware in ways that add tremendous business value.

System architecture is what takes  something on the left of the picture and then defines something on the right. `

Mouthful ain’t it?

So let’s break this up a little bit.

Abstract system architecture

When you consider a system of any kind, there exists a way to describe that system that is decoupled from any implementation yet at the same time is readily recognized by experts in the art.

Let’s consider something like a file system. A trivial file system has a block virtualization layer that maps logical blocks to physical blocks, a way to organize virtual disk blocks into containers and these containers can then be organized in various interesting ways involving different hierarchies, a mechanism for writing  disk, and a mechanism of reading blocks from disk into memory.

Immediately anyone who is expert in the art, will point out a shit-load of stuff that I glossed over. And that discussion is important because system architecture is about agreeing what are the important things that are always there. And more intriguingly what things that were important that can be dropped.

But beyond just being able to articulate the abstract system architecure, you also have to have a keen insight into the relative computational complexity of the pieces. For example, how much memory and CPU does maintaining a map consume versus doing a read or write.

And beyond the computational complexity understanding what pieces must be very robust and what pieces can be less robust is also important – for example the individual blocks are unimportant unless they contain map information.

Understand what kind of hardware options exist

Given an abstract system architecture, then the next question is how to manifest that model. The truth is that systems architecture is the ultimate wrong tool problem. The perfect abstract system can not be implemented without huge compromise to business value – either in terms of performance or cost or availability.

For example, in the case of file systems, there is the choice of processor and memory and what kind of storage you will use. The more CPU and memory you have the more computation you can do per IO and the less IO you need from the underlying physical sub-system. The faster the physical storage system you have, the more performance you need out of the CPU because you have less time to do work.

Understanding the tradeoffs and trends is really important. And understanding the different kinds of options within a category is really important.

What’s not important is understanding the exact details until you get to actually building a specific instance of your architecture.

Knowing that CPU’s of type A perform 3x better than of type B and that the projected performance curve over the next 5 years (from vendor roadmaps) is crucial especially if Type A has a different architecture than Type B with different tradeoffs on how you write software.

Knowing the ratio of performance between Disk and Memory is important.

And knowing how all of these ratios work with each other is also really important.

Add tremendous business value

So I am jumping ahead because this is the most peculiar statement. The thing about abstract systems architectures is that they can be viewed as an end goal in and of themselves. And there is a lot of value in pursuing that research and understanding.

In my mind, systems is really about building something in the hear and now, and something that is built in the hear and now needs to add value in some material way. And, for better or for worse, I use the term business value to define material way.

Perhaps more prosaically, a better definition is that systems architectures that are interesting have to deliver better top-line performance that is sustainable, better availability or better price/ performance where price can now include power consumption.

Define a software architecture

Most software architectures look like a collection of boxes that have arrows that point to each other.

Fundamentally a systems software architecture is a decomposition of a software system that maps to how the hardware can be taken advantage of to deliver exceptional business value. A system software architecture will not, almost definitionally, look like the abstract system architecture of the system because the implementation in the real world requires trade offs to deliver value.

This kind of decomposition, in my mind, is the essential difference between systems architecture and user application programming. Systems architecture considers how the hardware behaves and decomposes the software to take advantage of the hardware capabilities. User application programming considers how people behave and decomposes the software around that axis.

So why is this important?

Massive revenue and value opportunities exist when you are able to take an abstract system architecture, and then define a software architecture that leverages new hardware that allows you to get a 100x improvement along some kind of business value axis.

NetApp, back in the day, is an example of such an opportunity. Hitz and others were able to see a huge opportunity around RAID and were able to articulate a software architecture that exploited RAID in a unique way that then delivered massive business value.

Not every systems architecture is that valuable, mind you, but some are.

And this skill to defining architecture is applicable to whether you’re building an OS, a radically new kind of PHP compiler or a cloud application. And this skill is, in my mind, not a skill we spend enough time defining and examining and interviewing for.

In the digital download age, failure gets deleted from a collection of web servers. And nobody bothers to record your failure, there is nothing to left of your existence.

Does anyone remember the pet socket?

Other than digital images and entries in wikipedia the failure has disappeared.

It wasn’t always that way.

1n 1982, a ten year old version of me, begged and pleaded that his mom buy him a video game based on the hit movie ET. As a proud owner of an Atari gaming station, the expensive cartridge offered a way to relive the movie experience.

I don’t remember much of the game, other than owning it and waiting in line to get it. I think that was the high point of owning the game.

Many, many years later (approximately 20 and many eons after I tossed that Atari system and the game cartridges that went along with it)… I discovered on the web that apparently the game wasn’t that successful.

Apparently I wasn’t the only person to think the game wasn’t memorable.

There was an urban legend about the cache of games sitting in the desert because no one wanted to buy them. And lo-and-behold, the truth was out there…

It also turns out that another lesson in never saying never, is that I wondered if technology archaeologists would come to exist in 100 years, well it turns out that they are already here…

Yesterday I committed the cardinal sin, assuming a problem would not be solved.

I assumed that the state of the art of the storage was:

Also known as spinning magnetic rust.

And whenever you make an assumption about that you end up being screwed within 24 hours of saying it.

A buddy of mine pointed out that there is a solution.

Apparently we can use quartz to store data for 300 million years.

Well then.

The recent announcement from the Warhol archives about recovering some Amiga art

got me thinking about how miraculous this feels and I am not talking about Warhol using an Amiga, but us being able to recover the data.

When I think about signature scenes in science fiction, the activation of the ancient computer and recovery of the data from the data banks resonates.

The story resonates because it goes to the even deeper tale of finding ancient knowledge lost in the ground, a story that the historian in me finds particularly alluring.

As I got older, and especially after a decade at NetApp, I learned how absurd the tale was. There are several challenges in decoding data. The first is the media itself requires special devices that fail over time and are no longer built, the second is the data format itself is tied to some software that requires a special piece of hardware that may  no longer be manufactured and has failed, and finally there is the very unfortunate reality that the media itself fails over time. This doesn’t even begin to address the challenge of information encoding and decoding and, of course, encryption…

One of the things that fascinates me about the future and the cloud is that when some bad event happens, vast amounts of knowledge will be lost. What if all of science and technology abruptly vanishes in a sea of lost electrons leaving the future with nothing but documents about stuff we invented last century, that we might have destroyed because it was digitized … Who wants to keep dead tree media when you can digitize it?

And even if I don’t care about the cloud, I worry about me. Will everything about me get deleted because my credit card expired after I died?

And when I think about that some more, and I think about biodegradable infrastructures, and products, I begin to wonder, what if there was another civilization like ours, that had completely digitized their knowledge, had a biodegradable physical infrastructure and some catastrophic event occurred that only left these primitive dwellings behind?

So when I see someone pull up an old image that was stored on electronic media on a long-lost computer, I am amazed…

And then I wonder, will we have a profession of computer archaeology that emerges 100 years from now?

One of the most irritating conversations I am dragged into, is that moment when some senior management person or some senior technology boss says:

We need to do more innovation.

My heart stops. Oh-god-no… If we’re asking the question we’re already screwed.

And Lord Help Me, but the conversation that always follows the mandate for more innovation is the debate:

What is innovation…

And then every single person in the organization becomes a theologian debating how many angels can dance on a the head of a pin… We have offsites, and training classes, and meetings and posters and metrics and …

Because, ultimately, one man’s innovations is another woman’s obviously boring idea.

And that got me thinking. What if the man and the woman are right? That innovation is actually relative to your point of view?

When I think about innovation, I think of the process of doing something new. The question then becomes, what is new?

Let’s consider three interesting edges in the world of human knowledge. One edge is the true edge of the unknown, this is the limit of what is documented and understood by any one person. Another edge is the edge of what can be commercially built. And another edge is the mainstream of what can be commercially built.

The true edge of the unknown is where academic research lives. The second edge, is where new products and businesses live. The third edge is where the vast majority of all research and development occurs.

The picture above shows these three edges. What it also shows is how the further you are at the limit of human knowledge the more what every else is doing appears less innovative, and the further you are towards the mainstream the more magical everything else everyone else is doing appears. And I’ll contend here that that’s the wrong view of the universe. And that a lot of our discussion about innovation is really driven by this relativistic picture. Folks on the right look towards the left and want to be more like the people on the left without realising that they don’t have to.

One of my recent, hypotheses, is that anyone who is on any one of those edges will see any new idea as obviously flowing from the context of the ideas surrounding them. The idea may be novel, but not entirely surprising. At the edge, everyone is looking at the same sets of problems and data, it’s the solutions that are unique. Every so often, there will be something that comes out of left field, but that is for the most part rare.

You don’t come up with a research idea in a vacuum. There is a field, the field has a set of problems that are of interest, and your ideas are vetted in the context of those problems of interest and the quality of the ideas you put forth.

As a concrete example, consider theoretical physics. I recently read this fascinating book by Leo Smolin that has a long diatribe about string theory but beneath that surface offers a unique perspective on how science is done. Essentially what theoretical physicists do is they come up with theories by looking at data, exploring math  as they try to make fundamental insights about the universe. Every so often an idea will have enough merit that other theoretical physicists will take that idea and refine it further, until at some point in time the refinement results in an experiment that requires even more science to do that proves the theory. The whole process can take over 50 years from start to finish.

There are three essential edges, one edge is coming up with brand new fundamental ideas of the universe, another figuring out those ideas, and another figuring out how to turn them into experiments.

So where is innovation happening? In all three areas. Every person is moving the frontier of human knowledge a little bit further.

And yet, the guy who came up with the Higgs boson, is far more well known that the vast army of people who turned that idea into an experiment that could verify the result. I remember hearing an interview with the director of the experiment who had to explain to the NPR reporter how much science and engineering had to be created just to validate that theory.

So is Higgs the only innovator or is everyone else in between an innovator as well?

And that got me thinking that it’s all relative. Everyone is innovating at every step of the way, and our reward and star system makes it impossible for us to recognize all of those incremental innovations. We reward the guy with the fundamental idea with a Nobel prize and ignore everyone else who turned it into something real.

But so what? Well the reason this matters to me is that when my boss brings up the dreaded We need more innovation discussion, I worry that his idea is we need more Higgs’… And that by creating that bar for innovation we’ll essentially tell everyone that unless they come up with new fundamental ideas they don’t have anything of any value…

And so coming back to my discipline, I think of Google. Google is about a decade ahead of everyone else in data center infrastructure. You read their papers and think, my God they are magicians, and to a certain extent they are. More importantly they are also at the edge of what is commercially build-able and they, uniquely, have access to the data and problems that are worth solving in 10 years, and so are able to solve them first. For the rest of us struggling in the relativistic past, it can feel like we’re in the stone age, being blessed by these aliens with gifts from the future. But really what’s going on is that they are just seeing the problems before the rest of us have a chance.

And so if you’re in the same space as Google, you can feel worthless because you’re not innovating like them. They publish papers and industries emerge, you’re just trying to keep some MySQL DB’s from crashing over…

But I wonder, if at Google they feel the same way I do … I wonder if at Google the solutions are considered novel, and ingenious but not miraculous. Perhaps like Arthur C. Clarke said their sufficiently advanced technology appears like magic …

So then coming down to next edge which is the mainstream world of technology where you are not at the bleeding edge of the known or the commercially viable, is innovation possible?

And I think it is. Because innovation is about looking at data and the problems worth solving and solving them in a way that is novel. Now the novelty may not be that novel because you’re taking something someone else did and adapting it to your context, but that adaptation requires thought and some novel ideas.

But then isn’t that what almost all innovation is? Taking some ideas from some place, looking at some data, considering what the important problems are and trying to do something new?

So then if that’s the case, then the answer to my boss is relatively simple:

Yeah, I agree. Let’s look at some data, let’s look at the problems of interest and let’s try to solve them in some way that is different than how we were solving them yesterday.

And to my current boss’ credit, that’s where we landed. So if you’re job is to stop the MySQL DB’s from crashing and come up with a way of doing it that was better than the way things were working a day ago, then you just innovated and that’s cool.

But I can almost hear the voice of some smart engineer who says: but that’s not real innovation… real innovation is X or Y or Z … and I respond:

What is innovation is relative, and you can innovate on all edges and the only real innovation is the process I just described.

But I am a little bit dissatisfied with that answer. So I will amend it here.

Here’s what I will concede, although the innovations necessary to transform the Higgs’ theory into a workable innovation were vast and mysterious and magical, we have to give credit to those who do come up with the fundamental ideas because they are so rare. And in giving credit, we acknowledge their significance, but they should never, ever, ever get in the way of everyone innovating.