In a recent meeting, a hardware engineer turned and expressed his frustration at the state of a software stack. The frustration centered on the unbelievable complexity the software had accumulated over the years. There was a feeling that somehow, somewhere the software team had lost its way.

An attempt was made to make analogy to hardware. The claim was that hardware that tries to fight physics inherently fails to work. That physical laws constrain design and designs that fight those laws end up failing. And that therefore there were certain physical laws our software had violated and that as a result had become so complex.

In particular an appeal was made to modularity and strong API boundaries as the key to great software. That to not have those things, meant that we were breaking some laws of physics or their nearest software equivalent.

And it got me thinking. A lot. It was an interesting perspective. In all my discussions about software with software guys, the idea that physics would have anything to do with anything never came up.

Over the last month, I came to the conclusion that the hardware engineer was correct about the need to follow some principles but those principles were not physical laws.

All you have to do is read is this paper: Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs by John Backus to realize that software does not view physics or the underlying hardware as its master. In fact Backus calls on software practioners to create software that will force hardware people to create new kinds of hardware:

There are numerous indications that the applicative style of programming can become more powerful than the von Neumann style… when these models and their applicative languages have proved their superiority over conventional languages will we have the economic basis to develop the new kind of computer that can best implement them. Only then, perhaps, will we be able to fully utilize large-scale integrated circuits in a computer design not limited by the von Neumann bottleneck

Physics is the thing that gets in the way of our software, the thing that enables us to execute our software in the real world, but not our master.

Instead, I would argue that the laws that software ignores at it’s peril is the structure of reasoned arguments.

What do I mean? Software has the wonderful property that it’s correctness is formally unknowable. We can’t know if it is correct.  And if we can’t know, what do we do?

All we can do is reason about the software. And to reason about software is to try and understand an argument.

If that is the case, then an argument that is well crafted is:

1. Clear
2. Concise
3. Structured
4. Has a general structure based on specific facts
5. Complete
6. Novel
7. Builds on previous arguments

All attributes of good software are actually the attributes of good reasoning.

As an argument acquires complexity it becomes harder to follow, harder to re-use, harder to understand, harder to leverage, harder to understand it’s correctness.

And that fact, perhaps, explains why software is so hard. Good software reflects the quality of our ability to reason and articulate our reasoning.

Things like modularity and APIs are really techniques to construct good arguments.

Our software mess wasn’t because we had failed to followed some laws of physics, it’s because we had chosen to become sloppy in our thinking and our reasoning.

What does this mean, practically?

As we try and struggle on how to make software better, what we don’t talk about is the need to improve our engineers ability to reason. Our schools focused on mechanisms and techniques spend very little time on specific training on how to argue.

Perhaps, there is something we need to learn from the law. Unclear.

Software engineers, for all of their deep rational view of the universe, tend to sometimes approach debugging like soothsayers inspecting entrails trying to divine what the hell is going on.

The scene played out like this:

Bug report: XyZZy is happening

Engineer stares at bug report, stares at software, stares at test case that doesn’t reproduce it.

Engineer then tries things that make sense.

Bug doesn’t go away

Then Engineer tries random acts of coding to see if the bug goes away.

And his peers start saying: Wow, yeah I think I saw something like that. Did you try the toggle the frommlebit?

And the engineer says: But this has nothing to do with the frommle bit, this is tied to pluxion submodule.

And another peer says: Trying adjusting the memory size of the core structure.

And the engineer says: But my code never touches that core structure!

And another peer says: We’ve had success with this kind of problem that we don’t understand when we altered the return value of the function..

And the engineer says: But this makes no sense.

And they all respond like a Greek chorus: But it sometimes works.

At this point in time the engineer is hoping and praying that just moving the code around will cause the bug to go away.

My buddy and I used to call that last stage of debugging “Voodoo debugging”.

You don’t understand the problem, you don’t understand why the bug is happening, you have a deadline to meet, so you try random things hoping that the bug will just go away.

And the sad truth is that the simple act of moving things around may cause the bug report to go away. The bug may reappear later in some other poor bastards bug queue. And the Voodoo debugging will continue as people try this and try that.

In the pre-cloud era of computing, when software engineers believed that it was their divine right to understand how it all fit together, this was viewed as a moral failing. Back in the day we could understand how the entire system from the silicon up worked and we could use that information to thoroughly debug and understand the problem.

Those who lacked the moral fiber and intellectual rigor to do real software engineering practice Voodoo debugging. The rest of us understood how the entire system fit together and resolved the issue down to it’s root cause.

Then I moved to Zynga. And at Zynga it became apparent that Voodoo debugging wasn’t just a product of a lack of moral fiber or intellectual rigor it was a rational response to the environment.

I talk about this a lot in an answer on Quora. The essential elements of the arguments are that systems are in such continuous flux that understanding how the system is working in its entirety is impossible and worse by the time you understand it, the system has changed.

Bugs come and bugs go. And bugs have an element of Voodoo magic about them. Trying to understand bugs and resolving them is impossible. We are now in the era of trying to resolve issues without fixing them. We have become doctors who offer palliatives rather than cures in the hope that the problem will just go away.

And so I embrace my inner Voodoo doctor. I will try the magic cants, I will pray that the next update of the API I am using fixes the problem, and I will internalize the truth that finding the root cause is an illusion my earlier me had.

On Dec 1st, 2005, the Boston Bruins made a catastrophically piss poor decision to trade Joe Thornton to the San Jose Sharks. As a result of that trade, San Jose became a perennial contender. Boston did eventually become a great great team, but the trade of Thornton could only be described as the agonizingly slow route.

The rationale for the trade was the belief in the gutless Thornton mystique, a story created from the nonsense of his first foray into the playoffs when with a broken rib he was only able to get one measly assist in a 7 game series loss to the Montreal Canadiens.

Over the years I have seen Joe Thornton carry mediocre San Jose teams to the playoffs only to be continuously thwarted by Doug Wilson’s inability to get a decent third line…

Or so I think …

My point in all of this, is that Joe Thornton, to anyone who knows hockey, is most definitely not a Bruins player…

Unfortunately Facebook’s algorithms are a little bit less aware of the NHL and hockey… and produced this trending result:

And herein lies the danger of computers and their lack of semantic awareness. They create such delightful moments of absurdity. The algorithms confused Joe Thornton with Scott Thornton.

Without any knowledge of the internals of the implementation, I could suppose that Facebook used the fact that I post stuff about Joe Thornton, including posts I have made about how he once played for the Bruins to surface a note about Scott.

This is the inherent danger in an era of big data. We have all of this data that we can then use clever algorithms to create interesting and useful results that we can immediately share with the world. And most of the time the results are good.

The problem arises when the result is not good

And it’s not that computers are less reliable than human beings, humans are just as prone to absurd errors, the problem with computers is how we humans are interpreting the results. Or choosing to share the results without interpreting them because they are correct most of the time.

The key difference with the past is that the computers had access to less data, so our ability to find relationships where none existed was limited. We knew that the data was too small to trust any result that did not fit our intuition.

But now, with so much data, who is to say that the relationship doesn’t exist?

Perhaps our clever algorithms did find relationships that were only possible because of the amount of data we had collected.

And so we need to be careful and humble as we look at the results. Sometimes we will find things in the data that we didn’t know existed, and sometimes we might be looking at the night sky and seeing a horse in the stars.

Because, I can assure you that Joe Thornton is both not a Bruins player and not playing hockey right now.

And as I review my post, it’s tempting to ask myself – maybe the data is telling me a truth that Joe Thornton is not as great in the playoffs as I think he is. Maybe I need to listen the fact that he was -6 and scored only 3 points  this year and that that had something to do with the performance of the Sharks. Maybe. Or not.

As a lover of fine cuisine, I am always struck at the difficulty of choosing between a tasting menu and a massive steak. The tasting menu is about sampling a little bit of a lot of things. The chef is trying to maximize the number of unique flavors over calories. Whereas when you eat a steak you are trying to maximize the number of steak calories.

Tasting menu is about doing a lot of little things.

Steak is about doing a lot per thing.

And that is a very crude analogy about the difference between storage and networking.

If we consider a packet a unit of work, storage systems expend a lot of resources per packet as compared to network devices and so process vastly fewer packets.

Storage system architecture must optimize the use of very slow devices. The slower the device the more computation value you get out of clever algorithms on a per packet basis if they can avoid going to disk. So storage uses complex in-memory data structure with complex algorithms that blow out data caches because storage systems have to. The reason flash was so disruptive to the storage industry is because the amount of computation per packet became less as the performance of flash closed the CPU vs storage performance gap.

Network system architecture must, on the other hand, do the minimal amount of work because it is inherently slowing down the flow of electrons. A beam of light down a cable that hits a network device feels like it went from light speed to impulse speed. As a result networking systems must do heroics to minimize the amount of computation and simultaneously make the computation go as fast as possible.

Practically what this means is that a network system can benefit hugely from custom specialized computational hardware whereas storage can not.

In English: ASIC for networking making sense, no one uses anything but Intel CPU for storage. And every company that tried to use custom hardware in
storage.

This answers a long standing question I had, as to why storage companies never used custom computational hardware and networking companies did.

It also answers why server companies go after storage and not networking: the hardware server companies build is more aligned to a storage problem than a networking problem. For example Sun and Dell and HP went after storage in a way they never went after networking.

This also answers the mystery Cisco. Much like Intel won the server CPU war, a single dominant player can emerge in networking given the need for custom hardware. At some point in time only a small number of players can play the game.

And last it answers the software defined storage mystery. In software defined networking we move control out of the hardware device. Software defined storage just says I can run control and data path code on any piece of hardware… This is an important distinction that is not obvious.

Coolness.