wrong tool

You are finite. Zathras is finite. This is wrong tool.

Software Archaeologists

by Leave a Comment

2014-09-16_2231

Work at a large company with a massive code base that has evolved over many years, and you eventually have to engage in Software Archaeology.

Software archaeology is the process of trying to understand critical software systems that are poorly documented, poorly understood or not understood at all and super-critical to your  system.

Imagine you have some module that you are dependent on that has worked for years, and then a bug is uncovered.

You have to go and learn what exactly the module does. By the time you look at the code, the original authors, and their children have all left the company or even if they haven’t its been years since they worked on the code… Sometimes it’s been years since they looked at code period, now being managers or directors or vice-presidents…

At times you can feel like you’re one of those modern explorers violating some tribal rules by exploring in areas that are forbidden. And if you have to modify the code, you wonder if you are like Indiana Jones about to remove the gold idol…


The problem isn’t understanding the structure of the code, software is software. The problem is understanding the intent of the code: understanding where choices were made that were well reasoned, and where choices were made that were expedient. And the problem isn’t even the software that you have to inspect, but that it’s part of a broader sea of software that the ancients wrote that is equally opaque and mysterious.

And the real problem is that static analysis is fine, but what you really need is to understand is how the running system behaves. How it uses memory, how it uses the CPU, how the data structures grow and shrink, what the heart beat of the system is…

At SGI in the late 90’s I did some compiler research as a master’s student to try and address this specific problem. In particular how do you infer things like locking hierarchies in a multi-million line code base when the original authors of the code have since left? Just reading the code is insufficient. Knowing that locks are taken is important, but you also need to understand things about contention, frequency of locks, interplay between systems that are so widely split apart to be mysterious …

When I tried to solve the problem, I looked at using compilers to go and insert code everywhere where something looked like a lock and then use the testing infrastructure to find the locks and their hierarchies…

And then I discovered that the code of the ancients because it was always working has no tests.

Doing your run-time analysis involves figuring out how to test things that always worked. And then you uncover not just one bug, but hundreds… Or are your tests broken because you don’t understand the code… And you wonder as you play with this mysterious gadget… Are you about to destroy the world? Or save it? How many of these bugs are real? And how many of them are real but other pieces of software have worked around those bugs making the whole system work?

And what will be the blow-back of fixing them…

 

And while you sit there with the svn commit about to change the structure of the code of the ancients, you wonder if your hubris is about to bite you in the ass… How could code that has worked for so long, be broken?

And your management team looks at you like those tribal leaders who looked at the explorers, with suspicion and doubt and fear. And you can hear them telling the village youth to arm themselves and kill the interloper before he causes too much harm… Or are they like the hot woman or man begging the evil villain to not destroy the world, or asking the hero:

Are you sure this is going to work?

No fear, the ancients were humans, just like you, you tell them… And that bag of sand is about the same weight as the idol…

Run Indy! Run!

The iPhone did change everything revisiting my predictions from 7 years ago.

by Leave a Comment

7 years ago, I wrote a post questioning the Apple Fan Boys statements that the iPhone changed everything.

I got somethings very right. And I, obviously, got some things very wrong.

My assumption that unlike the iPod Apple would not rule the cellphone market turned out to be correct. My other assumption the iPhone would push the design of phones like the original Mac did turned out to be right.

My original assumption that MS and the cellphone vendors would create a viable alternative ecosystem that would own a much broader chunk than Apple turned out to be partially correct. Instead of MS, the real winner was Google who released the Android.

 

Let’s look at what I said and got right and wrong.

  1. The iPhone was crucial to Apple. And yes it was.
  2. The iPhone was going to push technology trends. Oh boy did it ever
  3. That it was going to be a marginal player. Oh boy was I wrong! WRONG. The share of profits is staggering.
  4. That integration with laptops was important. WRONG! WRONG! WRONG! Obviously I had no idea how much more important integration with cloud services was going to be.
  5. That Microsoft was going to be the bigger threat. OOOPS! Google wasn’t even mentioned! Of course, I wrote the post before Android shipped so that’s my defense. I could not imagine that it would take MS so frigging long to build a credible OS for the phone. And I suspect that Nadella will end that experiment soon.
  6. That the mobile phone providers were going to compete and push Apple into a niche. Right but I guessed wrong on who would do that. Samsung did. Nokia made the strategically boneheaded to go with Windows Mobile instead of Android. A bunch of other players did some good stuff.
  7. The laptop market that I thought was important turns out to be less important than I ever could have imagined for Apple.
  8. And of course, I completely misunderstood the app economy.

7 years later many it’s fun to point out how wrong everyone else got it – it’s even more fun to see how wrong YOU got it 😉

Grand Moff Tarkin Didn’t Want to Pay for Defense in Depth

by Leave a Comment

2014-09-07_1703

In Episode IV of Star Wars, Han Solo, Luke Skywalker, Obi-Wan, and Chewbacca are trapped on the Death Star after their jump from hyperspace.

The Storm Troopers are quickly overwhelmed, and our heroes can access a physical terminal. R2D2 is then able to plug into the computer systems of the Death Star.

R2D2 can quickly access all of the information, including schematics and prisoner information.

In the post-mortem on Coruscant, I can imagine the dialogue:

Palpatine: How were they able to access all of the information?

CISO for the fleet: Well, the Grand Moff decided that the cost of adding firewalls and security systems to partition the network was too costly. He chose to rely on a big ass external firewall. His priority was the ability for his teams to access the information not to protect it.

Palpatine: A single droid was able to quickly and trivially get all of our operational information … Because we had no firewall?

CISO for the fleet: visibly sweating Well it’s more complicated than that. A firewall would have delayed the attack, and at the very least made it harder but nothing could have protected us against a determined attack.

Palpatine: A single bot that was put behind our firewall was able to get everything…

CISO for the fleet: Grand Moff Tarkin felt that it was impossible for a bot to escape the station or communicate externally…

Palpatine: Grand Moff is dead?

CISO for the fleet: Yes, Grand Moff is dead.

Palpatine: Pity. At least we won’t need to replace the commander of our space station. I suppose we’ll need a new CISO for the fleet.

Blue lightning crackles from the Emperor’s hand. The CISO for the fleet crumbles. His second in command steps forward… 

New CISO for the fleet: Emperor, we’ll re-organize our security protocols immediately.

At Zynga, our security team was – actually – ahead of the curve. Our strategy was not to rely just on a hard shell. We also created internal segmentation of our systems.  Basically, we created firewalls around each of our games and each of our systems. This kind of internal segmentation was a layer of protection that I thought was standard practice. More honestly, I thought this kind of protection was unnecessary. The recent disasters show that it is not. Too many people rely on a single external hard shell … unfortunately, once you get through the hard shell, everything is available.

This kind of internal segmentation is not as yet standard practice across the industry, nor was it standard in a galaxy far, far away…

And in all cases, the results were not that pretty…

death-star-explosion-o

 

 

Market signals and the shortage of CS majors

by Leave a Comment

One of the ongoing themes in the tech industry is the shortage of qualified engineers and what can be done about it.

Having lived through one bubble, I thought I might share this picture:

10348778_10152569856434590_8185462582899704564_o

 

This image was taken from a talk delivered by Ed Lazowska at the NCWIT 10th anniversary Summit in 2014. The notes are mine.

The general theme of the talk was what to about the increase in computer science students and was this increase a one time event or part of a broader secular growth.

What was of interest to me was the sudden increase in computer science graduates coinciding with a sudden perceived increase in the financial outcomes of CS students.

Very smart people have a lot of options. And all things being equalled, people will choose things that have a higher payout. If the payout is significantly larger than other options, then they will pick a sub-optimal option to get the money.

Philip Greenspun has a really good post about this. The money quote is the following:

A good career is one that pays well, in which you have a broad choice of full-time and part-time jobs, in which there is some sort of barrier to entry so that you won’t have to compete with a lot of other applicants, in which there are good jobs in every part of the country and internationally, and in which you can enjoy job security in middle age and not be driven out by young people willing to work 100 hours per week.

This is how people actually make professional decisions. And as long as the tech industry compares unfavorably to other career choices there will be a smaller number of people in tech than other fields.  And that shortage is not just about exposure to CS, it is also about financial outcomes. And yes, articles about ageism don’t help.

Update: A friend of mine made another interesting observation about cyclicality of software. Given that stability is important, the cyclical nature of software also pushes people away from this industry. People who are unwilling to tolerate the kind of risk profile that might result in protracted periods of unemployment or work at substantially lower pay.  Combine salaries with job uncertainty and ageism it’s a frigging miracle that anyone is in this industry… Of course, if you read the rest of Greenspun’s article about why men predominate in science you realize that there is analogous argument to be made about the computer industry. Hmm….

As a personal anecdote, in 1994 a very famous professor of computer science told everyone in the room that was studying to get a degree in CS at Brown that we were all doomed. That our jobs were going to go to India. Unsurprisingly the total number of CS graduates that year was 13.

By 2001 the total number of graduates at my school had gone over 100 (in a class size of 2001) a 10x increase.

The dot-com bubble had made CS and the web the way to make money.

And then it collapsed in 2004.

I always wondered if that was a Brown University phenomenon. Turns out it is an industry wide phenomenon.

And … here we go again… As the payout increases, the number of CS majors increases…

My one – obviously self-serving –  observation is that if tech companies really are experiencing a shortage of employees, they do have a strategic option open to them – dramatically increase the salaries of folks in tech to compete with other industries like finance.

 

 

Storage Arbitrage

by 2 Comments

Have been looking at options for cloud storage at the 1TB capacity limit.

Google offers 1TB for 9.99$ a month.

MSFT offers 1TB for 69$ a month <ooops> a year! with MS office apps and 1TB per user for 5 users for 100$

Dropbox offers 500GB f0r 99$ <000ops> it’s actually 499$ a year

 

Why I hate the Great Filter known as Code Reviews

by Leave a Comment

Code reviews have in the wrong environment enabled engineers to relieve themselves of the accountability for writing good code, turned technology leaders into powerless whiners, and enabled managers to ship on time crappy products while remaining blameless for the crap.

Much like the great filter that supposedly eliminates civilizations, code reviews are supposed to eliminate bad code and in the process of being used in that way, eliminate good code as well. The net effect is that the code reviewer is reviewing crap all day. And as a technologist that is very very annoying.

Let’s step back.

A lot of management culture in many large organizations is focused on making sure the lazy ass employees do their jobs correctly. The theory being that at scale, the average employee is … average.

Given the average nature of the average employee, then how do you make sure that the quality doesn’t degrade below average?

Hold the managers accountable for the quality of their work.

In tech companies, a key part of the work of engineers is writing code.

The approach some companies take is to have the managers accountable for the quality of the code.

The problem is that managers are also accountable for shipping on time. And the pressure to ship on time creates an unbearable pressure on the the managers to create a culture that pushes for date over quality over code.

The solution is to create a separate set of leaders who are responsible for technology quality, people like me who have titles like Distinguished Engineer, Architect etc.

The theory being that the tension between the technology leader and the manager will result in solutions that meet the business requirements for both date and quality.

An unfortunate outcome of this process is that the technology leaders are then tasked with ensuring that the code quality is good enough because that is their job. And the process that is used to ensure that code is good enough is the code review.

The problem, and this drives me nuts, is that in the wrong hands what happens is the accountability for the quality of the code is on the reviewer not the author.

A big bug escapes and the first question isn’t:

Who wrote the code

but

Who reviewed it?

Who tested it?

This creates a perception that the author of the code isn’t actually accountable for the bug. The author is, even though they wrote the code, blameless.

This makes the reviewer of the technology the organization’s bitch. On the one hand you’re accountable for the quality of the technology, on the other hand no one reports to you, and managers decide bonuses.

Guess what loses?

Yup, the quality of the technology…

How do you fix this mess?

The first is that you need to change the culture of what a code review is there for and you need to change the relationship between the manager and the technologist.

The code review has to be not about finding bugs but improving the overall quality of the product being produced. The way I like to describe it, everyone is supposed to be doing their best work, and reviews are to make great work better not filtering out bad work.

The manager and the technologist have to both be responsible for the quality of the technology and the date. If the project misses the date or the technology sucks they both failed.

Returning to the title of my post, code reviews when they act as a filter instead of a booster reflect a dysfunctional organization that is broken at it’s core. And the reason I have hated code reviews is that the problem wasn’t the review, the problem was the underlying first principles that motived their existence.

 

Rethinking the Internet of Things

by 3 Comments

Over the last month I’ve been struggling with the Internet of Things. My scale has an internet connection, my TV has an internet connection, my toe has an internet connection and soon my watch will have an internet connection and managing the WIFI passwords and connectivity was a Pain-in-the-Ass.

I kept telling my wife that we need a better solution.

Turns out two very interesting technology trends are going to radically remake the internet of things.

The first is that 4G LTE chips are really cheap. For those not in the know, it used to be the case that building a chip that could do cellular was black-magic, but with 4G LTE this is no longer the case. Thus the how do I connect to the internet is really a how do I make a cellular call and that’s a much simpler user experience.

The second is that BlueTooth LE is transformative. If you think about the internet of things, the things are actually generating a small amount of data very frequently or infrequently. For devices that don’t have enough power to justify placing a 4G LTE chip, BlueTooth LE is a really interesting alternative. BlueTooth LE, if you believe the marketing buzz, will allow a device to transmit longer than the life time of the battery without requiring a battery replacement. Given the range of BlueTooth LE and the existence of the ultimate BlueTooth LE receiver – your cell phone, I can totally imagine a combination of BlueTooth LE talking transparently to your cell phone and your cell phone talking transparently to the internet to transmit the data.

The core objection I had to Internet of Things seems to be addressable with existing technology that is going to market right now.

Cool.

 

Synchronicity

by Leave a Comment

For many different reasons, I finally got around to reading Beautiful Code.

Because Bryan Cantrill happened to be a classmate of mine, I started with his chapter.

The synchronicity is that in 1999 he was working on a problem related to priority inversion on Solaris. Intriguingly, at the same time, a friend of mine at SGI was also working on a similar problem related to priority inversion.

In effect, both companies had decided – independently – to build real time kernels and had independently ran into similar, although very different problems. At SGI we had to figure out to make reader-writer locks deal with priority inversion while not crippling performance.

What made this fascinating is that I tend to agree with Bryan when he talks about software being like math.

 

135651

 

Although I dread using that analogy because real mathematicians would raise an eyebrow….

The more important point is that once you decide to tackle an area in software, the reality is that the solutions tend to have the same attributes and the challenges tend to be the same, what makes software engineering is that the actual solutions differ greatly because they are embodied in a specific implementations.

 

Lessons on Mobile Gaming and Cross platform

by 3 Comments

We have a skype chat where a whole bunch of the ex-Zynga senior technologists (we called them CTO’s) get to chat about stuff we learned.

One thing that came up was our attempts to come up with web-mobile-backend cross platform technology. As a key driver of this effort, I learned that this was a misguided attempt.

Because I like the immediacy of the dialog I am including the skype chat:

[] kostadis roussos: if I was going to enumerate my biggest professional mistake it would be to completely misunderstand the mobile gaming market need for specialized gaming engines targeted at specific games.
[] kostadis roussos: The reality was that I didn’t understand the importance of having a hyper-optimized game for a specific platform and that it was more important to have the BEST game implementation than leverage.
[] kostadis roussos: That leverage was basically unimportant – as a gaming company – … What was important was delivering the best possible user experience.
[] XXX: it’s a fundamentally hard problem to solve and i think what I took away is being polyglot is preferable
[] XXX: it seems like you’re really just talking about the failures of flash/html5 on mobile, not so much a “specialized gaming engine” since cocos2d / unreal / unity are all cross platform
[] kostadis roussos: @XXX – intriguingly in 2013 when we did a whole bunch of analysis 8/10 games had their own custom engines.
[] kostadis roussos: Maybe the data has shifted – and I’ll accept that notion but you really need to have a highly customizable engine where you are not giving up any performance or user-experience .
[] kostadis roussos: My point is that I completely misunderstood the centrality of that point. I didn’t get that we need to think of a game as CMS with a very optimized and specialized “rendering” system and that we needed to have the best one for any game genre or we would lose.
[ kostadis roussos: And so I pursued – html5 as a solution, flash as a solution  and then PlayScript language we built – all of these were attempts to solve the wrong problem. We should have been building the best UX system for I/E games in native languages with some C++ for the business logic bits that could be shared across multiple platforms.
[] kostadis roussos: Oh well. Lesson learned.

My only personal consolation was that I gave the prezi guys some good advice based on my learnings. They feel good about the outcome.

It’s not physics but reasoned arguments

by Leave a Comment

In a recent meeting, a hardware engineer turned and expressed his frustration at the state of a software stack. The frustration centered on the unbelievable complexity the software had accumulated over the years. There was a feeling that somehow, somewhere the software team had lost its way.

An attempt was made to make analogy to hardware. The claim was that hardware that tries to fight physics inherently fails to work. That physical laws constrain design and designs that fight those laws end up failing. And that therefore there were certain physical laws our software had violated and that as a result had become so complex.

In particular an appeal was made to modularity and strong API boundaries as the key to great software. That to not have those things, meant that we were breaking some laws of physics or their nearest software equivalent.

And it got me thinking. A lot. It was an interesting perspective. In all my discussions about software with software guys, the idea that physics would have anything to do with anything never came up.

Over the last month, I came to the conclusion that the hardware engineer was correct about the need to follow some principles but those principles were not physical laws.

All you have to do is read is this paper: Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs by John Backus to realize that software does not view physics or the underlying hardware as its master. In fact Backus calls on software practioners to create software that will force hardware people to create new kinds of hardware:

There are numerous indications that the applicative style of programming can become more powerful than the von Neumann style… when these models and their applicative languages have proved their superiority over conventional languages will we have the economic basis to develop the new kind of computer that can best implement them. Only then, perhaps, will we be able to fully utilize large-scale integrated circuits in a computer design not limited by the von Neumann bottleneck

Physics is the thing that gets in the way of our software, the thing that enables us to execute our software in the real world, but not our master.

Instead, I would argue that the laws that software ignores at it’s peril is the structure of reasoned arguments.

What do I mean? Software has the wonderful property that it’s correctness is formally unknowable. We can’t know if it is correct.  And if we can’t know, what do we do?

All we can do is reason about the software. And to reason about software is to try and understand an argument.

If that is the case, then an argument that is well crafted is:

  1. Clear
  2. Concise
  3. Structured
  4. Has a general structure based on specific facts
  5. Complete
  6. Novel
  7. Builds on previous arguments

All attributes of good software are actually the attributes of good reasoning.

As an argument acquires complexity it becomes harder to follow, harder to re-use, harder to understand, harder to leverage, harder to understand it’s correctness.

And that fact, perhaps, explains why software is so hard. Good software reflects the quality of our ability to reason and articulate our reasoning.

Things like modularity and APIs are really techniques to construct good arguments.

Our software mess wasn’t because we had failed to followed some laws of physics, it’s because we had chosen to become sloppy in our thinking and our reasoning.

What does this mean, practically?

As we try and struggle on how to make software better, what we don’t talk about is the need to improve our engineers ability to reason. Our schools focused on mechanisms and techniques spend very little time on specific training on how to argue.

Perhaps, there is something we need to learn from the law. Unclear.