wrong tool

You are finite. Zathras is finite. This is wrong tool.

Grand Moff Tarkin Didn’t Want to Pay for Defense in Depth

by Leave a Comment

2014-09-07_1703

In Episode IV of Star Wars, Han Solo, Luke Skywalker, Obi-Wan, and Chewbacca are trapped on the Death Star after their jump from hyperspace.

The Storm Troopers are quickly overwhelmed, and our heroes can access a physical terminal. R2D2 is then able to plug into the computer systems of the Death Star.

R2D2 can quickly access all of the information, including schematics and prisoner information.

In the post-mortem on Coruscant, I can imagine the dialogue:

Palpatine: How were they able to access all of the information?

CISO for the fleet: Well, the Grand Moff decided that the cost of adding firewalls and security systems to partition the network was too costly. He chose to rely on a big ass external firewall. His priority was the ability for his teams to access the information not to protect it.

Palpatine: A single droid was able to quickly and trivially get all of our operational information … Because we had no firewall?

CISO for the fleet: visibly sweating Well it’s more complicated than that. A firewall would have delayed the attack, and at the very least made it harder but nothing could have protected us against a determined attack.

Palpatine: A single bot that was put behind our firewall was able to get everything…

CISO for the fleet: Grand Moff Tarkin felt that it was impossible for a bot to escape the station or communicate externally…

Palpatine: Grand Moff is dead?

CISO for the fleet: Yes, Grand Moff is dead.

Palpatine: Pity. At least we won’t need to replace the commander of our space station. I suppose we’ll need a new CISO for the fleet.

Blue lightning crackles from the Emperor’s hand. The CISO for the fleet crumbles. His second in command steps forward… 

New CISO for the fleet: Emperor, we’ll re-organize our security protocols immediately.

At Zynga, our security team was – actually – ahead of the curve. Our strategy was not to rely just on a hard shell. We also created internal segmentation of our systems.  Basically, we created firewalls around each of our games and each of our systems. This kind of internal segmentation was a layer of protection that I thought was standard practice. More honestly, I thought this kind of protection was unnecessary. The recent disasters show that it is not. Too many people rely on a single external hard shell … unfortunately, once you get through the hard shell, everything is available.

This kind of internal segmentation is not as yet standard practice across the industry, nor was it standard in a galaxy far, far away…

And in all cases, the results were not that pretty…

death-star-explosion-o

 

 

Market signals and the shortage of CS majors

by Leave a Comment

One of the ongoing themes in the tech industry is the shortage of qualified engineers and what can be done about it.

Having lived through one bubble, I thought I might share this picture:

10348778_10152569856434590_8185462582899704564_o

 

This image was taken from a talk delivered by Ed Lazowska at the NCWIT 10th anniversary Summit in 2014. The notes are mine.

The general theme of the talk was what to about the increase in computer science students and was this increase a one time event or part of a broader secular growth.

What was of interest to me was the sudden increase in computer science graduates coinciding with a sudden perceived increase in the financial outcomes of CS students.

Very smart people have a lot of options. And all things being equalled, people will choose things that have a higher payout. If the payout is significantly larger than other options, then they will pick a sub-optimal option to get the money.

Philip Greenspun has a really good post about this. The money quote is the following:

A good career is one that pays well, in which you have a broad choice of full-time and part-time jobs, in which there is some sort of barrier to entry so that you won’t have to compete with a lot of other applicants, in which there are good jobs in every part of the country and internationally, and in which you can enjoy job security in middle age and not be driven out by young people willing to work 100 hours per week.

This is how people actually make professional decisions. And as long as the tech industry compares unfavorably to other career choices there will be a smaller number of people in tech than other fields.  And that shortage is not just about exposure to CS, it is also about financial outcomes. And yes, articles about ageism don’t help.

Update: A friend of mine made another interesting observation about cyclicality of software. Given that stability is important, the cyclical nature of software also pushes people away from this industry. People who are unwilling to tolerate the kind of risk profile that might result in protracted periods of unemployment or work at substantially lower pay.  Combine salaries with job uncertainty and ageism it’s a frigging miracle that anyone is in this industry… Of course, if you read the rest of Greenspun’s article about why men predominate in science you realize that there is analogous argument to be made about the computer industry. Hmm….

As a personal anecdote, in 1994 a very famous professor of computer science told everyone in the room that was studying to get a degree in CS at Brown that we were all doomed. That our jobs were going to go to India. Unsurprisingly the total number of CS graduates that year was 13.

By 2001 the total number of graduates at my school had gone over 100 (in a class size of 2001) a 10x increase.

The dot-com bubble had made CS and the web the way to make money.

And then it collapsed in 2004.

I always wondered if that was a Brown University phenomenon. Turns out it is an industry wide phenomenon.

And … here we go again… As the payout increases, the number of CS majors increases…

My one – obviously self-serving –  observation is that if tech companies really are experiencing a shortage of employees, they do have a strategic option open to them – dramatically increase the salaries of folks in tech to compete with other industries like finance.

 

 

Storage Arbitrage

by 2 Comments

Have been looking at options for cloud storage at the 1TB capacity limit.

Google offers 1TB for 9.99$ a month.

MSFT offers 1TB for 69$ a month <ooops> a year! with MS office apps and 1TB per user for 5 users for 100$

Dropbox offers 500GB f0r 99$ <000ops> it’s actually 499$ a year

 

Why I hate the Great Filter known as Code Reviews

by Leave a Comment

Code reviews have in the wrong environment enabled engineers to relieve themselves of the accountability for writing good code, turned technology leaders into powerless whiners, and enabled managers to ship on time crappy products while remaining blameless for the crap.

Much like the great filter that supposedly eliminates civilizations, code reviews are supposed to eliminate bad code and in the process of being used in that way, eliminate good code as well. The net effect is that the code reviewer is reviewing crap all day. And as a technologist that is very very annoying.

Let’s step back.

A lot of management culture in many large organizations is focused on making sure the lazy ass employees do their jobs correctly. The theory being that at scale, the average employee is … average.

Given the average nature of the average employee, then how do you make sure that the quality doesn’t degrade below average?

Hold the managers accountable for the quality of their work.

In tech companies, a key part of the work of engineers is writing code.

The approach some companies take is to have the managers accountable for the quality of the code.

The problem is that managers are also accountable for shipping on time. And the pressure to ship on time creates an unbearable pressure on the the managers to create a culture that pushes for date over quality over code.

The solution is to create a separate set of leaders who are responsible for technology quality, people like me who have titles like Distinguished Engineer, Architect etc.

The theory being that the tension between the technology leader and the manager will result in solutions that meet the business requirements for both date and quality.

An unfortunate outcome of this process is that the technology leaders are then tasked with ensuring that the code quality is good enough because that is their job. And the process that is used to ensure that code is good enough is the code review.

The problem, and this drives me nuts, is that in the wrong hands what happens is the accountability for the quality of the code is on the reviewer not the author.

A big bug escapes and the first question isn’t:

Who wrote the code

but

Who reviewed it?

Who tested it?

This creates a perception that the author of the code isn’t actually accountable for the bug. The author is, even though they wrote the code, blameless.

This makes the reviewer of the technology the organization’s bitch. On the one hand you’re accountable for the quality of the technology, on the other hand no one reports to you, and managers decide bonuses.

Guess what loses?

Yup, the quality of the technology…

How do you fix this mess?

The first is that you need to change the culture of what a code review is there for and you need to change the relationship between the manager and the technologist.

The code review has to be not about finding bugs but improving the overall quality of the product being produced. The way I like to describe it, everyone is supposed to be doing their best work, and reviews are to make great work better not filtering out bad work.

The manager and the technologist have to both be responsible for the quality of the technology and the date. If the project misses the date or the technology sucks they both failed.

Returning to the title of my post, code reviews when they act as a filter instead of a booster reflect a dysfunctional organization that is broken at it’s core. And the reason I have hated code reviews is that the problem wasn’t the review, the problem was the underlying first principles that motived their existence.

 

Rethinking the Internet of Things

by 3 Comments

Over the last month I’ve been struggling with the Internet of Things. My scale has an internet connection, my TV has an internet connection, my toe has an internet connection and soon my watch will have an internet connection and managing the WIFI passwords and connectivity was a Pain-in-the-Ass.

I kept telling my wife that we need a better solution.

Turns out two very interesting technology trends are going to radically remake the internet of things.

The first is that 4G LTE chips are really cheap. For those not in the know, it used to be the case that building a chip that could do cellular was black-magic, but with 4G LTE this is no longer the case. Thus the how do I connect to the internet is really a how do I make a cellular call and that’s a much simpler user experience.

The second is that BlueTooth LE is transformative. If you think about the internet of things, the things are actually generating a small amount of data very frequently or infrequently. For devices that don’t have enough power to justify placing a 4G LTE chip, BlueTooth LE is a really interesting alternative. BlueTooth LE, if you believe the marketing buzz, will allow a device to transmit longer than the life time of the battery without requiring a battery replacement. Given the range of BlueTooth LE and the existence of the ultimate BlueTooth LE receiver – your cell phone, I can totally imagine a combination of BlueTooth LE talking transparently to your cell phone and your cell phone talking transparently to the internet to transmit the data.

The core objection I had to Internet of Things seems to be addressable with existing technology that is going to market right now.

Cool.

 

Synchronicity

by Leave a Comment

For many different reasons, I finally got around to reading Beautiful Code.

Because Bryan Cantrill happened to be a classmate of mine, I started with his chapter.

The synchronicity is that in 1999 he was working on a problem related to priority inversion on Solaris. Intriguingly, at the same time, a friend of mine at SGI was also working on a similar problem related to priority inversion.

In effect, both companies had decided – independently – to build real time kernels and had independently ran into similar, although very different problems. At SGI we had to figure out to make reader-writer locks deal with priority inversion while not crippling performance.

What made this fascinating is that I tend to agree with Bryan when he talks about software being like math.

 

135651

 

Although I dread using that analogy because real mathematicians would raise an eyebrow….

The more important point is that once you decide to tackle an area in software, the reality is that the solutions tend to have the same attributes and the challenges tend to be the same, what makes software engineering is that the actual solutions differ greatly because they are embodied in a specific implementations.

 

Lessons on Mobile Gaming and Cross platform

by 3 Comments

We have a skype chat where a whole bunch of the ex-Zynga senior technologists (we called them CTO’s) get to chat about stuff we learned.

One thing that came up was our attempts to come up with web-mobile-backend cross platform technology. As a key driver of this effort, I learned that this was a misguided attempt.

Because I like the immediacy of the dialog I am including the skype chat:

[] kostadis roussos: if I was going to enumerate my biggest professional mistake it would be to completely misunderstand the mobile gaming market need for specialized gaming engines targeted at specific games.
[] kostadis roussos: The reality was that I didn’t understand the importance of having a hyper-optimized game for a specific platform and that it was more important to have the BEST game implementation than leverage.
[] kostadis roussos: That leverage was basically unimportant – as a gaming company – … What was important was delivering the best possible user experience.
[] XXX: it’s a fundamentally hard problem to solve and i think what I took away is being polyglot is preferable
[] XXX: it seems like you’re really just talking about the failures of flash/html5 on mobile, not so much a “specialized gaming engine” since cocos2d / unreal / unity are all cross platform
[] kostadis roussos: @XXX – intriguingly in 2013 when we did a whole bunch of analysis 8/10 games had their own custom engines.
[] kostadis roussos: Maybe the data has shifted – and I’ll accept that notion but you really need to have a highly customizable engine where you are not giving up any performance or user-experience .
[] kostadis roussos: My point is that I completely misunderstood the centrality of that point. I didn’t get that we need to think of a game as CMS with a very optimized and specialized “rendering” system and that we needed to have the best one for any game genre or we would lose.
[ kostadis roussos: And so I pursued – html5 as a solution, flash as a solution  and then PlayScript language we built – all of these were attempts to solve the wrong problem. We should have been building the best UX system for I/E games in native languages with some C++ for the business logic bits that could be shared across multiple platforms.
[] kostadis roussos: Oh well. Lesson learned.

My only personal consolation was that I gave the prezi guys some good advice based on my learnings. They feel good about the outcome.

It’s not physics but reasoned arguments

by Leave a Comment

In a recent meeting, a hardware engineer turned and expressed his frustration at the state of a software stack. The frustration centered on the unbelievable complexity the software had accumulated over the years. There was a feeling that somehow, somewhere the software team had lost its way.

An attempt was made to make analogy to hardware. The claim was that hardware that tries to fight physics inherently fails to work. That physical laws constrain design and designs that fight those laws end up failing. And that therefore there were certain physical laws our software had violated and that as a result had become so complex.

In particular an appeal was made to modularity and strong API boundaries as the key to great software. That to not have those things, meant that we were breaking some laws of physics or their nearest software equivalent.

And it got me thinking. A lot. It was an interesting perspective. In all my discussions about software with software guys, the idea that physics would have anything to do with anything never came up.

Over the last month, I came to the conclusion that the hardware engineer was correct about the need to follow some principles but those principles were not physical laws.

All you have to do is read is this paper: Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs by John Backus to realize that software does not view physics or the underlying hardware as its master. In fact Backus calls on software practioners to create software that will force hardware people to create new kinds of hardware:

There are numerous indications that the applicative style of programming can become more powerful than the von Neumann style… when these models and their applicative languages have proved their superiority over conventional languages will we have the economic basis to develop the new kind of computer that can best implement them. Only then, perhaps, will we be able to fully utilize large-scale integrated circuits in a computer design not limited by the von Neumann bottleneck

Physics is the thing that gets in the way of our software, the thing that enables us to execute our software in the real world, but not our master.

Instead, I would argue that the laws that software ignores at it’s peril is the structure of reasoned arguments.

What do I mean? Software has the wonderful property that it’s correctness is formally unknowable. We can’t know if it is correct.  And if we can’t know, what do we do?

All we can do is reason about the software. And to reason about software is to try and understand an argument.

If that is the case, then an argument that is well crafted is:

  1. Clear
  2. Concise
  3. Structured
  4. Has a general structure based on specific facts
  5. Complete
  6. Novel
  7. Builds on previous arguments

All attributes of good software are actually the attributes of good reasoning.

As an argument acquires complexity it becomes harder to follow, harder to re-use, harder to understand, harder to leverage, harder to understand it’s correctness.

And that fact, perhaps, explains why software is so hard. Good software reflects the quality of our ability to reason and articulate our reasoning.

Things like modularity and APIs are really techniques to construct good arguments.

Our software mess wasn’t because we had failed to followed some laws of physics, it’s because we had chosen to become sloppy in our thinking and our reasoning.

What does this mean, practically?

As we try and struggle on how to make software better, what we don’t talk about is the need to improve our engineers ability to reason. Our schools focused on mechanisms and techniques spend very little time on specific training on how to argue.

Perhaps, there is something we need to learn from the law. Unclear.

How to program in C

by 14 Comments

For lots of reasons this past week I decided to look into books on the C programming language.

There are very few books that have been published in the last 10 years on C.

You’d think that given the amount of code that is still being written, the amount of code that must be supported, new books would exist.

Heck, you would expect web pages to exist.

Instead, crickets.

Or I am just doing my web searches wrong.

The challenge with C, if you’re an author of a book, is that the language is pretty simple. The libraries are also pretty simple. The complexity of the language is that, unlike almost every other language out there, C does very little to obscure or hide the underlying hardware. To program in C is to program, for better or worse, directly on the underlying hardware.

Hardware doesn’t have garbage collection, memory hierarchies exist, CPU’s have error handlers and has registers that need to be carefully programmed. Hardware has errata that makes your code break in weird ways.

There is a temptation to write a book about the C language that quickly turns into an apologia for the limitations of the language definition instead of an exploration of how and why it’s used and the value it brings.

Most texts and books that exist for other programming languages advocate a style of programming that tries to create a nerfed environment that hides the complexity of hardware. The theory of those authors and language being that the physical reality of hardware gets in the way of creating magical software that only the Turning Tar Pit constrains. Heck, Apple just released such a language called Swift to get away from Objective-C because — in many ways — it didn’t abstract the hardware enough.

There is a book crying out to be written about how to program to the hardware-software interface. A book that demystifies a lot of what I have learned through painful, bloody and miserable training.

If someone has a good book, just drop me a comment.

 

 

Caribbean Cruise Lines and the Turning Test

by Leave a Comment

When the Turning Test was first posited, we viewed that as a terrifying and beautiful milestone. Today I see that it was in fact an evil milestone: when computers are able to pass the Turing Test, computers will be used to sell empty seats on the Caribbean Cruise lines.

I got a robocall where the caller was a robot that responded to my questions about cruises before I figured out it was a machine.

2014-06-05_1138

 

The robot call didn’t immediately start spewing about cruises, instead it waited for me to respond. And then the response was contextual… Only when I remembered the stupid robot voice did I realize what was going on.

The Turing Test is an attempt to articulate when a computer is intelligent without trying to understand if a computer is intelligent. The goal was to sidestep the philosophical debates that made the engineers trying to build stuff cry.

The hope was that an intelligent computer could make better decisions than machines because it could make decisions faster with more information in the best case, and in the worst case we could have infinite free labor to free man to have more time for himself.

Except we, computer scientists, lacked vision.

Like the folks in computational photography believed that unless you could make pictures more realistic, there was no point in the technology.

ig-logo

 

I mean who wants a picture that has been made to look worse….

AI missed its true calling. The Turning Test is really about sales people calling you 24×7 to irritate you with offers you don’t want.