46 architecturalist papers: the hybrid cloud

In 2013, after being pushed out of my role at Zynga, I was noodling on what to do next. And was thinking about working on hybrid cloud technologies, but I figured that with everything Zynga had achieved, this was an area that was well-trodden and not one where a new startup was welcome.

Wrong.

So let me rewind the clock.

Here’s what I had learned at Zynga.

1. The public cloud is costly, and a well-run private cloud can save you tons of cash when you are short on cash.
2. The public cloud is a very efficient way to get something started.

In many ways, it’s the basic tension between using a prototyping environment or language like python versus a more statically typed language like C++.

And that creates a fundamental dilemma, how do you get to the private cloud economics on a public cloud?

Let me start by saying the non-obvious. In 2013, running on the cloud was 1/3 the cost of running AWS with everything priced in.

If Zynga had tried in 2013 when our business was imploding to get to a private cloud, we would have never done it. But in 2009, our prescient CTO Cadir Lee made the strategic bet that we needed to have our own cloud. And by 2012, we had made the transition, allowing us in 2013 to manage our cash very effectively and survive a very long downturn without bleeding cash.

It’s important to realize that in 2011, I was in charge of what we would call DevOps / SRE / AppOps. And I loved AWS. And I was not interested in moving to zCloud. And then we launched CityVille. And the game was crashing, and we had to increase our server count by a factor of three. And when we begged for insight from Amazon, we learned that the problem was that the AWS team had switched NIC’s that had fewer IO buffers, which forced us to get more x86 servers to handle the IO load. That decision ate into our margins decisively, consumed lots of engineering cycles, and convinced me that we had to get off AWS ASAP.

I figured that this knowledge was so obvious that somebody somewhere was working on it as we spoke. And so, I didn’t bother looking for employment in that area.

The other thing I learned at Zynga was about the mobile business and software stack.

I learned at Zynga that vendors like Apple and Google make a lot of money adding software layers on top of Arm instructions to make it hard to write x-platform code.

Games are weird. They are very heavy in terms of graphics and very light in terms of UI. And so, a game can benefit significantly from being written in an x-platform toolkit because most of the time, you are playing on the game board.

And so, I pursued this strategy of trying to enable x-platform development, even to the point where we created our own programming language, playscript.

But it turns out that the x-platform strategy was a losing strategy because if the UI wasn’t native, the experience was marginally worse than some other game. The liquidity of games was so high that it made much sense to use those native features.

And so, the x-platform game strategy wasn’t quite as successful as I would have liked.

But I also learned that whereas with the UX, native integration matters a lot, the value of deep native integration with the back-end systems isn’t.

Customers do not use the back-end systems. Engineers use them. And those engineers – especially the software variety – are surprisingly willing to rebuild or change things.

And a game could be easily transferred from AWS to an on-prem data center if you just organized your system correctly and were prescient.

So how did the system have to be organized?

From my mobile experience, I realized that at the end of the day – every time you used something other than an x86 server, you were paying for someone else’s retirement. And every time you used some service, you were taking a dependency on someone else’s business plan.

Many companies tried to build meta-management layers that spanned the cloud and were a layer above the cloud API. I viewed that as a mistake. The only sane strategy was to have the same x86 server running in the cloud as you had on-prem.

And the only way to achieve that was to use the same kind of hypervisor.

At Zynga, we deliberately chose to run the same kind of Xen Hypervisor that AWS did for that reason. |

When I left Zynga, I was determined to make things like zCloud easy to build. And through a series of convoluted decisions, I ended up at VMware.

Now Kit Colbert talks a lot about multi-cloud in this post, which I urge you to read it.

My personal spin is that ultimately having a common layer that you can reliably deploy anywhere you want is critical. And the companies that last are the ones that can move their workloads and take advantage of the multi-cloud.

In 2013, I had not realized how special the Zynga team was. The guy who built our data centers was Allan Leinwand (currently SVP at Slack). The guy who built our storage system layer was the chief architect at Flipkart, Vinay YS. A key player on the infrastructure side of the house was Nick Tornow (VP of Engineering at Twitter).

The talent we had was staggering.

What VMware is doing is making the ability to run in multiple clouds something that can be done by small teams easily.

It’s a journey and a long one, but one that I am excited about working on.