wrong tool

You are finite. Zathras is finite. This is wrong tool.

55 architecturalist papers: Facebook v Zynga Microchannel, the Apple Store, D&D, and killing platforms

by 2 Comments

IBM PS2 60 and 80 side-by-side

In 1981, IBM introduced a revolutionary computer that radically transformed the tech industry, the IBM-PC.

What made the IBM PC revolutionary was its open-system architecture and the royalty-free nature of its use.

Specifically, anyone could create an IBM clone, and anyone could develop software for said IBM clone and make money without paying a dime to IBM.

IBM hated that. So their solution was in 1987 to create the Microchannel PS/2. The intent was to create a new divergent market of PC’s that no one could make clones of.

What happened was that the market split between PS/2 and EISA, and the PS/2 became a failed computer system and a historical artifact and a warning to those who would want to close an open platform.

In June 2008, Apple introduced the App Store. The Apple App Store was the first massive commercial success of a cell phone app store. And it created a standard for how to take royalties from creators. And it was a huge success. So much so that we have had lawsuits that are still going through the court systems to determine the boundaries and limits of the market rules Apple can enforce in the market they created.

In June 2010, Facebook saw the success of the Apple Store and decided they wanted a piece of the Zynga action. Facebook had created an Open Platform, and that Open Platform enabled Zynga to grow like wildfire. But Facebook wasn’t making any money from Zynga. And so, there was a stand-off between Zynga and Facebook. Zynga and Facebook signed a deal that required Zynga to hand over 30% of its revenue to Facebook. That 30% revenue haircut killed Zynga.

Facebook’s thesis was that by creating a new currency, Facebook Credits, that Zynga customers would use, they would increase the total number of people who had digital money, and thus more money would be spent. The thesis failed, and Zynga suffered.

In both Apple and Facebook’s cases, they saw intellectual property creators who used their platforms as free-loaders. In Apple’s case, the tax was declared up-front, so you knew what you were getting into. In Facebook, it was an after-the-fact revision that destroyed a business.

But what happened to said intellectual property creators? As my son recently said – “why does mobile gaming suck?”

Gaming is a hit-driven business. One hit makes all the money, allowing you to make the next game. 30% is a massive tax, restricting the money you have to make further investments. And so the gaming industry has moved back to open platforms, ironically, the Windows PC.

This now brings us to the recent decision by Hasbro to introduce OGL 1.1. A lawyer covered it well here – https://medium.com/@MyLawyerFriend/lets-take-a-minute-to-talk-about-d-d-s-open-gaming-license-ogl-581312d48e2f

If you parse the Lawyer’s responses, it boils down to trying to create an Apple-like App Store for D&D content. Essentially, you hand over your financials and content for a smaller slice of the pie for the right to play.

In effect, it destroys the open ecosystem that D&D had. For example, suppose I have a website with a random generator of D&D content. That random generator is illegal.

Now Hasbro is betting that people play D&D and don’t care about the open content and that the creators will have to suck it up and deal.

Except, and this is a big exception, that isn’t true.

The iPhone was a singular technology with no ability to be replaced. Folks used Facebook because they wanted to connect with friends. Those platforms had value outside of the gaming industry. D&D is a game and a platform.

But it’s an extraordinary game where the player and the GM create content while playing. And the GM can adapt content from other gaming systems to their game. And the GM can adapt rules to their game.

In short, I expect the TTRPG community to discover the power of system-neutral gaming. And that the internet will increase with systems that allow you to convert to the gaming system of your choice. Except for the new restricted one.

My take, and it’s hopeful, is that D&D will continue, but tabletop role-playing will finally escape the long shadow of its creator and his original game.

But going back to software architecture and platforms, it’s always tempting to control a market, and there is a lot of value in doing that. But when you extract a lot of money from a market, you eventually kill the market. And over time, those creators whose businesses are hits will move to open platforms.

Open Facebook API or what to do about Facebook

by Leave a Comment

When I left Zynga in 2013, I was convinced that Facebook was a malevolent entity run by leaders who could not be trusted. But I was also bitter about a 6$ stock price and my life choices.

Fast-forward to 2019, and it turns out that what I thought was just sour grapes, undersold the net harm Facebook has created.

An option that isn’t considered very seriously is the following simple proposal. Don’t break up Facebook, but regulate the access to and control of the friend graph and the ability to use the friend graph to publish information.

In 2012, when Facebook and Zynga stood off, the debate that was at the heart of the disagreement was ownership of the friend graph. Facebook believe they owned the friend graph and by extension owned how it could be used. We disagreed. In the end, we caved. I know this because I worked on the software systems necessary to create a parallel friend graph of people who were friends with other people who played Zynga games.

Facebook would love for us to spend time talking about breaking things up, instead of talking about the one thing that matters, a regulated open-api and regulated data portability.

Consider the messenger space. Because the friend graph is in my personal address book, it’s trivial to talk to several dozen different friends. Because the content is on my phone, typically pictures or documents, I can share anything with anyone.

Consider how many more messenger apps there are, versus how many social networks there are.

But let’s look to the past. During the failed MSFT anti-trust trial, a peculiar part of the agreement said that MSFT could no longer have private APIs, and that they had to communicate changes in a very specific public way.

This ruling enabled NetApp, which had built a reverse engineered CIFS server to survive and thrive. Because MSFT was losing the CIFS business, it also pushed MSFT to look for alternatives to CIFS, like SharePoint for document sharing and collaboration.

But over the long term, it enabled companies like Box and Google Drive and other file-sharing companies to emerge. Without the guarantee that a single man couldn’t break an API, a healthy and vibrant ecosystem in data storage has emerged.

If we had an open-social graph, and an open api, and data portability then I suspect that over time new social networks would emerge. Every social network would probably cater to different kinds of people.

In many ways Facebook does this today with Facebook Groups. For example, I happen to have joined two Facebook groups, one dedicated to old-school rpg, and another to 5E. The two groups hate each other. But because my social graph is portable, I can communicate to both groups within facebook.

Or we can even go back to Facebook’s origins. When Mr. Zuckerberg opened up the API, he promised it was going to be open and portable. He lied, of course, but not before Mark Pincus and Zynga figured out how to exploit the graph to grow Facebook’s business. Once, Mr. Zuckerberg figured out that owning the graph and how you communicate with it was very valuable, he squashed us like a bug. And destroyed the Facebook app eco-system.

Which brings me to regulation, we can’t trust Mr. Zuckerberg . Like we couldn’t trust Mr. Gates. And breakups don’t always work. Look at ATT, 40 years after the breakup, they control everything, again.

The problem with Facebook and Machine Learning

by Leave a Comment

At Zynga, Facebook and us got into a game of cat and mouse. Facebook would tweak it’s algorithms and rules to limit our reach, and we would work around it.

Facebook never could win. We had 2000 people working to figure out how to work around their algorithms, and they had a few dozen.

They only won when they just put limiters on Zynga and our APIs. And they won when they forced us to sign a contract or go out of business.

This taught me a powerful lesson, Machine Learning and it’s derivatives, are terrible tools when the cost of a mistake is non-zero.

For Facebook, when the cost of a mistake was non-zero, they didn’t rely on machine learning, they relied on regulations and laws.

But when a Facebook post goes viral about the evils of vaccination, when a Facebook post goes viral about how Clinton murdered children, this is to Facebook’s benefit. It drives engagement, it drives advertising revenue, it accrues tremendous benefit to them.

Over the past few years smart folks, friends of mine even, have tried and are trying to do something about this. But at the end of the day, as long as the damage to society is acceptable to society, Facebook has no incentive to do anything.

Because the cost of mistake is zero to Facebook. No one at Facebook or it’s shareholders is foolish enough to not get a vaccination. No one at Facebook is in danger of being physically assaulted on the streets of New York for being LGBTQA+ (oh wait, some of them are, but I digress).

When Facebook exec’s say that nothing can be done, they are lying. When the cost of mistakes was non-zero to them, they discovered the power of regulation. And they embraced it.

19 architecturalist papers: why doing the right thing matters, a tale of Facebook and charities.

by Leave a Comment

When I was at Zynga, Mark Pincus and the executive team had this brilliant idea on how to raise money for charity, selling virtual goods.

The idea was pretty simple, they had a virtual good, that virtual good was relevant to the game, and if you used real money, we gave a portion of the money to some charity.

This technique generated a lot of money for charities. And, to be fair, it was great for Zynga as well. Even if we did not keep the money, getting people to spend on a free game was hard, but once you got them to pay, it was straightforward to get them to pay more.

But we had to stop.

Why?

Facebook Credits.

See Facebook and Zynga signed a deal to have Zynga use Facebook Credits instead of real dollars. Feels a lot like Libra, but I am bitter. And because we used Facebook Credits, we needed to get them to do some back-office paperwork.

So I got the foundation to agree to do anything and everything that Facebook needed.

And they said, no.

I said that I would write a blog raking them over the coals for not prioritizing incremental revenue over doing good.

And they said, “Do it, we do not care.”

So I worked with our MarComm team to put something together. And we had layoffs, and our business was imploding and they asked me to not post it. They had so many other fires to put out, that this felt over the top.

And I agreed.

And I was wrong to agree.

Because, since then, no one has done this. Not one single freemium game has done this. Nada. Not one.

At Zynga, we pioneered a lot of the pay-to-play game mechanics. But Facebook’s payment team of the time pioneered the idea that charity was not a business priority.

It’s my fault for not having a spine six years ago. I wonder if I wrote that blog, things would be different. How many people would be alive if I had just done what was right?

When Facebook started it’s “charitable” giving on their timeline, I puked. I got so angry that I donated 1000$ to Mother Jones because they were the only publication that was willing to call out Facebook. Heck, I offered to give another 500$ as a matching donation. No one from Mother Jones asked, I just did it. I went on twitter and said if people sent me a note with a proof of a donation, I would donate 500$ to Mother Jones; I was that angry. And while we are here, give to Mother Jones, they are an excellent liberal paper that fights the power.

I screwed up.

So why am I writing now? Because a friend of mine saw a freemium game that did something for charity, and it made me happy. It meant that some games were trying to do the right thing again.

The Elder Scrolls Online 

@TESOnline

Thousands of Dragons have been slain since Elsweyr released – but now you can continue defeating them for a good cause! Raise money for real-world charities that support pets in need with each Dragon kill in #ESO. beth.games/2oVobFW #SlayDragonsSaveCats

And I also wanted to remind everyone that there are consequences to not doing the right thing. I get angry when I see folks ask how do we incentivize tech companies to do the right thing. We should be asking them what kind of moral bankruptcy exists that says the right thing to do isn’t something you do? But I didn’t do the right thing. And the industry is different as a result. And worse, a lot of people are not better off because I didn’t bother to write that blog.

As software architects, we make choices, and we are accountable for those choices.

Facebook hitting a billion people – is this the day that open messaging died?

by Leave a Comment

In the 1980’s visionary technologists created e-mail as an open non-proprietary messaging system. This allowed anyone to communicate with anyone on open networks.

With Facebook hitting a billion, and WhatsApp hitting 900 million people, we now have a new proprietary network that has the reach that email does.

In the open messaging world, messages were owned by the sender, and the recipient and were portable. Your social network – the set of people you interact with – and your chats were owned by the people who created them.

Facebook and Whatsapp have now up-ended that open communication channel. They own your social network and they own your messages and they have the reach to displace open communications. A private entity owns your friends and the relationship to your friends.

And hence snapchat and wickr. If there is no portability and durability that is independent of the service provider, then you may as well treat the messages as ephemeral.

Who would have thought, my most private and important data would be owned by someone else.

Oh brave new world!

Looking back on “Here Comes Another Bubble”

by 1 Comment

In 2007 the Richter Scales captured the zeitgeist of the Silicon Valley bubble atmosphere with this hysterical video:

The irony about this song was that the 2007 bubble was killed by Lehman’s. A lesson in mass hysteria.

And as I was re-watching it last night I wondered – so what happened with those scrolling startups? And was Peter Thiel right or wrong?

bubble-3

When it comes to Facebook, he was probably right. In fact, the valuation of Facebook is staggering and an amazing achievement for the folks at that company!

That got me wondering about the rest of the companies … The hype around MySpace wasn’t probably justified:

myspace

And then if you look at the scrolling startups, I tried to circle the ones I knew were still in business. Admittedly I didn’t look through every single one — there are a lot but was struck by how few of them are around:

bubble-2 bubble

Starting a company is hard. Creating investor value is hard.

And sometimes it’s worth remembering that for ever Facebook there is a scrolling morass of companies that go nowhere.

And like the song says, those folks are probably back at work looking to start another new thing…

 

 

How Firewalls Killed Zynga or Why the Mobile Internet Took Off in 2012

by 7 Comments

In life, when confronted with a mystery I always use this principle:

Occam’s conspiracy razor: Never ascribe to malice when stupidity will suffice. 

In 2012, Zynga experienced a rapid decline in numbers while at the same time mobile gaming and mobile traffic took off. The expectation that mobile would take over wasn’t a surprise. We all knew it was going to happen, what was surprising was how fast web based browsing dropped.

Mobile was going to be big, but why was it at the price of web? Why did the web die out?

Occam’s conspiracy razor was that people just wanted to move to mobile, and that 2012 was the year it happened. There was no conspiracy, no external event, just a mass migratory movement.

Early 2012

In early 2012, as I looked over the landscape of mobile internet, my reaction was that this was going to be a platform that added more people, not took away from the web. Adults would both  use the mobile device and the desktop to do all of their activities. Most people, sitting in front of computers during working hours, would play their games, do their Facebooking, do their shopping on their computers while working. Later in the day, when they went home they would use their mobile devices on the couch while they hung out with their families and friends.

That. Did. Not. Happen.

At the end of 2012, we could confidently call the death of the desktop web. Desktop DAU for Zynga cratered, and Facebook web DAU had flatlined.

What happened?

What happened was that while Zynga and Facebook were creating the ultimate form of escapist fun for the masses, corporate America noticed.

And what they noticed was that web-surfing that had been an annoying tax on employee productivity was becoming a massive time sync. Employees were playing their Zynga games, sharing on Facebook with their family and friends, instead of doing theirs jobs and incidentally consuming vast amounts of bandwidth. Later Netflix entered the picture and the amount of lost productivity and bandwidth was getting serious.

Before Facebook, social non-professional interaction was hidden in corporate and private email or the phone. The water cooler, the smoke break, the lunch room, the break room where we hid minutes of wasted time every day.

Before Zynga, it was really hard to play a game on a corporate laptop. The security teams locked computers down so tight, that nothing could be installed. And to be honest, you’re not going to take a 5 minute break to play some Call of Duty. Most employees played Minesweeper because that was the only game they could install.

So lost productivity was visible but unblockable.

Enter Zynga. Farmville at it’s peak had 30 million DAU. That’s an insane number for a game. 30 million people were not playing at home. Peak DAU hours were during the working day.  Corporate America noticed.

At the same time as Facebook and Zynga were taking over the world, ng-fw‘s (next generation firewalls) came into existence. Their claim to fame was that they could identify the applications and then apply security policies at the application layer.

Apparently, as I recently discovered, for the last several years the basic pitch of an ng-fw begins with:

How do you stop your employees from playing Farmville?

Obviously it’s more nuanced … Not really. I mean sales guys as recently as last week positioned ng-fw as how to block people from playing Farmville. The thing was that corporate america figured out how to control what applications their employees were using, even when they were on the web.

Managers wanted to stop their employees from goofing off, and since employees were using communication channels that the managers could choke, management did.

What about SSL? Well it turns out that the firewall vendors figured that out as well. The firewalls would terminate the SSL connection to the external site, and then in turn re-sign the certificate. The IT teams would helpfully disable the warnings associated with the re-signed certificate making people oblivious to the fact that their traffic was being man-in-the-middled.

Net effect, throughout corporate America, firewalls were quietly and silently blocking access to consumer web-sites that employers felt were not really related to work.

So what does this have to do with Zynga?

At the same time, at Zynga I was observing this really weird unexplained phenomenon.I was  seeing evidence of people trying to start our games and failing. Stuck in the cauldron of Zynga, we had no insight into what was going on.

We assumed, applying Occam’s razor, that people were starting to load the game and then quitting. The sheer scale of attempts was mystifying, but we assumed that people had their reasons.

I mean, how else could games be blocked?

Last week I went to Hawaii and experienced ng-fw first hand.

1. Sometimes I couldn’t load the game.

2. Sometimes I could load the game but then the connection would be dropped.

And I realized that our customers had been blocked from accessing our games.

And then it clicked. All of those mysterious reports, all of those users who couldn’t get to our games, all of those bizarre late night sessions trying to understand what the hell was going on, and we missed the most obvious thing of all:

Employers didn’t think their employees should be playing games at work and were doing something about it.

So why was this bad for Zynga?

Let me caveat the next bit with the following: Mark Pincus never accepted the idea that Zynga was a victim. We didn’t fail because other people did things to us, we failed because we didn’t execute because we didn’t deliver. And I agree. If we failed, it’s because we failed not because other people screwed us.

Let me also caveat, that this is pure speculation. I don’t have any data to back any of this up. At the time I didn’t know what to collect, and now I don’t have access. 

DAU and Engagement

Spend enough time on the Facebook platform, and you know that the only thing that matters is engagement. Facebook wants people to be engaged with their product, they’ll flood you with users to see if they’ll stick, but if you don’t keep those users be afraid, because eventually they’ll point those users somewhere else.

Firewalls were breaking Zynga’s engagement in two ways.

The first was that users were unable to get to the game. With games that had a plant-harvest cycle, if you can’t make it into the game to harvest, then you’re likely to quickly give up with the game itself.

The second was that users were conditioned to stop clicking on our feeds and virals. If clicking on the feed or viral would result in nothing because of firewalls, people stopped clicking on them.

As a result Facebook was seeing a decline in engagement in Zynga games. And because they care about their users, first and foremost, they started to point their users away from our games. And good for them.

Unfortunately this fed a vicious cycle of decline. The more our users left, the more we tried to reach them through virals, the more Facebook users got annoyed with those virals, the more Facebook throttled our virals. And because Facebook was always looking for some other content they could get people engaged with, Facebook pointed the firehose of users elsewhere.

Enter mobile

If there was no mobile platform, I posit Facebook would have seen a flatlining of users as did Zynga. Instead what happened was that people discovered the one device that had a network their corporate bosses could not control, their mobile phone.

With their access to Facebook blocked, employees discovered what teenagers have known since forever, mobile phones are the only way to talk to your friends without anyone getting in the way.

Much like teenagers who used SMS as a cheap way to talk forever with their friends, adults reached out to the mobile device as a cheap way to keep connected once their internet access got blocked.

And mobile Facebook traffic skyrocketed.

And once everyone got used to using their mobile device instead of their corporate laptop to connect with the Internet, the rest was history.

The transition was inevitable, and firewalls forced the transition to happen ridiculously fast in 2012.

Some final thoughts

When I look back at my own mistakes, what I missed about mobile was not the form factor but the access to the users.

At the end of the day, consumer websites live and die by their ability to reach end-users. And unfortunately, the last mile is controlled by corporate entities that may have a dim view of people goofing off on the job.

Mobile is the future because it represents the only reliable way for consumer businesses, net-neutrality excepting, to reach their customers without some intermediary blocking them.

What I missed at Zynga wasn’t mobile, it was the fact that only on the mobile platform could we be guaranteed access to our users.