wrong tool

You are finite. Zathras is finite. This is wrong tool.

55 architecturalist papers: Facebook v Zynga Microchannel, the Apple Store, D&D, and killing platforms

by 2 Comments

IBM PS2 60 and 80 side-by-side

In 1981, IBM introduced a revolutionary computer that radically transformed the tech industry, the IBM-PC.

What made the IBM PC revolutionary was its open-system architecture and the royalty-free nature of its use.

Specifically, anyone could create an IBM clone, and anyone could develop software for said IBM clone and make money without paying a dime to IBM.

IBM hated that. So their solution was in 1987 to create the Microchannel PS/2. The intent was to create a new divergent market of PC’s that no one could make clones of.

What happened was that the market split between PS/2 and EISA, and the PS/2 became a failed computer system and a historical artifact and a warning to those who would want to close an open platform.

In June 2008, Apple introduced the App Store. The Apple App Store was the first massive commercial success of a cell phone app store. And it created a standard for how to take royalties from creators. And it was a huge success. So much so that we have had lawsuits that are still going through the court systems to determine the boundaries and limits of the market rules Apple can enforce in the market they created.

In June 2010, Facebook saw the success of the Apple Store and decided they wanted a piece of the Zynga action. Facebook had created an Open Platform, and that Open Platform enabled Zynga to grow like wildfire. But Facebook wasn’t making any money from Zynga. And so, there was a stand-off between Zynga and Facebook. Zynga and Facebook signed a deal that required Zynga to hand over 30% of its revenue to Facebook. That 30% revenue haircut killed Zynga.

Facebook’s thesis was that by creating a new currency, Facebook Credits, that Zynga customers would use, they would increase the total number of people who had digital money, and thus more money would be spent. The thesis failed, and Zynga suffered.

In both Apple and Facebook’s cases, they saw intellectual property creators who used their platforms as free-loaders. In Apple’s case, the tax was declared up-front, so you knew what you were getting into. In Facebook, it was an after-the-fact revision that destroyed a business.

But what happened to said intellectual property creators? As my son recently said – “why does mobile gaming suck?”

Gaming is a hit-driven business. One hit makes all the money, allowing you to make the next game. 30% is a massive tax, restricting the money you have to make further investments. And so the gaming industry has moved back to open platforms, ironically, the Windows PC.

This now brings us to the recent decision by Hasbro to introduce OGL 1.1. A lawyer covered it well here – https://medium.com/@MyLawyerFriend/lets-take-a-minute-to-talk-about-d-d-s-open-gaming-license-ogl-581312d48e2f

If you parse the Lawyer’s responses, it boils down to trying to create an Apple-like App Store for D&D content. Essentially, you hand over your financials and content for a smaller slice of the pie for the right to play.

In effect, it destroys the open ecosystem that D&D had. For example, suppose I have a website with a random generator of D&D content. That random generator is illegal.

Now Hasbro is betting that people play D&D and don’t care about the open content and that the creators will have to suck it up and deal.

Except, and this is a big exception, that isn’t true.

The iPhone was a singular technology with no ability to be replaced. Folks used Facebook because they wanted to connect with friends. Those platforms had value outside of the gaming industry. D&D is a game and a platform.

But it’s an extraordinary game where the player and the GM create content while playing. And the GM can adapt content from other gaming systems to their game. And the GM can adapt rules to their game.

In short, I expect the TTRPG community to discover the power of system-neutral gaming. And that the internet will increase with systems that allow you to convert to the gaming system of your choice. Except for the new restricted one.

My take, and it’s hopeful, is that D&D will continue, but tabletop role-playing will finally escape the long shadow of its creator and his original game.

But going back to software architecture and platforms, it’s always tempting to control a market, and there is a lot of value in doing that. But when you extract a lot of money from a market, you eventually kill the market. And over time, those creators whose businesses are hits will move to open platforms.

44 architecturalist papers: the value of a college degree

by Leave a Comment

Over the last twenty years, I have been the least impressed with the value of formal education in the field of computer science for most practitioners who do most of the work.

As someone who struggled to do well in exams and avoided classes with exams, I never understood what they measured. I know that I never applied to Google because they had this ridiculous requirement to see my GPA. I graduated with honors and summa cum-laude. It wasn’t my grades; it was the principal.

Maybe it was being bullied. As a weird, obnoxious Greek, my experience at Brown was toxic. I didn’t fit in. And all I wanted was to put as much physical distance with that part of my life as I could. And maybe the experience taught me that maybe this degree wasn’t as valuable as I thought. And maybe I thought, if everyone at top-colleges were like some of the specific people I had to deal with, then I would prefer to never work with them again. And the easiest thing to do was never to go work where they worked.

Over the years, I had no issues with hiring or promoting people on my team with no CS degree. At Zynga, we did real paper facebook of every senior leader with no CS degree, which was illuminating. I do remember that the guy who was the architect for cafe-world had no degree in CS.

Schools teach you the wrong skills, abstract concepts that are of meandering value, and most importantly, the wrong interpersonal values. Production software is a team sport. School is an individual sport. Production software is about maintaining software over time, not hitting a deadline. Production software is about customers and business requirements, not some contrived technical problem that some poor Professor invented to grade your attention span in a class. Production software is about the wisdom of how broken something can be.

In fact, I will observe my high-school history class, and the value it placed in critically understanding the nature of information and sources has proven to be far more valuable than any CS class I ever took. I would trade any CS class for that one.

But I *never* dismissed a top college’s most important value to a student. In fact, while simultaneously shitting on the value of a CS degree, I would tell folks, “a CS degree at a top-school has extraordinary value for your first set of jobs that set you up for your next set.”

If you want to get hired, out-of-school, graduating from a top school is orders of magnitude more valuable than anything else. And then, because the next set of jobs is a function of your relationships that you build while you work, the next set of jobs. I met an Israeli sales guy, and he told me the same thing. He was infuriated that he never got a job at a top-tech company because he felt he didn’t have the right degrees from the right school

I graduated from Brown University with a degree in CS, where I barely understood algorithmic analysis (I still don’t get how to do induction). I didn’t know what a database was. I didn’t know what a compiler was.

But I got a job at SGI in the kernel group because Forest Baskett’s daughter was a student at Brown. Forest Baskett was the then CTO of SGI, and he interviewed students at Brown as part of a recruiting project.

That had more to do with anything.

At SGI and later NetApp, I got a part-time Master’s degree at great personal expense and filled in gaps in my understanding of the field. But I did that to get a military deferral. I was wealthy enough to do that.

Having said all of that, I think learning and a life-long love of learning are crucial to your personal happiness and success. And a college education, if you can afford it, can help in that way, maybe. And I do believe learning to analyze how people talk and think and learn critically is valuable. And if you need to learn some boutique knowledge and a college setting works for you, by all means, take a class.

But the real reason a top-college degree is valuable? Because recruiters go there. That’s all.

Open Facebook API or what to do about Facebook

by Leave a Comment

When I left Zynga in 2013, I was convinced that Facebook was a malevolent entity run by leaders who could not be trusted. But I was also bitter about a 6$ stock price and my life choices.

Fast-forward to 2019, and it turns out that what I thought was just sour grapes, undersold the net harm Facebook has created.

An option that isn’t considered very seriously is the following simple proposal. Don’t break up Facebook, but regulate the access to and control of the friend graph and the ability to use the friend graph to publish information.

In 2012, when Facebook and Zynga stood off, the debate that was at the heart of the disagreement was ownership of the friend graph. Facebook believe they owned the friend graph and by extension owned how it could be used. We disagreed. In the end, we caved. I know this because I worked on the software systems necessary to create a parallel friend graph of people who were friends with other people who played Zynga games.

Facebook would love for us to spend time talking about breaking things up, instead of talking about the one thing that matters, a regulated open-api and regulated data portability.

Consider the messenger space. Because the friend graph is in my personal address book, it’s trivial to talk to several dozen different friends. Because the content is on my phone, typically pictures or documents, I can share anything with anyone.

Consider how many more messenger apps there are, versus how many social networks there are.

But let’s look to the past. During the failed MSFT anti-trust trial, a peculiar part of the agreement said that MSFT could no longer have private APIs, and that they had to communicate changes in a very specific public way.

This ruling enabled NetApp, which had built a reverse engineered CIFS server to survive and thrive. Because MSFT was losing the CIFS business, it also pushed MSFT to look for alternatives to CIFS, like SharePoint for document sharing and collaboration.

But over the long term, it enabled companies like Box and Google Drive and other file-sharing companies to emerge. Without the guarantee that a single man couldn’t break an API, a healthy and vibrant ecosystem in data storage has emerged.

If we had an open-social graph, and an open api, and data portability then I suspect that over time new social networks would emerge. Every social network would probably cater to different kinds of people.

In many ways Facebook does this today with Facebook Groups. For example, I happen to have joined two Facebook groups, one dedicated to old-school rpg, and another to 5E. The two groups hate each other. But because my social graph is portable, I can communicate to both groups within facebook.

Or we can even go back to Facebook’s origins. When Mr. Zuckerberg opened up the API, he promised it was going to be open and portable. He lied, of course, but not before Mark Pincus and Zynga figured out how to exploit the graph to grow Facebook’s business. Once, Mr. Zuckerberg figured out that owning the graph and how you communicate with it was very valuable, he squashed us like a bug. And destroyed the Facebook app eco-system.

Which brings me to regulation, we can’t trust Mr. Zuckerberg . Like we couldn’t trust Mr. Gates. And breakups don’t always work. Look at ATT, 40 years after the breakup, they control everything, again.

The problem with Facebook and Machine Learning

by Leave a Comment

At Zynga, Facebook and us got into a game of cat and mouse. Facebook would tweak it’s algorithms and rules to limit our reach, and we would work around it.

Facebook never could win. We had 2000 people working to figure out how to work around their algorithms, and they had a few dozen.

They only won when they just put limiters on Zynga and our APIs. And they won when they forced us to sign a contract or go out of business.

This taught me a powerful lesson, Machine Learning and it’s derivatives, are terrible tools when the cost of a mistake is non-zero.

For Facebook, when the cost of a mistake was non-zero, they didn’t rely on machine learning, they relied on regulations and laws.

But when a Facebook post goes viral about the evils of vaccination, when a Facebook post goes viral about how Clinton murdered children, this is to Facebook’s benefit. It drives engagement, it drives advertising revenue, it accrues tremendous benefit to them.

Over the past few years smart folks, friends of mine even, have tried and are trying to do something about this. But at the end of the day, as long as the damage to society is acceptable to society, Facebook has no incentive to do anything.

Because the cost of mistake is zero to Facebook. No one at Facebook or it’s shareholders is foolish enough to not get a vaccination. No one at Facebook is in danger of being physically assaulted on the streets of New York for being LGBTQA+ (oh wait, some of them are, but I digress).

When Facebook exec’s say that nothing can be done, they are lying. When the cost of mistakes was non-zero to them, they discovered the power of regulation. And they embraced it.

19 architecturalist papers: why doing the right thing matters, a tale of Facebook and charities.

by Leave a Comment

When I was at Zynga, Mark Pincus and the executive team had this brilliant idea on how to raise money for charity, selling virtual goods.

The idea was pretty simple, they had a virtual good, that virtual good was relevant to the game, and if you used real money, we gave a portion of the money to some charity.

This technique generated a lot of money for charities. And, to be fair, it was great for Zynga as well. Even if we did not keep the money, getting people to spend on a free game was hard, but once you got them to pay, it was straightforward to get them to pay more.

But we had to stop.

Why?

Facebook Credits.

See Facebook and Zynga signed a deal to have Zynga use Facebook Credits instead of real dollars. Feels a lot like Libra, but I am bitter. And because we used Facebook Credits, we needed to get them to do some back-office paperwork.

So I got the foundation to agree to do anything and everything that Facebook needed.

And they said, no.

I said that I would write a blog raking them over the coals for not prioritizing incremental revenue over doing good.

And they said, “Do it, we do not care.”

So I worked with our MarComm team to put something together. And we had layoffs, and our business was imploding and they asked me to not post it. They had so many other fires to put out, that this felt over the top.

And I agreed.

And I was wrong to agree.

Because, since then, no one has done this. Not one single freemium game has done this. Nada. Not one.

At Zynga, we pioneered a lot of the pay-to-play game mechanics. But Facebook’s payment team of the time pioneered the idea that charity was not a business priority.

It’s my fault for not having a spine six years ago. I wonder if I wrote that blog, things would be different. How many people would be alive if I had just done what was right?

When Facebook started it’s “charitable” giving on their timeline, I puked. I got so angry that I donated 1000$ to Mother Jones because they were the only publication that was willing to call out Facebook. Heck, I offered to give another 500$ as a matching donation. No one from Mother Jones asked, I just did it. I went on twitter and said if people sent me a note with a proof of a donation, I would donate 500$ to Mother Jones; I was that angry. And while we are here, give to Mother Jones, they are an excellent liberal paper that fights the power.

I screwed up.

So why am I writing now? Because a friend of mine saw a freemium game that did something for charity, and it made me happy. It meant that some games were trying to do the right thing again.

The Elder Scrolls Online 

@TESOnline

Thousands of Dragons have been slain since Elsweyr released – but now you can continue defeating them for a good cause! Raise money for real-world charities that support pets in need with each Dragon kill in #ESO. beth.games/2oVobFW #SlayDragonsSaveCats

And I also wanted to remind everyone that there are consequences to not doing the right thing. I get angry when I see folks ask how do we incentivize tech companies to do the right thing. We should be asking them what kind of moral bankruptcy exists that says the right thing to do isn’t something you do? But I didn’t do the right thing. And the industry is different as a result. And worse, a lot of people are not better off because I didn’t bother to write that blog.

As software architects, we make choices, and we are accountable for those choices.

Engagement and Retention Do Not Move and the limits of Big Data

by 3 Comments

When I was at Zynga, we shipped a game that had marginal success. And in the deep dive the product management lunch lead said:

Engagement is really tough to move

At Zynga, Mark would demand game-changing features, demanding that we change the product in place and if we could pull that off the theory was we could change the retention curve.

And being an engineer, and surrounded by entrepreneurs, my assumption was that through the application big data and science we could change this number.

Then I had the opportunity to try and drive machine learning models into games to improve core metrics of the game.

The theory was that auto-tuning the game would improve engagement and retention.

And it worked, to a point.

And what I realized was that the hard problem is building a fabulous product. And a great product has high engagement. Everything else we do is about tuning or improving the excellent product at the fringes. And that changing engagement is equivalent to creating a new product.

And that got me thinking as to why that is impossible. And what I realized is that big data collects information about the product that is. And can only answer questions about what your product is doing.

To change engagement, you have to build a new product with new features and net new capabilities. And the data for that product doesn’t exist in any of your big data systems.

The short version of this story is the following, engagement is what it is, and if it isn’t what it needs to be, you need to scrap the damn thing and start all over.

 

x-platform mobile technologies

by Leave a Comment

The folks at Google wrote about their new toolkit  for their new mail app. An app, by the way, that is actually great.

I was convinced – from using the app – that they had a lot of platform specific code. Instead, being great engineers, they cracked a hard nut – how do you build a UI rich application without writing most of the code twice that I didn’t think was going to get cracked anytime soon.

The challenge with UI rich products is that they must interact with the native software interface of the device. And the native interface is very different and is written in very different programming langues.

What Google has done is very interesting. My stock recommendation for UI rich applications  is that you have a core that is in C++ and a bunch of platform specific code for each device, the approach Google has taken may indicate a new third way. The Prezi guys did this to great effect. And many of my former Zynga colleagues are doing the same.

I must dig in some more…

 

Some performance benchmarks of browser languages

by Leave a Comment

Read this really interesting performance analysis of various browser programming languages

Two things that come out of this.

The first is the power of meta-programming languages like Haxe. Using a better programming language and a typeless goop as your target machine-language can deliver great performance. Not entirely surprising – of course – because hardware is un-typed 🙂

I first encountered Haxe four years ago as an answer to my prayers around the mess of Php and Javascript inflicting typeless chaos on the world of web programming.

We actually used Haxe at Zynga to write a renderer if memory serves me write and got better performance than we could have if we had written it in entirely in Flash.

The second is how underwhelming the performance of emscrimpten is compared to Haxe when you don’t have asm.js. There was a time before the mobile web died that I though that C++ to JS would be important. Now that it’s clear that the mobile web is deader than a door knob, the value of Javascript as a front-end mobile programming language is trending to zero fast.

 

 

Fascinating Account on Recent Gaming History

by Leave a Comment

I had the chance to stumble on this answer on quora that discussed in long detail how computer gaming had metastasized from a minority group activity to a majority group activity and the social implications there in.

The summation can be found here:

[..] appeals to people who feel alienated by the changing face of gaming, people who feel criticized when they’ve been the minority, people who want to keep gaming the way it was, people who are already prone to assuming conspiracies, and people who feel as if they’re being disenfranchised by the changes in society being carried out in gaming. It has been timely, in the sense that it is happening during a particularly pessimistic period in game journalism (see all the “gaming is dead” articles) and during a period where there’s an active series of cultural debates occurring on the role of gaming in culture.

Working at Zynga in the period 2009->2011 and seeing the abrupt transformation of gaming to a mainstream activity was disorienting. Making games that everyone played was not the experience I had with games. Games were a niche activity that some people did, not everyone. .

Many people who were involved in gaming hated Zynga because we were building games they didn’t like. I am not a game desiger, and I am not an expert in the art of gaming and I do know Mark Skaggs (FarmVille and CityVille and Red Alert) and Brian Reynolds (FrontierVille and CivII) and they built some products that millions loved.

And what they showed was that there was this vast untapped market desire for games that was unanticipated.

For a while, many folks in the industry looked at the games we built and said – these are not games. And then some folks people looked at the games we built, and said: I can do better. And much better games that targeted the markets Zynga had shown existed emerged.

The wold of gaming is very different from the world I grew up in. And that’s a good thing …

Grand Moff Tarkin Didn’t Want to Pay for Defense in Depth

by Leave a Comment

2014-09-07_1703

In Episode IV of Star Wars, Han Solo, Luke Skywalker, Obi-Wan, and Chewbacca are trapped on the Death Star after their jump from hyperspace.

The Storm Troopers are quickly overwhelmed, and our heroes can access a physical terminal. R2D2 is then able to plug into the computer systems of the Death Star.

R2D2 can quickly access all of the information, including schematics and prisoner information.

In the post-mortem on Coruscant, I can imagine the dialogue:

Palpatine: How were they able to access all of the information?

CISO for the fleet: Well, the Grand Moff decided that the cost of adding firewalls and security systems to partition the network was too costly. He chose to rely on a big ass external firewall. His priority was the ability for his teams to access the information not to protect it.

Palpatine: A single droid was able to quickly and trivially get all of our operational information … Because we had no firewall?

CISO for the fleet: visibly sweating Well it’s more complicated than that. A firewall would have delayed the attack, and at the very least made it harder but nothing could have protected us against a determined attack.

Palpatine: A single bot that was put behind our firewall was able to get everything…

CISO for the fleet: Grand Moff Tarkin felt that it was impossible for a bot to escape the station or communicate externally…

Palpatine: Grand Moff is dead?

CISO for the fleet: Yes, Grand Moff is dead.

Palpatine: Pity. At least we won’t need to replace the commander of our space station. I suppose we’ll need a new CISO for the fleet.

Blue lightning crackles from the Emperor’s hand. The CISO for the fleet crumbles. His second in command steps forward… 

New CISO for the fleet: Emperor, we’ll re-organize our security protocols immediately.

At Zynga, our security team was – actually – ahead of the curve. Our strategy was not to rely just on a hard shell. We also created internal segmentation of our systems.  Basically, we created firewalls around each of our games and each of our systems. This kind of internal segmentation was a layer of protection that I thought was standard practice. More honestly, I thought this kind of protection was unnecessary. The recent disasters show that it is not. Too many people rely on a single external hard shell … unfortunately, once you get through the hard shell, everything is available.

This kind of internal segmentation is not as yet standard practice across the industry, nor was it standard in a galaxy far, far away…

And in all cases, the results were not that pretty…

death-star-explosion-o