wrong tool

You are finite. Zathras is finite. This is wrong tool.

Open Facebook API or what to do about Facebook

by Leave a Comment

When I left Zynga in 2013, I was convinced that Facebook was a malevolent entity run by leaders who could not be trusted. But I was also bitter about a 6$ stock price and my life choices.

Fast-forward to 2019, and it turns out that what I thought was just sour grapes, undersold the net harm Facebook has created.

An option that isn’t considered very seriously is the following simple proposal. Don’t break up Facebook, but regulate the access to and control of the friend graph and the ability to use the friend graph to publish information.

In 2012, when Facebook and Zynga stood off, the debate that was at the heart of the disagreement was ownership of the friend graph. Facebook believe they owned the friend graph and by extension owned how it could be used. We disagreed. In the end, we caved. I know this because I worked on the software systems necessary to create a parallel friend graph of people who were friends with other people who played Zynga games.

Facebook would love for us to spend time talking about breaking things up, instead of talking about the one thing that matters, a regulated open-api and regulated data portability.

Consider the messenger space. Because the friend graph is in my personal address book, it’s trivial to talk to several dozen different friends. Because the content is on my phone, typically pictures or documents, I can share anything with anyone.

Consider how many more messenger apps there are, versus how many social networks there are.

But let’s look to the past. During the failed MSFT anti-trust trial, a peculiar part of the agreement said that MSFT could no longer have private APIs, and that they had to communicate changes in a very specific public way.

This ruling enabled NetApp, which had built a reverse engineered CIFS server to survive and thrive. Because MSFT was losing the CIFS business, it also pushed MSFT to look for alternatives to CIFS, like SharePoint for document sharing and collaboration.

But over the long term, it enabled companies like Box and Google Drive and other file-sharing companies to emerge. Without the guarantee that a single man couldn’t break an API, a healthy and vibrant ecosystem in data storage has emerged.

If we had an open-social graph, and an open api, and data portability then I suspect that over time new social networks would emerge. Every social network would probably cater to different kinds of people.

In many ways Facebook does this today with Facebook Groups. For example, I happen to have joined two Facebook groups, one dedicated to old-school rpg, and another to 5E. The two groups hate each other. But because my social graph is portable, I can communicate to both groups within facebook.

Or we can even go back to Facebook’s origins. When Mr. Zuckerberg opened up the API, he promised it was going to be open and portable. He lied, of course, but not before Mark Pincus and Zynga figured out how to exploit the graph to grow Facebook’s business. Once, Mr. Zuckerberg figured out that owning the graph and how you communicate with it was very valuable, he squashed us like a bug. And destroyed the Facebook app eco-system.

Which brings me to regulation, we can’t trust Mr. Zuckerberg . Like we couldn’t trust Mr. Gates. And breakups don’t always work. Look at ATT, 40 years after the breakup, they control everything, again.

Book Recommendations

by Leave a Comment

A friend of mine asked on Facebook, what books should he read, that others found great from last year.

And so here are mine. This isn’t in any particular order.

The Expanse by James S. A. Corey

This is a great series on great power politics written in a Game of Thrones style. The first set of books feel too much like a dungeon and dragons adventure, the latter set of books really hit their stride.

Imperial by Thomas Vollman

Bought it 10 years ago, finally started to read it this year. OMFG. This is the best book I have read since Victor Hugo’s Les Miserables. The French version, because the one in English is unreadable.

Barbarian Days: A surfing life by William Finnegan.

This book made me regret not learning to surf 15 years ago when I first went to Hawaii. A co-worker of mine and I have agreed to go get a lesson in Santa Cruz because we both had the same regrets.

Master Switch and the Attention Merchants by Tim Wu.

Advertising and communication lines are tied at the hip. The original advertisers pandered to poor taste to make money. The original owner of the network, ATT and the radio companies, controlled what could and could not be done, delaying television for almost a decade.

Circe by Madeleine Miller

She wrote Achilles Heel. And this is an incredible book. She is able to capture the magic and faith and belief system of the Ancient Greeks from a unique perspective, a powerless nymph. The powerless nymph, however, acquires power through science (aka medicine) and becomes powerful.

Great book. My favorite part is the Goddess Athena.

The Marne, 1914: The Opening of World War I and the Battle That Changed the World by Herman Hedwig.

Hedwig’s book has excruciating details about every twist and turn of every unit. But if you step back, you see that this was the defining moment of the 20th century. The Germany Army was the 19th-century army. It was cabal of kings and queens of Germany with very poor co-ordination. The French Army was the first 20th-century army under the firm grip of a single general. At the Marne, the feudal nature of the 19th century ends, and the totalitarian era begins. After the defeat, the German army gets re-organized under Ludendorf, and everything we recall happens. The horrors of the 20th began with the French victory at the Marne.

The other thing the book makes clear, even if it does it in a backhanded way, is that even if Paris had been conquered, the French were not done, and the Germans were exhausted. There was no path to victory.

The problem with Facebook and Machine Learning

by Leave a Comment

At Zynga, Facebook and us got into a game of cat and mouse. Facebook would tweak it’s algorithms and rules to limit our reach, and we would work around it.

Facebook never could win. We had 2000 people working to figure out how to work around their algorithms, and they had a few dozen.

They only won when they just put limiters on Zynga and our APIs. And they won when they forced us to sign a contract or go out of business.

This taught me a powerful lesson, Machine Learning and it’s derivatives, are terrible tools when the cost of a mistake is non-zero.

For Facebook, when the cost of a mistake was non-zero, they didn’t rely on machine learning, they relied on regulations and laws.

But when a Facebook post goes viral about the evils of vaccination, when a Facebook post goes viral about how Clinton murdered children, this is to Facebook’s benefit. It drives engagement, it drives advertising revenue, it accrues tremendous benefit to them.

Over the past few years smart folks, friends of mine even, have tried and are trying to do something about this. But at the end of the day, as long as the damage to society is acceptable to society, Facebook has no incentive to do anything.

Because the cost of mistake is zero to Facebook. No one at Facebook or it’s shareholders is foolish enough to not get a vaccination. No one at Facebook is in danger of being physically assaulted on the streets of New York for being LGBTQA+ (oh wait, some of them are, but I digress).

When Facebook exec’s say that nothing can be done, they are lying. When the cost of mistakes was non-zero to them, they discovered the power of regulation. And they embraced it.

19 architecturalist papers: why doing the right thing matters, a tale of Facebook and charities.

by Leave a Comment

When I was at Zynga, Mark Pincus and the executive team had this brilliant idea on how to raise money for charity, selling virtual goods.

The idea was pretty simple, they had a virtual good, that virtual good was relevant to the game, and if you used real money, we gave a portion of the money to some charity.

This technique generated a lot of money for charities. And, to be fair, it was great for Zynga as well. Even if we did not keep the money, getting people to spend on a free game was hard, but once you got them to pay, it was straightforward to get them to pay more.

But we had to stop.

Why?

Facebook Credits.

See Facebook and Zynga signed a deal to have Zynga use Facebook Credits instead of real dollars. Feels a lot like Libra, but I am bitter. And because we used Facebook Credits, we needed to get them to do some back-office paperwork.

So I got the foundation to agree to do anything and everything that Facebook needed.

And they said, no.

I said that I would write a blog raking them over the coals for not prioritizing incremental revenue over doing good.

And they said, “Do it, we do not care.”

So I worked with our MarComm team to put something together. And we had layoffs, and our business was imploding and they asked me to not post it. They had so many other fires to put out, that this felt over the top.

And I agreed.

And I was wrong to agree.

Because, since then, no one has done this. Not one single freemium game has done this. Nada. Not one.

At Zynga, we pioneered a lot of the pay-to-play game mechanics. But Facebook’s payment team of the time pioneered the idea that charity was not a business priority.

It’s my fault for not having a spine six years ago. I wonder if I wrote that blog, things would be different. How many people would be alive if I had just done what was right?

When Facebook started it’s “charitable” giving on their timeline, I puked. I got so angry that I donated 1000$ to Mother Jones because they were the only publication that was willing to call out Facebook. Heck, I offered to give another 500$ as a matching donation. No one from Mother Jones asked, I just did it. I went on twitter and said if people sent me a note with a proof of a donation, I would donate 500$ to Mother Jones; I was that angry. And while we are here, give to Mother Jones, they are an excellent liberal paper that fights the power.

I screwed up.

So why am I writing now? Because a friend of mine saw a freemium game that did something for charity, and it made me happy. It meant that some games were trying to do the right thing again.

The Elder Scrolls Online 

@TESOnline

Thousands of Dragons have been slain since Elsweyr released – but now you can continue defeating them for a good cause! Raise money for real-world charities that support pets in need with each Dragon kill in #ESO. beth.games/2oVobFW #SlayDragonsSaveCats

And I also wanted to remind everyone that there are consequences to not doing the right thing. I get angry when I see folks ask how do we incentivize tech companies to do the right thing. We should be asking them what kind of moral bankruptcy exists that says the right thing to do isn’t something you do? But I didn’t do the right thing. And the industry is different as a result. And worse, a lot of people are not better off because I didn’t bother to write that blog.

As software architects, we make choices, and we are accountable for those choices.

Rage quitting

by Leave a Comment

The only rule in tech is you never, ever, ever, rage quit.

What’s rage quitting?

It’s getting so pissed off, that you walk out the door and take whatever next job you find.

Now, let me point out there are circumstances where rage quitting is the right thing to do, and that’s where there are moral and ethical lines being crossed that you can not abide. Or where someone does something that is truly horrible. I have no experience in those kinds of circumstances, and I offer no useful advice. All I can say is – do what you need to do to be safe, and talk to a labor attorney.

But if you aren’t in one of those really bad places, why is rage quitting a bad plan?

First, it’s a bad plan because your next job is probably not going to be the next job you actually want. And so you’ll be looking for another job pretty soon after that.

Second. it’s a bad plan, because the person who pushed you to rage quit, is forcing you to leave on their terms, not yours.

Third, because it puts you in a vulnerable position. A lot of very bad people can tell if you are the kind of person who will rage quit. They will exploit that, to get you to quit. They will push your buttons, and get you so angry that you will quit.

And they don’t get to run your life, you get to run your life, and surrendering that much control to them is never a good idea.

So you are angry, and furious and pissed off and want to quit, what do you do?

First, take some time off. You can take time off at work, you can take time off at home, you can find the mental space to not be angry. As Joe Beda said, don’t rage quit, rage vacation.

Once you are done with the vacation, then figure out what the right next move is. Be honest with yourself, and determine what you want. In some cases, it’s going to find another job. In other cases, its talk to your boss or his boss about the fact you wanted to rage quit, and see if a change is possible.

But figure that out.

And then go get it.

They is quitting, and what to do about it

by Leave a Comment

Over the years my teams have gone through waves of attrition. And typically those waves of attrition hit me like a punch to the gut, because every time they have happened, I was asked to do a dive and catch.

I was asked to try and convince someone to not take a job that they had already accepted. And that’s a tough place to be.

And it sometimes worked, but most often did not.

And even it did, they probably quit within 6 months.

And especially for senior people, their departures at critical phases of a project or business was very disruptive.

So if you get int that situation, what can you offer?

1. Offer more money

2. Offer another title

3. Offer time off

4. Offer a new project.

But the problem with these offers is that you are just buying yourself some time. Whatever underlying issues caused the person to quit will resurface after 6 months.

And what really upset me was that I was always felt they were leaving, not going. That we failed to show them the opportunity and deal with their real legitimate grievances.

So why bother? Because you get six months to figure out, on your terms, what the transition plan should be.

And that brings me to what you actually have to do.

From painful personal experience, I learned that 1×1’s are the best path to ensuring that folks are feeling connected to the organization.

The more 1×1’s you have across the org, the more people are connected to the team, the more you can deal with problems earlier rather than later.

Sometimes people leave, and that’s okay. Change is not a bad thing. But you owe it to your team for them to feel that there was a better opportunity somewhere else, and that they are not fleeing a bad place.

And so in my latest gig we mandated that our teams and leads do 1×1 and report the number of 1×1’s. And our attrition rate dropped to historic lows. Yes, part of it was the awesomeness of the Project Pacific, and our stock price, and part of it was that folks were feeling very connected to the mission as a whole.

And I am doing a lot fewer dive and catches these days….

We live in a magical era.

by 5 Comments

Last night, my father-in-law and I were discussing the notion of truth, and in particular, how the anti-Vax movement exists and why is the truth is so malleable.

And what I observed, and thought may be worth sharing, is that the human-made world is as magical as the natural world. And that has profound implications for us as human beings.

What do I mean by magical?

If we consider something like math, there are fundamental axioms that allow us to create or discover theorems. Those axioms make it possible to understand the world more deeply. Those axioms are not provable.
As mathematics is layered, a typical mathematical proof will have some reference to some theorem that has been proven elsewhere, that the paper hinges on.

And mathematicians assume that a theorem is correct. It is right because it both can be used to derive more math, and because it has resisted the assault on its correctness from other mathematicians.

Every so often, however, a theorem is proven to be wrong, or unproven. When that happens, a whole sub-branch of mathematics that depended on that theorem dissolves into nothingness.

In short, mathematicians rely on the assumption that the other parts of mathematics are correct.

They believe the theorems are correct, without necessarily understanding the entire depth of their correctness. Mathematicians also simultaneously understand that at some fundamental level, their truth depends on unprovable but useful axioms.

And that faith is magical thinking. We believe that the system works; therefore, it works, while realizing it may not.

And is this fundamentally different from the Ancient Greek perspective on Helios? After all, they believed that a titan dragged the Sun was dragged across the sky. And that explanation, although absurd, had sufficient predictive power that the farmers of the time could make other inferences and assumptions. And it was a perfectly workable model of the Sun. Absurd, but workable.

In short, the mathematicians’ faith in a theorem is the same faith of the Ancient Greek farmer in a titan who drags the Sun across the sky. Both could be proven to be absurdly incorrect, and both useful.

My intent is not to suggest that mathematics is not rigorous, but to illustrate how the most stringent of intellectual disciplines is also based somewhat on faith.

But why do I say that the world is magical?

Because, in the past, the human-made world was not magical. A person could understand everything about his house. They could understand how to build it, and they could understand how to make the tools they used, he could understand why some tools worked better than others. The natural world was magical, but the person-made world was rational.

The triumph of modern science is a person-made world that is as magical as the natural world.

The rationalist project to reduce the natural world to the well understood human-made world failed. We know that we don’t know and must make assumptions.

And if it only ended at science, the rest of us could live in a perfectly rational and understood world.

But the engineers created a magical world that defies understanding. Consider the micro-processor. It represents several hundred thousand person-years of engineering and science (if not millions). No one can understand every piece of it, because there is not enough time in a human life-time to understand it.

Or consider the reader of this blog, that the set of technologies and science to make reading this blog possible, make the Apollo project look like a ten-person startup.

So engineers must rely on faith. Things work, so we continue to use them. And we build stuff on top of them.

But unfortunately, this faith has destroyed the truth. Since no one can understand the human-made world, based on a natural world we don’t understand, then everything is based on faith.

And if we base everything on faith, then everything is magical. We believe things to be true, because they work and because no one has disproved them or because they are useful. And if some truths are useful, then others are also useful. And if all truths are contingent and based on assumptions, then …

Why then it becomes a natural slippery slope to anti-Vaxers. Their truth is just as valid as anyone else’s in their minds; they have a different faith. They have a particular faith that science is wrong. They have a particular belief that big-pharma is evil. They have a particular faith that they have been lied to.

Because it turns out that everything about our world is magical. Everything we touch works because we believe other parts work, and as long as all of our assumptions hold, things hold. But, deep down, we all know that these assumptions are fragile things, like the mathematician and their axioms. And we know that something we don’t understand could up-end them, like the Ancient Greek farmer and his belief in Helios.

18 architecturalist papers: As was foretold.

by Leave a Comment

Recently, a coworker of mine approached me and said, “All you have to do is figure out what the final right answer is.”

And I stared at him, and I was surprised at how ridiculous the comment sounded.

I turned to him and I said, “anyone can do that!”

In fact, it was at that point in time that I realized how little I understood what the nature of system architecture is. Figuring out how to build the correct answer is the least interesting part. Figuring out how to build the next part, while giving you optionality to build the correct solution later is the real job.

The job is to see what is being done today, understand where you want to go, and course correct efforts that are going in the wrong direction.

System architecture lives between the now, and perfect future, an area of complete grey. And the challenge is in that gray area; there are no correct answers.

The Minbari Grey Council is a perfect metaphor. Their job was to stand before the now, and the future that they knew was coming and making the hard choices. That space between the now and the future was unclear and uncertain. And they chose, when confronted with the future, to not make a choice.

The job of the system architect, is to know when the right thing to do is to break the Grey Council and when it is not.

The challenge for the system architect is that when you see a project that is going off the rails, you need to understand how much you need to get involved. Is this a project that if it succeeds will take the company in the new wrong direction? Or is this an effort that will open new opportunities that currently don’t exist? The height of hubris is to assume you know the answer. But to do nothing, is to say yes to everything.

Ultimately system architecture is a reflection of the taste of the architect. And like fake turning machines, not all taste is correct all problems.

The challenge, is to understand when you were taste is getting in the way of new opportunity and when your taste is telling you that something is going wrong.

17 architecturalist papers: go fast and build things

by Leave a Comment

Over the last ten years, I have struggled with a Facebook motto: Go Fast and Break Things.

It reminds me too much of the Great Gatsby quote:

They were careless people, Tom and Daisy—they smashed up things and . . . then retreated back into their money . . . and let other people clean up the mess they had made.

The process “go fast and break things” does not describe a method for creating value; it represents a process for an adrenaline high, an excuse to do whatever you want.

Let me ground this into a real-world example. Suppose I have a system, and I want to radically re-imagine what the system can do.

There are two paths.

The first is to create a brand new system, that is entirely incompatible and breaks everything. What do I mean by everything?  Any system is part of an ecosystem of tools and operations, and people that interact with the system. When you break the system, you are breaking that web of relationships and interactions. The net outcome is a radical change of that web.

So why do it? Well, because the cost of that change is borne by the people who use the system, not by the people who built the system. The more powerful the market position, the easier it is for the entrenched system to break things.

For example, Facebook used to break APIs all of the time. And that was okay for Facebook, because their captive audience, had no choice but to change to use the new APIs. The consumer-owned the full cost of the disruption.

The second approach is to evolve the system in a way that doesn’t break anything. In this model, instead of forcing the world to adapt to your system, you figure out how to integrate into their world.

Intel is an excellent poster-child for both of those. The first time was when they delivered the Pentium. At the time of the 486, there was a bunch of RISC processors like MIPS, Alpha, SPARC, that those of us in the field thought had a legitimate chance of dethroning Intel because the CISC core was intrinsically slower than RISC systems. But getting off of Intel meant breaking things. Instead, Intel did something that was a surprise to the casual followers of the industry, they embedded a RISC-like core into their processor, preserving the CISC instructions. By choosing not to break things, they won the CPU core wars.

Ironically, a little later, Intel pursued a foolish – in retrospect – strategy of the Itanium. The thing about the Itanium was that at the time there was no 64 bit Intel processor. Intel planned to move away from the x86 instruction set to go to a new kind of instruction set. The switch was highly disruptive. And AMD delivered what the market wanted, a 64-bit x86 processor and achieved a huge market opportunity.

In both cases, the winner deliberately chose to break as little as possible and add value in a way that did not disrupt the consumers of the technology.

To me, that is the best kind of engineering.

16 architecturalist papers: you work for the future GM

by Leave a Comment

One of the most challenging parts of the job of strategic software architect is that your job is to think about the future, and the GM’s job is to think about the present. And what’s worse, your planning horizon is typically beyond the planning horizon of the current GM.

Why is that a problem? Because we hate our future selves.

There is a lot of behavioral research that suggests we hate our future selves. We will do things that optimize for current happiness vs future happiness. Explains so many things about our choices. 

This, for example, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5611653/ or a more accessible and possibly more useful version here: https://www.anderson.ucla.edu/faculty-and-research/anderson-review/future-self-health

And this leads to my favorite story about the conflict between GM’s and their Strategic Software Architects.

In 2006 I sat in a room with Guy Churchward while at NetApp.
He was the new GM, and it was our first 1:1. And I told him: Look, Guy, you’re going to be gone in 18 months. And I’m going to be here for 3-5. My job is to make sure you don’t screw this technology up. If you do something I think is stupid, I will assure you it will not happen. Because I will make sure the dumbest, worst engineers are working on it. If you want to do something smart, I will make sure it succeeds. So you need to get me on board. And even if you get the smart guys to work on it, I will undermine their success, because it’s my job to make sure there is a technology that the next guy sitting across me has to go to market with.
It was a breathtakingly arrogant comment. But it was a true comment. Guy looked at me wondering if Technical Directors at NetApp could be fired.
Unfortunately for NetApp, he left way too soon from his job. And I left shortly thereafter.

He left not because he failed, but the nature of the GM job tenure is less than the tenure of the strategic software architect, and that is by design. We want the GM to be more short term focused and we want the strategic software architect to take the longer view.

The challenge is that we are working for the next GM. And the current GM is not interested in helping the future GM who is probably going to be somebody different. 
So the architect – in some sense – is that person who gets in the way of the current GM plan’s to help the future GM, someone the current GM hates (even if it’s him in 2 years). 
So then what? 
As architects, we are constantly fighting our current boss.
How does this manifest itself concretely:
If I am a GM and I have a product, to hit my numbers, I only need junior engineers. But if none of those turn into senior engineers in 1-2 years then product will have problems in 4. And if none of them turn into architects in 3, the product is dead in 7.
We can argue about the dates and ranges but the story holds true. If you don’t have senior technologists thinking about the future, then you miss the future.
So now what?
As strategic software architects, our job is to make the current GM think that the future GM is working for him.
Here’s how I always try to do that.
1. Make sure that the Strategic Software Architecture is something that the current GM will profit from. A GM has to make sales to companies who want to believe the company has a future, he has to attract technologists to build the current stuff, having a compelling technology strategy is very useful for both.
2. Make sure that the Strategic Software Architecture adds value all of the time to the current GM. This means that future pieces deliver value now.
3. Hire for the people you need to build the future, and have them build the present. This is this weird thing. You bring in someone to build your future, and have them work on the immediate problem. While you are doing that you wonder if you are making a mistake. I think of it as a twofer. The new gal learns something new AND gains credibility AND will build the future thing better.

4. Be flexible in planning. Every new GM will have new priorities, so be willing to change what you recommend to be built.