wrong tool

You are finite. Zathras is finite. This is wrong tool.

25 architecturalist papers: playing chess while everyone else plays checkers

by Leave a Comment

One of the tough challenges of acting as a strategic software architect is that it’s not precisely an understood job. Many people ask me what I do, and after several fumbling minutes, I point them to this blog.

The most recent analogy that became somewhat useful is that strategic software architecture is about playing chess while the rest of the world plays checkers. And what I mean is that the world is looking at short time horizons, planning the next step, while the role requires planning two or more steps—and simultaneously making moves during the current phase.

Strategic software architecture is not about the stuff that is shipping now. It’s about the thing that will ship in the future. And the job is not to deliver the current thing. The job is to make sure that the team can deliver the current stuff without you.

As that strategic software architect, when you are working on an immediate deliverable, what it means is you failed your team a long, long time ago. If they need your help, it means that you either didn’t provide them the resources, the people, or strategy that would ensure their success. Fail enough times, and there is a new strategic software architect.

In most software companies, the planning cycle doesn’t extend out longer than 18 months. The world changes too fast for anything more than that. And so nobody is thinking past 18 months.

There is one group that is thinking past 18 months, these strategic thinkers. They are a critical, sufficiently unrecognized group across a large number of business functions, but this is about engineering, so I am focusing on that. As this role doesn’t exist and isn’t recognized, and there are no rewards for long term success and, it begs the question of ‘does it exist’?

There are many people like that at a company. They are the ones who seem to generate magic just when the company needs it. That continues to deliver value, and nobody knows why or how. As engineers, they do it with well-timed code-reviews, speaking whispers to the right people, working on the right project, checking in something that nobody expected. They call meetings to discuss things in private, and thereby create a social network that is impenetrable and built around the respect they have earned and the reputation they have acquired.

And over time, the company strategy is the strategic thinkers’ strategy, even if the company thinks otherwise. For many reasons, beginning with hiring that is shaped through their biases. What is easy to build and hard to make is what they and their social network think is easy and hard. What can be created is controlled by their tastes. What is easy and hard to do is shaped through the myriad of small technical decisions that make change very hard. And their software architecture ossifies their decisions through org charts that can endure long past the code choices that formed them are relevant.

Currently, this entire area is left to chance. We are lucky to hire people who can do the job. And I have seen it in my career. Where there are groups that seemingly out of nowhere, keep doing the right thing. And things keep getting better, but I can’t figure out why. And finally, somewhere someone turns up that has a plan . That plan exists in their heads, or on a piece of paper or a confluence page that nobody reads but that everybody is working on.

I believe that a company that figures out how to do strategic software architecture as a discipline and incorporates it into the 18-month planning has a decisive advantage. I also believe that if they can couple this approach with a rabid focus on immediate delivery, they can’t lose.

24 architecturalist papers: how to not engage the A-team.

by Leave a Comment

There are many reasons to change jobs. Some are better than others. But the best is when your peers or your boss don’t engage you for your best work.

Let me start with a story. In 2009, I was a technical director at NetApp. At many other companies, this role is analogous to a senior technical leader with a pay grade that is equivalent to that of a director-level manager.

At the time, NetApp was engaged in a multiyear effort to converge the operating system of the company they had acquired Spinnaker and their platform OnTAP. After the first unsuccessful attempt with the product known as OnTAP GX, the technical and business leadership of the company rallied around a strategy that eventually became what is now known as OnTAP 8.0.

This effort required vast amounts of synchronization. At one point, I was the architect for the data protection portion of the business. The overall architect for the effort sent me an email with a detailed task breakdown of what the data protection team needed to accomplish over the next two years. As I looked at the list I had some concerns with some of the details and the overall general direction, and so I flippantly responded with a “this is so detailed that I’m not sure what value I’m going to add. Why don’t you just send it directly to the management team in the product managers.” And so the author of the email took my response and forwarded the email with his comment, “you’re right.”

After that, it took very little for me to want to leave NetApp.

Over the years I have wondered why this particular exchange was so critical in my leaving NetApp. NetApp was at the time treating me well. I would’ve made more money at NetApp than I did at Zynga. I had a five-week vacation at NetApp. I was well respected by the then CTO and chief science officer. My boss at the time and I had some differences of opinion, but those differences of opinion were resolvable. And the problem space remained interesting.

So why did I leave?

I left because at the end of the day that other architect did not want to engage with my best self. He was not interested in working with me to come up with the right answer. He just wanted me to do exactly as I was told and to take accountability for his decisions.

In short, what I heard him say is, “I don’t need you to think. I need you to do exactly as you’re told and to make sure that the things I need done are done.”

Over the years, I have seen this pattern play out again and again. Sometimes with me on receiving that message or the author of such a message. What I have concluded is that if you’re engaging someone to solve a critical business problem and you talk to them in a way that demonstrates you do not see value in their abilities, then you’re asking them to leave.

But there is another insidious problem that also can occur. If you’re trying to engage with another team where you happen to know a lot about how that team’s systems are built, it is tempting to bypass the current architect and just tell them to do this and this. The problem is that what you’re doing is not engaging the team to be able to solve your problems. And what this kind of communication does is discourage precisely the people you need. The people who want to have a scope to think and to imagine possible solutions. They decide to not want to work on your problems.

If the problem is small in scope, then this is not necessarily a bad thing. If the problem is significant in scope, this is a calamitous decision. You’ve traded off the delivery of a small set of critical capabilities at the expense of a deeper understanding of a hard, complex problem. You have reduced a team that could potentially add value by understanding the problem and thinking about it hard and long for a team that will do exactly as they’re told and no more.

“But that wasn’t the goal,” I have said on more than one occasion.

No, it wasn’t. But that is precisely what I have achieved in the past. Because the people who could think realized that there was room for them to think, only to do. So instead of looking at a problem and trying to solve it, they saw a list of activities that could be done by someone who couldn’t think as deeply as they could. At the end of the day knowledge workers have a tremendous amount of freedom to choose what problems they work on. They also have a tremendous amount of freedom to decide how long and how hard they want to work on an issue. The single most calamitous decision you can make as an architect is to engage with another architect by treating them as less than an equal.

Over the years, I have made this mistake. To those that I treated poorly, my loss is massive but nothing as compared to how poorly I treated you. All I can say is that life is about getting better and learning more new things. And maybe this goes a little distance as an apology.

 

 

 

23 architecturalist papers: latency kills

by 2 Comments

While at NetApp, I saw the incredible effort that became known as ONTAP 8.0 and was part of the spinnaker acquisition.

From that experience, I learned a few seminal things that continue to resonate. The short version is that latency kills.

Let me start by saying, that the hard problem in storage is how to deliver low-latency and durability. Enterprise storage vendors earn their 70% gross margin because of the complexity in solving two issues that appear to conflict. The conflict is that durability requires a copy, and making a copy slows things down.

The solution was, and is, to use algorithms, in-memory data structures,  and CPU cycles to deliver latency and durability.

When Spinnaker was acquired, there was a belief within the storage industry that single-socket performance had reached a tipping point, and that performance could only be improved if we threw more sockets at the problem.

And, in retrospect, they were right. Except, we collectively missed another trend. Although the single-thread performance was no longer going to double at the same rate, the performance of media was going to go through a discontinuity and radically improve its performance.

But at the time, this wasn’t obvious.

And so many folks concluded that you could only improve performance through scale-out architectures.

The problem with scale-out architectures is that although single node-latency can be as good as local latency, remote latency is worse than local latency.

And application developers prefer, for simplicity, to write code that assumes uniform latency of the infrastructure.

And so applications tend to be engineered for the worst-case latency.

And single-node systems were able to compete with clustered systems. As media got faster, and as single-node performance improved, application performance on non-scale-out architectures was always better.

In short, the scale-out architectures delivered higher throughput, but worse latency.

And it turns out that throughput workloads are not, generally, valuable.

And so scale-out for performance has it’s a niche, but it was not able to disrupt non-scale out architectures.

Over time, clustered storage systems added different value than performance, but the whole experience taught me that customers will always pay for better latency. And that if there is enough money to be made in the problem space, it will be solved in such a way to avoid applications from changing.

22 architecturalist papers: multi-tenancy and quotas

by Leave a Comment

Over the last years, I have gotten into a series of protracted debates about multi-tenancy.

What I have begun to understand is that it is essential to define the objectives of multi-tenancy before one starts to talk about it.

And even before we get to that need to define what is multitenant.

Consider a piece of hardware, say a server with four sockets. An individual owns the server. Another individual owns the building in which the server resides.

In effect, when there are two actors Mary and Tom, that have access to a system, that system is said to be multitenant if Mary and Tom do not trust each other.

But how much do they trust each other? The trust goes to how much the system must protect Mary from Tom and vice versa. For example, suppose Mary trusts Tom. Then Mary doesn’t care that Tom has physical access to the hardware. And Mary takes no actions to protect her data or her applications running on that server. In effect, Mary and Tom are the same people; they have different roles.

But suppose Mary trusts Tom, but Tom doesn’t want to damage Mary’s system accidentally. Identities and roles play a factor. What Mary would like to do is have a role that Tom can use that allows him to do the things he needs to do to Mary’s server and no more.

And so this is where things get complicated. There are two basic approaches; the first is to bake into the system the set of controls that Tom has access to and to use some role-based access system integrated with some identity system that determines what Tom can do. The problem with such an approach is that if Tom needs to do something that is not in the system, he has no way to do it and has to ask Mary. Now, if Mary is okay with that, all good, however, Mary may not want to do the task and may wish to allow Tom to do the job. But if the system has no way for her to do that, then she is forced to give him access to more controls than he is capable of using.

The second approach is to use layering. You create a net new interface that interacts with Mary system through some APIs, and that net new interface is what Tom uses. Thus when Mary wants to enable Tom to do something new, she will, Tom, can extend his tool to do that. The problem with this approach is that Tom now has access to a whole bunch of operations he shouldn’t have. The only thing preventing Tom from using those operations is his adherence to procedure and the fact that at the end of the day, Tom isn’t malicious. He’s a good guy.

My observation is that approach one doesn’t work. The reason it doesn’t work is the set of operations that Tom needs to perform is ever-evolving. Worse, the collection of activities that Mary wishes Tom to do is ever-expanding. And as a result, they end up using the second approach.

Okay, so what?

The problem is that too many people attempt to build the first model. For example, suppose I have an interface for interacting with the system. That interface allows me to create objects delete objects, or modify objects. Then what happens is that somebody decides that the hierarchy of those objects should reflect some authorization scheme. Then what happens is that Tom and Mary can’t do their jobs because the hierarchy or the complexity of configuring and setting up the hierarchy and setting up authorization is not expressible by the system. In effect, the hierarchy and system that allows you to create edit and manipulate objects for one task is not the same hierarchy you would use for another.

And so, ultimately, what you do is you create a tool that has a specific set of operations that Tom needs. Mary and Tom configure the tool so that it only does what it needs to do.

But, the advocates of the first system point out that the second approach is less secure. And they are right. Or I’ll take them at their word.

They ask, what if Coke and Pepsi want to run their software on the same physical servers. I always found that to be an absurd question. Even if we could assume that the system was entirely secure, there is human error. I thought that Coke and Pepsi would always buy their servers. What is interesting is that the market seems to be doing that even in the public cloud. The Nitro hardware that Amazon has produced mainly provides physical instances on a shared server. And this was before we discovered that there are architectural holes in our systems that allow data to leak between programs running on the same physical server that belong to different tenants.

And so, my assumption has always been that if you care about security, air gaps are about the only thing you should trust.

What does this mean?

Consider the server. With no software, it can do anything you could imagine. The minute you start running software, the set of things you can do becomes increasingly more limited. It turns out that there is a whole slew of user interfaces that are a lot more useful than just starting with the hardware. Over time, a set of interfaces for using a system and controlling access to that system have emerged. And we have figured out over many years how to make them address both Tom and Mary’s needs. A great example is the use of root and less privileged users on most operating systems.

A user interface for the system is handy to both Mary and Tom is incredibly powerful. And therefore, whenever a new way of interacting with servers emerges, there is a temptation to try to figure out what the boundary between the two tenants should be. The reality is that such interfaces developed after years of hard work and experience in operational practicality. Therefore in a new system, you are most likely to draw the boundary in the wrong place. And thus, in my mind, how you access a system should be independent of how you control access.

Okay, I’m dangerously close to talking about security, and when it comes to security, I know that I know nothing.

The problem is that even if you don’t care about security, another critical use case of multi-tenancy is to reduce the cost for the infrastructure provider, Indigo. What Indigo wants to be able to do is assign quotas to Mary and Jane and Tom. And what they want is to ensure that Mary, Jane, and Tom don’t ever exceed their quotas.

Amazon’s solution was to create an infinite supply of servers and bill Mary Jane and Tom for their usage.

The limitation of such a system is that you can only buy the set of servers that Amazon has decided to make available. The other limitation is that it assumes an infinite infrastructure.

If, however, Indigo does not have access to an infinite infrastructure or it’s inappropriate for their use case, what to do?

In my opinion, they should choose approach number two. What does this mean? There are a set of objects that Mary Jane and Tom use to do their jobs. Indigo has a set of quotas that they assigned to Mary, Jane, and Tom. Mary, Tom, and Jane’s need to refer to the quotas and use them transparently. And so there is a temptation to encode them in the objects. But instead of having quota enforcement done at every access to the objects, it should be done lazily unless you exceed some threshold.

And if you look at what Amazon does, they do the same thing.

If you want to use one more server, they will give it to you. If you’re going to employ 10,000 more servers, that involves a phone call. They have their quotas that are lazily enforced and, at some point in time, block access.

In effect, Amazon has decoupled secure isolation from quota enforcement.

And so, when we talk about multi-tenancy, what we need to do is ask, are we trying to solve for secure isolation, or are we trying to solve for quota enforcement? The requirements for security depend on the customer, the trust, the legal requirements, etc. How you do quotas is independent of all of those security restrictions and should be treated as such.

Career trajectory

by 2 Comments

The last ten years of my career are an example of how linear progression is a myth.

In 2009, I left NetApp in a huff. My attempts to redirect our investment strategy to either pursuing aggressive investments in replication or what later on became known as HCI went nowhere. Worse, for me, NetApp was more interested in what was known as convergence and later on became known as 8.0, and I wasn’t interested in that effort.

And so I left NetApp.

And I went to Zynga. Zynga was an incredible experience. The first startup, first pre-IPO, first IPO, cloud, on-prem, hybrid.

And for the first three years, I was flying on top of the world. I remember feeling that I had finally arrived!

Then Facebook changed its algorithms, did Libra 1.0 (aka Facebook credits), and Zynga fell, And we failed to deliver on mobile games. And I went from speaking to hundreds of analysts, building cool games, to being out. In less than 12 months, I went from the top of the world, to out.

So I looked for a job and took one at Juniper. And that was a professional experience that was surreal. In short, I joined to help transform the culture and got laid-off in 14 months.

And then, I ended up at VMware. By then, I had gone from a high-flying senior architect to joining VMware at a time when VMware was not that cool.

And the team I joined had just endured a lay-off, a re-org, and after I joined, the attrition was high.

I like to describe our situation in the following way. We were a team in a dark freezing room. Inside the room were a few pieces of dry lumber. And we had two matches. With the first match, we found the wood, and with the second, we lit it. We were great, and we were lucky.

The gap between success and failure of 6.5 was minimal. We had managed to assemble the right kind of team that could pull it off, but we didn’t know it at the time.

And then, after we worked to figure out our k8s strategy, we got lucky again.

The Heptio team decided to join VMware. And thanks to their decisive contribution, what was an excellent k8s technology strategy, became a tremendous k8s strategy.

The wheel has turned, and I gave a keynote at k8s about the work my team did. And the first time I did my talk at Zynga, I thought that this is why it must be because I am that good. And now, nearly seven years later, I realize that I wasn’t that good. I was lucky.

 

21 architecturalist papers: always be right, right now.

by 1 Comment

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

One of the particular challenges of being a software architect is that the average manager and vice-president of engineering assume you are incapable of adding value, right now.

Engineering managers have releases they need to deliver to now. They have engineers who have problems now. The managers have budget squabbles now. The teams have debates with product management now.

An architect has visions of the future, and those visions are often years away from delivery, and worse, even more years away from solving any of the immediate problems engineering management has.

And so the VP of engineering begins to see the architect as a distraction. The architect distracts in two ways. The first is that they are always complaining that the team is not investing in the future. The second is that the things they want are unimplementable. And worse, the architect is typically unwilling or unable to go figure anything out. Powerpoint gets produced, confluence pages get edited, wiki pages get updated, and in some cases, small prototypes get written, but no useful code gets changed.

And thus, the VP tries to manage that distraction. There are two fundamental approaches. The first is to keep them away from anything that matters and fire them at some opportune moment. The second is to focus them on a small problem. A small problem keeps the architect from distracting other people and makes it easy to measure their delivery of value. In other words, force the powerpoint jockey to write code, and either they figure out how to write code, or they get fired.

And you know what, the VP of engineering is right.

The thing about being a software architect is that you always have to be right. What I mean is that you are guiding a team, and the direction has to be correct. If it’s not correct, then the team is heading towards a catastrophe. You can change course over time, but it always has to be the right course.

In addition to always being right, you also have to be sufficiently high-level, so everyone is doing the right thing. Pulling this off is another tricky thing to master. If the correct answer is not inclusive of the entire organization, then somebody is doing something wrong. Furthermore, if it’s too low-level, then there are no white-spaces for engineers to innovate.

Another customer for the long term architecture is the CEO/GM and Product Management team. They have to see enough value that they can talk about it to their customers.

In short, you need to be right, and high-level enough that you can’t be wrong, but if that’s where you end, you fail.

The software architect must also add value right now. What does that mean? It means if the head of product management chooses to fund X features, all X are stepping stones to your architectural vision. If the engineers have to design something, it should be evident from the high-level architecture what they should do. If the managers have to figure out how to trade off long term vs. short-term execution, they should make that decision in the context of the global vision.

But it’s more than that. Managers operate in terms of things that need to get done with some set of resources. And so it’s vital that the architecture, be broken down into discrete tasks.

And so my mantra,

  • I must always be right.
  • I must be sufficiently high-level to be always right.
  • I must be right, right now.

Open Facebook API or what to do about Facebook

by Leave a Comment

When I left Zynga in 2013, I was convinced that Facebook was a malevolent entity run by leaders who could not be trusted. But I was also bitter about a 6$ stock price and my life choices.

Fast-forward to 2019, and it turns out that what I thought was just sour grapes, undersold the net harm Facebook has created.

An option that isn’t considered very seriously is the following simple proposal. Don’t break up Facebook, but regulate the access to and control of the friend graph and the ability to use the friend graph to publish information.

In 2012, when Facebook and Zynga stood off, the debate that was at the heart of the disagreement was ownership of the friend graph. Facebook believe they owned the friend graph and by extension owned how it could be used. We disagreed. In the end, we caved. I know this because I worked on the software systems necessary to create a parallel friend graph of people who were friends with other people who played Zynga games.

Facebook would love for us to spend time talking about breaking things up, instead of talking about the one thing that matters, a regulated open-api and regulated data portability.

Consider the messenger space. Because the friend graph is in my personal address book, it’s trivial to talk to several dozen different friends. Because the content is on my phone, typically pictures or documents, I can share anything with anyone.

Consider how many more messenger apps there are, versus how many social networks there are.

But let’s look to the past. During the failed MSFT anti-trust trial, a peculiar part of the agreement said that MSFT could no longer have private APIs, and that they had to communicate changes in a very specific public way.

This ruling enabled NetApp, which had built a reverse engineered CIFS server to survive and thrive. Because MSFT was losing the CIFS business, it also pushed MSFT to look for alternatives to CIFS, like SharePoint for document sharing and collaboration.

But over the long term, it enabled companies like Box and Google Drive and other file-sharing companies to emerge. Without the guarantee that a single man couldn’t break an API, a healthy and vibrant ecosystem in data storage has emerged.

If we had an open-social graph, and an open api, and data portability then I suspect that over time new social networks would emerge. Every social network would probably cater to different kinds of people.

In many ways Facebook does this today with Facebook Groups. For example, I happen to have joined two Facebook groups, one dedicated to old-school rpg, and another to 5E. The two groups hate each other. But because my social graph is portable, I can communicate to both groups within facebook.

Or we can even go back to Facebook’s origins. When Mr. Zuckerberg opened up the API, he promised it was going to be open and portable. He lied, of course, but not before Mark Pincus and Zynga figured out how to exploit the graph to grow Facebook’s business. Once, Mr. Zuckerberg figured out that owning the graph and how you communicate with it was very valuable, he squashed us like a bug. And destroyed the Facebook app eco-system.

Which brings me to regulation, we can’t trust Mr. Zuckerberg . Like we couldn’t trust Mr. Gates. And breakups don’t always work. Look at ATT, 40 years after the breakup, they control everything, again.

Book Recommendations

by Leave a Comment

A friend of mine asked on Facebook, what books should he read, that others found great from last year.

And so here are mine. This isn’t in any particular order.

The Expanse by James S. A. Corey

This is a great series on great power politics written in a Game of Thrones style. The first set of books feel too much like a dungeon and dragons adventure, the latter set of books really hit their stride.

Imperial by Thomas Vollman

Bought it 10 years ago, finally started to read it this year. OMFG. This is the best book I have read since Victor Hugo’s Les Miserables. The French version, because the one in English is unreadable.

Barbarian Days: A surfing life by William Finnegan.

This book made me regret not learning to surf 15 years ago when I first went to Hawaii. A co-worker of mine and I have agreed to go get a lesson in Santa Cruz because we both had the same regrets.

Master Switch and the Attention Merchants by Tim Wu.

Advertising and communication lines are tied at the hip. The original advertisers pandered to poor taste to make money. The original owner of the network, ATT and the radio companies, controlled what could and could not be done, delaying television for almost a decade.

Circe by Madeleine Miller

She wrote Achilles Heel. And this is an incredible book. She is able to capture the magic and faith and belief system of the Ancient Greeks from a unique perspective, a powerless nymph. The powerless nymph, however, acquires power through science (aka medicine) and becomes powerful.

Great book. My favorite part is the Goddess Athena.

The Marne, 1914: The Opening of World War I and the Battle That Changed the World by Herman Hedwig.

Hedwig’s book has excruciating details about every twist and turn of every unit. But if you step back, you see that this was the defining moment of the 20th century. The Germany Army was the 19th-century army. It was cabal of kings and queens of Germany with very poor co-ordination. The French Army was the first 20th-century army under the firm grip of a single general. At the Marne, the feudal nature of the 19th century ends, and the totalitarian era begins. After the defeat, the German army gets re-organized under Ludendorf, and everything we recall happens. The horrors of the 20th began with the French victory at the Marne.

The other thing the book makes clear, even if it does it in a backhanded way, is that even if Paris had been conquered, the French were not done, and the Germans were exhausted. There was no path to victory.

The problem with Facebook and Machine Learning

by Leave a Comment

At Zynga, Facebook and us got into a game of cat and mouse. Facebook would tweak it’s algorithms and rules to limit our reach, and we would work around it.

Facebook never could win. We had 2000 people working to figure out how to work around their algorithms, and they had a few dozen.

They only won when they just put limiters on Zynga and our APIs. And they won when they forced us to sign a contract or go out of business.

This taught me a powerful lesson, Machine Learning and it’s derivatives, are terrible tools when the cost of a mistake is non-zero.

For Facebook, when the cost of a mistake was non-zero, they didn’t rely on machine learning, they relied on regulations and laws.

But when a Facebook post goes viral about the evils of vaccination, when a Facebook post goes viral about how Clinton murdered children, this is to Facebook’s benefit. It drives engagement, it drives advertising revenue, it accrues tremendous benefit to them.

Over the past few years smart folks, friends of mine even, have tried and are trying to do something about this. But at the end of the day, as long as the damage to society is acceptable to society, Facebook has no incentive to do anything.

Because the cost of mistake is zero to Facebook. No one at Facebook or it’s shareholders is foolish enough to not get a vaccination. No one at Facebook is in danger of being physically assaulted on the streets of New York for being LGBTQA+ (oh wait, some of them are, but I digress).

When Facebook exec’s say that nothing can be done, they are lying. When the cost of mistakes was non-zero to them, they discovered the power of regulation. And they embraced it.

19 architecturalist papers: why doing the right thing matters, a tale of Facebook and charities.

by Leave a Comment

When I was at Zynga, Mark Pincus and the executive team had this brilliant idea on how to raise money for charity, selling virtual goods.

The idea was pretty simple, they had a virtual good, that virtual good was relevant to the game, and if you used real money, we gave a portion of the money to some charity.

This technique generated a lot of money for charities. And, to be fair, it was great for Zynga as well. Even if we did not keep the money, getting people to spend on a free game was hard, but once you got them to pay, it was straightforward to get them to pay more.

But we had to stop.

Why?

Facebook Credits.

See Facebook and Zynga signed a deal to have Zynga use Facebook Credits instead of real dollars. Feels a lot like Libra, but I am bitter. And because we used Facebook Credits, we needed to get them to do some back-office paperwork.

So I got the foundation to agree to do anything and everything that Facebook needed.

And they said, no.

I said that I would write a blog raking them over the coals for not prioritizing incremental revenue over doing good.

And they said, “Do it, we do not care.”

So I worked with our MarComm team to put something together. And we had layoffs, and our business was imploding and they asked me to not post it. They had so many other fires to put out, that this felt over the top.

And I agreed.

And I was wrong to agree.

Because, since then, no one has done this. Not one single freemium game has done this. Nada. Not one.

At Zynga, we pioneered a lot of the pay-to-play game mechanics. But Facebook’s payment team of the time pioneered the idea that charity was not a business priority.

It’s my fault for not having a spine six years ago. I wonder if I wrote that blog, things would be different. How many people would be alive if I had just done what was right?

When Facebook started it’s “charitable” giving on their timeline, I puked. I got so angry that I donated 1000$ to Mother Jones because they were the only publication that was willing to call out Facebook. Heck, I offered to give another 500$ as a matching donation. No one from Mother Jones asked, I just did it. I went on twitter and said if people sent me a note with a proof of a donation, I would donate 500$ to Mother Jones; I was that angry. And while we are here, give to Mother Jones, they are an excellent liberal paper that fights the power.

I screwed up.

So why am I writing now? Because a friend of mine saw a freemium game that did something for charity, and it made me happy. It meant that some games were trying to do the right thing again.

The Elder Scrolls Online 

@TESOnline

Thousands of Dragons have been slain since Elsweyr released – but now you can continue defeating them for a good cause! Raise money for real-world charities that support pets in need with each Dragon kill in #ESO. beth.games/2oVobFW #SlayDragonsSaveCats

And I also wanted to remind everyone that there are consequences to not doing the right thing. I get angry when I see folks ask how do we incentivize tech companies to do the right thing. We should be asking them what kind of moral bankruptcy exists that says the right thing to do isn’t something you do? But I didn’t do the right thing. And the industry is different as a result. And worse, a lot of people are not better off because I didn’t bother to write that blog.

As software architects, we make choices, and we are accountable for those choices.