wrong tool

You are finite. Zathras is finite. This is wrong tool.

A rebuttal of some of the points of Mr. Jassy

by 2 Comments

Note I come to work five days a week. I like being in the office. I enjoy being around human beings. And have been leading remote teams for almost 20+ years. In that time, my groups have delivered over 10 billion dollars in actual money, many multiples of that in shareholder value.

Mr. Jassy wrote this memo to his staff, telling them why he would require everyone to come to work.

Update from Andy Jassy on return to office plans (aboutamazon.com)

So he starts with this:

It’s easier to learn, model, practice, and strengthen our culture when we’re in the office together most of the time and surrounded by our colleagues. It’s especially true for new people (and we’ve hired a lot of people in the pandemic); but it’s also true for people of all tenures at Amazon. When you’re in-person, people tend to be more engaged, observant, and attuned to what’s happening in the meetings and the cultural clues being communicated. 

Except if you have ADD and sitting in a meeting where you cannot sit still gets in the way. Or you are deaf, can’t hear what people are saying, and don’t have zoom translate what is being said in real-time.

For those unsure about why something happened or somebody reacted a certain way, it’s easier to ask ad-hoc questions on the way to lunch, in the elevator, or the hallway; whereas when you’re at home, you’re less likely to do so

I am surprised by this one. The emergence of technology like Slack or the DM function in Zoom has democratized communication. Finding time for a senior leader to ask them a question is tricky. And the senior leader in the elevator can walk away from you. But when you send them a message, they tend to take the time to respond because they can do so asynchronously.

In the more productive brainstorm sessions I’ve been a part of over the years, people get excited and blurt out new ideas or improvements to prior proposals, quickly advancing the seed of an idea, and leading to the broader group getting energized and feeling that it’s onto something. This rapid interjecting happens more often in-person because people feel less inhibited about jumping in or even interrupting sometimes. 

I am a cis-gendered white man at the apex of my profession, which worked for me. How many women and non-white and non-cis-gendered men have we trashed over the years because they didn’t speak up?

My first meeting back in person was the same old guys talking with all of the women and others silent.

It was so bad we had to introduce zoom to deal with the problem.

Serendipitous interactions help it, and there are more of those in-person than virtually

This one surprised me. The number of in-person interactions is an order of magnitude smaller than the interactions with people digitally. Just count the volume of people you contact over email, the number of people your email goes out to, etc. Mr. Jassy feels that is low-value communication.

Regardless, Mr. Jassy has made his choices, declared that he believes they are correct over other people’s lived experiences, and can force his employees to choose to work for Amazon or leave.

55 architecturalist papers: Facebook v Zynga Microchannel, the Apple Store, D&D, and killing platforms

by 2 Comments

IBM PS2 60 and 80 side-by-side

In 1981, IBM introduced a revolutionary computer that radically transformed the tech industry, the IBM-PC.

What made the IBM PC revolutionary was its open-system architecture and the royalty-free nature of its use.

Specifically, anyone could create an IBM clone, and anyone could develop software for said IBM clone and make money without paying a dime to IBM.

IBM hated that. So their solution was in 1987 to create the Microchannel PS/2. The intent was to create a new divergent market of PC’s that no one could make clones of.

What happened was that the market split between PS/2 and EISA, and the PS/2 became a failed computer system and a historical artifact and a warning to those who would want to close an open platform.

In June 2008, Apple introduced the App Store. The Apple App Store was the first massive commercial success of a cell phone app store. And it created a standard for how to take royalties from creators. And it was a huge success. So much so that we have had lawsuits that are still going through the court systems to determine the boundaries and limits of the market rules Apple can enforce in the market they created.

In June 2010, Facebook saw the success of the Apple Store and decided they wanted a piece of the Zynga action. Facebook had created an Open Platform, and that Open Platform enabled Zynga to grow like wildfire. But Facebook wasn’t making any money from Zynga. And so, there was a stand-off between Zynga and Facebook. Zynga and Facebook signed a deal that required Zynga to hand over 30% of its revenue to Facebook. That 30% revenue haircut killed Zynga.

Facebook’s thesis was that by creating a new currency, Facebook Credits, that Zynga customers would use, they would increase the total number of people who had digital money, and thus more money would be spent. The thesis failed, and Zynga suffered.

In both Apple and Facebook’s cases, they saw intellectual property creators who used their platforms as free-loaders. In Apple’s case, the tax was declared up-front, so you knew what you were getting into. In Facebook, it was an after-the-fact revision that destroyed a business.

But what happened to said intellectual property creators? As my son recently said – “why does mobile gaming suck?”

Gaming is a hit-driven business. One hit makes all the money, allowing you to make the next game. 30% is a massive tax, restricting the money you have to make further investments. And so the gaming industry has moved back to open platforms, ironically, the Windows PC.

This now brings us to the recent decision by Hasbro to introduce OGL 1.1. A lawyer covered it well here – https://medium.com/@MyLawyerFriend/lets-take-a-minute-to-talk-about-d-d-s-open-gaming-license-ogl-581312d48e2f

If you parse the Lawyer’s responses, it boils down to trying to create an Apple-like App Store for D&D content. Essentially, you hand over your financials and content for a smaller slice of the pie for the right to play.

In effect, it destroys the open ecosystem that D&D had. For example, suppose I have a website with a random generator of D&D content. That random generator is illegal.

Now Hasbro is betting that people play D&D and don’t care about the open content and that the creators will have to suck it up and deal.

Except, and this is a big exception, that isn’t true.

The iPhone was a singular technology with no ability to be replaced. Folks used Facebook because they wanted to connect with friends. Those platforms had value outside of the gaming industry. D&D is a game and a platform.

But it’s an extraordinary game where the player and the GM create content while playing. And the GM can adapt content from other gaming systems to their game. And the GM can adapt rules to their game.

In short, I expect the TTRPG community to discover the power of system-neutral gaming. And that the internet will increase with systems that allow you to convert to the gaming system of your choice. Except for the new restricted one.

My take, and it’s hopeful, is that D&D will continue, but tabletop role-playing will finally escape the long shadow of its creator and his original game.

But going back to software architecture and platforms, it’s always tempting to control a market, and there is a lot of value in doing that. But when you extract a lot of money from a market, you eventually kill the market. And over time, those creators whose businesses are hits will move to open platforms.

Where else to find me

by Leave a Comment

kostadis_roussos | Twitter | Linktree

54 architecturalist papers: career progression for those of us who are under-represented

by 1 Comment

I intend to write a lot about why you should go and read this https://hackingcapitalism.io/

And then realized I could write very little.

It would be best if you read Kris Nóva’s Hacking Capitalism because it is a highly well-written document by an outsider on how to function and succeed in the alien tech world. It would be best if you read it because Kris Nóva (@nova@hachyderm.io) is a fantastic person who is impressive and distills a lifetime of insight into a tight actionable document.

And that’s the only reason you should read it. And if you need me to tell you why beyond that, here’s my life story.

At NetApp circa 2002, a great manager named Jonathan Crowther took me to lunch to discuss my career. It was the first time any manager had done that.

I was frustrated because my career had stalled out. And I didn’t understand why.

And he smiled and said that the problem was that I only approached people as functional units that needed to solve my problems or I needed to solve theirs. That I never treated people as people.

And I remember staring at him, and then he explained, “when you come in on Monday, you never say – how was your weekend? It’s always – I need this.”

I stared at him. I suspect I am somewhere on the spectrum. And it was a moment of clear revelation.

I spent the entire weekend thinking about this one conversation. A key benefit I had was that I had done a lot of acting, enjoyed D&D, and had an absurdly high executive function. And so I came on Monday morning and acted. I have a youtube video where I talk about this here.

What resonates with Ms. Nóva’s document is that she detaches herself from the reality of understanding why and accepts that things are and that she must devise rules on how to win based on that.

I lack her precision of English, wit, insight, and, to be quite frank, experience. And I wish, when I was 22, she had given me this document.

So who should read it? If you were not born in the United States, you were not a white CIS-gendered man, and you are not privileged, you should read this document.

The United States is an alien country. Its rules are strange. And its culture is derived from 300+ years of interaction between property rights, the evil belief that labor should be enslaved, that extreme Christian Dogmas are fundamental, and a sense of supreme righteousness and contempt.

For example, let’s consider ownership.

A year ago, I had an opportunity to talk to a Chinese woman, from mainland China. And she was talking about ownership and how she was struggling with her career because of that. And I realized she was talking about ownership in a way that sounded like a colorblind person talking about colors.

And then we talked about “what is ownership in the USA.” In the USA, ownership of property is a sacred right. When you own something, you own it. You can do whatever you want to it. And nobody can tell you otherwise. A considerable amount of the conflict in this country is putting boundaries on the ownership of people by others and the limits of ownership of common goods by individuals at the expense of the group. The idea that the country protected property rights from others and the government was eye-opening to her. Her life experience in Xi’s China and property rights was not that. And as we talked about ownership, she said – “oh, so that’s what ownership means.”


I love this country, and as a Greek from a village, a Canadian from Montreal surrounded by more small-town Greeks who fled Greece due to a civil war, a world war, and a junta, the culture of the United States and beliefs frequently leave me perplexed.

Hacking Capitalism is a guide to making this system work for you without understanding the why and just understanding its weak points.

Ubiquity and the ventilator and adding 15 years to everyone’s life

by Leave a Comment

My father is Prof. Charis Roussos. He’s currently 78 years old. And he’s a legendary figure.

To make Prof. Roussos’s impact tangible, let’s compare how set designers imagined the future of hospital medicine.

In 1966, the set designers of the sci-fi show Star Trek imagined the medlab of ~2254 looked like this.


The hospital was a single doctor with a digital chart.

But in 2009, the set designers for the film Star Trek imagined the medlab in 2233 looking like this.

Notice the ventilator, the machines surrounding the hospital bed? Prof. Roussos made them a necessary part of medicine.

But why did this ventilator become so ubiquitous in just 43 years?

Before his research, there was this thing called “shock.” Patients would enter “shock” and die. And the why was a mystery.

But its existence meant that a whole class of things we take for granted are impossible. Surgeries that could save your life but might lead to “shock” were very risky.

But was shock?

He and his teams proved that shock happened when blood pressure dropped.

Let me explain.

Typically you have a specific volume of oxygenated blood pumping through your body. We all know that the heart is a muscle, and we all know about the lungs, but what we didn’t realize was that there were some equally if not more essential muscles, “the lung muscles or the thoracic muscles.”

The body needs oxygen; otherwise, it dies. And without the lung muscles moving, you don’t breathe and die.

But for the lung muscles to move, they need oxygen as well.

Under normal conditions, this is not a problem. There is enough oxygen coming in and enough blood going around for everyone.

But when a patient is under distress, and there is less blood to go around, the brain needs to make some tradeoffs. Well, it can’t stop breathing, so it keeps the lungs working and starves everything else, including itself.

But how much blood do these lungs need? Well, about 25% of the total blood flow.

So what was shock – less blood being pumped through the heart meant less oxygenated blood for everyone, which meant that vital organs started to fail, which led to death. To deal with that, the brain then tells the lung muscles, “WORK HARDER,” and sends even more blood, and the lungs work harder, and vital organs get further starved. . Until they are exhausted, and the brain realize it can do nothing, gives up, and the patient dies.

Shocking.

These lung muscles were the key to keeping people alive!

But how to do that? Well, if the brain only cared about oxygen, what if we fed the patient oxygen or artificially moved the lungs?

And that’s where ventilators come in.

What if we used ventilators to move the lungs artificially, allowing more blood to flow to other parts of the body?

That simple idea saved people who would otherwise have died and enabled surgeries and procedures that in the past were too risky.

You can hear him talk about this here.

52 architecturalist papers: the different flavors of availability

by Leave a Comment

In a recent sequence of discussions on availability at work, I realized that there are different flavors.

But first, a history lesson.

Before virtualization became a thing, the physical data center physically existed. And its destruction, although possible, was extremely rare.

The destruction of an entire physical data center was extremely rare, and we referred to it as a “meteor strike or nuclear war.”

For practical purposes, availability was about dealing with physical component failures. For example, if a server had a disk, and the disk died, could the application on the server keep running?

As hardware components became increasingly resilient and the cost of hardware plummeted, the weak link became the software.

Because rearchitecting software to become intrinsically highly available was impossible, new software frameworks came into existence.

One of the most impressive was vSphere-HA because it said the following

  1. If you have highly available storage and enterprise storage was highly available
  2. If you have shared storage, and enterprise storage was shared
  3. And you capacity connected to that storage, all of your applications will restart after 5 minutes.

vSphere HA and its more boutique solutions more or less solved the availability problem.

And the power of vSphere-HA to systematically solve the HA problem across an entire application fleet greatly expanded the value of VMware’s technology value proposition.

Except, a new availability problem emerged.

In the past, a single software could not destroy a single data center or all data centers. The scope of the blast radius of an operator error was quite limited.

But with virtualized infrastructures and cloud infrastructure, a single button click can destroy an environment in minutes.

For example, at Zynga, an operator error caused by a lousy user-experience design deleted the production Cityville deployment. Cityville had over 10 million Daily Active Users (DAU) and 30 million Monthly Active Users (MAU).

Why was this possible?

Because in the past, you couldn’t delete a server, the applications in the server, the network connections associated with that server, and the data contained in those servers with a single button click in minutes because it would require logging into hundreds of systems. Why? Because the nature of the systems relied on multiple administrators of infrastructure coordinating to either provision or delete infrastructure.

Let’s use an analogy from social media. Before Facebook, I couldn’t share the news with millions of people using a few button clicks to use an analogy. I had to talk to a lot of people. And so, the blast radius of my news was quite limited. Facebook made it easier to share good information like I have a new child very quickly, and sad news like a friend died very quickly. It also made it possible to share fake news and lies very quickly as well. The same technology could be used to do both. And the societal damage of that rapid dissemination of phony information is confirmed.

Virtualization and later cloud moved the entire data center that the application depended on into a single database. The single database allowed agility to grow, shrink, and adjust infrastructure demand to meet the changing application demand. It also made this database the single most significant vulnerability to your environment.

And so software engineers rightfully focused on how to make that database available. But the problem is that no amount of software available can solve operator error, and worse, mistakes that destroy the database in its entirety.

As more and more effort was put into making the central system more robust and available, the overall design became more and more fragile.

But what kinds of errors? Remember, the physical data center rarely goes away in its entirety, but several servers? Those fail all of the time. Worse, as the servers become less available to become cheaper and more disposable because technologies like vSphere HA or cloud-native 12-factor application styles expand, their failure rate increases.

And this cheaper hardware that is more disposable is possible because of the very same centralized databases that when they fail destroy everything. Antifragility makes it clear that the more and more you make everything depend on a single system, the more and more the entire system becomes fragile.

So we have this peculiar phenomenon that the entire virtual infrastructure of the data center resides in a single piece of software that if it fails, everything is gone. And the thing that can cause it to fail is not under the control of software: things like Human error, or data corruptions in parts of the stack that are unknowable, or ransomware, etc.

Without any personal internal knowledge, the Facebook DNS debacle is a great example. The software allowed the network engineering team to make widespread infrastructure changes across Facebook in minutes. And human error combined with technology errors (just another word for human errors) resulted in the entire Facebook infrastructure is unavailable.

IT operators understand this point, as did DevOps teams and the SRE teams. And this is why they refrain from having a single system control their entire infrastructure. The scope of a single error is why those groups like to have more k8s clusters instead of one large one. Or why backup admins have backup software from a different vendor from their primary storage.

Centralizing things allows for unprecedented agility and failure.

What to do?

My take is threefold.

The first is that the era of centralized databases that control the entire infrastructure is closing. Instead, we will have many databases. In the k8s space, the idea of a few large k8s clusters is an anti-pattern. And this is why I am such a huge fan of products like Tanzu Mission Control. They line up with the future.

The second is that those centralized databases will have to be built leveraging blockchain systems that protect against byzantine errors and, in particular, human errors.

The third is that the ability to recover from those database errors in a timely fashion will become a critical differentiator. A once-in-a-decade event is terrible if it happens on a Friday and a company ending event on Black Friday. Software stacks that make that recovery quick and easy will win.

51 architecturalist papers: transactions and arrangements and architecture reviews

by Leave a Comment

One of the best program managers I have had the chance to work with observed that companies have challenges when transactional-based decision making and arrangement-based decisions come into conflict.

Transactional-based decision-making is what I call strict enumeration and documentation in the form of a contract. Once both sides agree on the document, then even if people change, the decision remains. The trust is in the record, and the change controls surrounding the document. This kind of decision depersonalizes the decision. These decisions tend to be transparent.

Arrangement-based decision-making relies on trust between the two parties. The parties establish trust in various ways, but typically it’s about making sure that the personal goals between both leaders align. Arrangements are remarkably durable and transportable. Meaning that once you establish an arrangement with someone, trust is the basis of subsequent decisions much faster. It also means that decisions without going through a complex process. The problem is that these decisions tend to be opaque.

Both models taken to extremes can be a disaster. I have worked in both. And they both suck.

The law is THE example of a transactional system. Overly transactional environments turn into bureaucratic, legalistic environments.

A pure transactional system can turn into a totalitarian state. A great example is – “the only way to change this spec is to call a meeting of the change control board and submit a request.”

A pure arrangement system can turn into a cult where belief in the great leader is paramount and the phrase “blah said” is used to justify or denounce everything. That disagreement with the great leader is an unforgivable sin. A great example of this is when someone says – “but SR ARCHITECT FOO said”. Or alternatively – an old-boys network – that is impenetrable.

I’ll also observe that both approaches feel natural to different people. My personal experience is being Greek and Canadian and living in the USA. As a Greek, I believe that laws are suggestions. That relationships, and in particular, family relationships, trump everything. As someone who lives in Sunnyvale, CA, I have learned that in the USA, laws matter, but that the laws are structured to satisfy arrangements among the wealthy.

A transactional system feels like a spectacular waste of time because personal relationships trump everything and that, ultimately, transactional systems are a facade for arrangements.

But I have learned that that is naive. The critical flaw of arrangements is the opacity of building trust and the boundaries of trust. Transactional-based decision-making creates a public record of the moment trust was built and describes the trust boundary. Without such a record, trust-building naturally devolves to family, culture, background, and other attributes.

This brings me to how I think about architecture reviews. The purpose of the review is to create a trust boundary between the approver and the author. The more high-level the spec, the more trust that has to exist. That trust is typically built over a series of smaller successes or previous professional successes. The more detailed the spec, the less trust that exists. A key trust moment as an architect is when you are willing to stop being the sole approver of an area.

The key illusion in all of this is that as an architect the spec actually defines what is built. The reality is that unless you are writing the software, some other human being is going to take that document and do what they think the right thing is. My job is to make sure that they are thinking about the problem in a way that aligns with a reasonable solution.

Decrying arrangements because they are old boys networks is wrong. Decrying transactional-based systems because Process is also wrong. Like most everything in life, there is a balance. And like most everything in life, navigating that balance is the art of living and being a software architect.

50 architecturalist papers: portability and the multi-cloud

by Leave a Comment

Over the last 15 years, I’ve been noodling about portability. And the more I noodle about portability, the more I think it’s very poorly defined.

So, let me try.

In the context of computers, there are two flavors of portability, data path portability, and control plane portability. The data path is the bits of code that do real work. The control path is the code that sets up the data path, monitors the data path, and takes corrective actions.

In that weird way that computers work, the control path itself is just another data path. And so – in a bizarre way – there is only data path portability. Simplistically, the control path is instructions that execute on processors, store data in memory and storage.

And although Paris Kanelakis would be pleased to know I learned something in CS051, things are not- quite that simple in the world of commerce that I live in.

Except when they are.

The VM encapsulates a traditional single VM applications control and data path. And the VM, as a result, is a highly portable abstraction because the hypervisor can ignore which instructions are from the control path and which data path; to it, they are all data path instructions. And we know that if you don’t care about performance, any data path can simulate any other data path.

So far, so good.

VMware demonstrated that the portability of a VM is very valuable because you can move it between generations of hardware, move it around to utilize the efficiency of your infrastructure better, and because it’s a convenient way to manage workloads.

And so, the VM abstraction and its portability had another interesting effect. You can hand over the infrastructure operation to another person.

But while I was at NetApp and Zynga, there were all sorts of other portability that people discussed. This data path portability was nice but not sufficient, and it wasn’t enough if you couldn’t encapsulate the entire app into a single VM.

So sure, I could move the VM around, but if I didn’t have the control path, what value was the VM?

And so there was this thought process that said, “VM’s are not that interesting.” are

And I looked into all sorts of interesting programming languages and tool chests.

And yet, along the way, something funny happened. If I cared about performance, I cared a lot about the hardware I was running on. And all of a sudden, the behavior of the hypervisor matter a lot.

At Zynga, when Amazon changed their NIC, our costs skyrocketed. Their decision to optimize their business almost tanked ours.

But performance, who cares about performance? Right? Except, here’s the rub, when you don’t have pricing power for a feature, and you sell the hardware that the feature runs on. The only way it can be worth implementing is to implement the feature without meaningfully increasing the hardware costs.

And then the performance of the feature matters a lot. And some features don’t get implemented because there isn’t an efficient way to do that.

But if you can’t charge more money, why implement the feature? Because – here’s the rub – your competition is adding features very fast. And if you don’t stay ahead, the whole SaaS model suggests that eventually, your customers will move. So as a vendor, you have to keep adding features while keeping the cost of the features in check.

Hoom. So – wait, if I want to grow my business and add more services, and can’t meaningfully increase the price, then, erm, the performance of the feature matters a lot.

And yes, you can make code more efficient and thoughtful, but at some point, you start caring about whether the disk is an SSD or whether the NIC has enough buffers, or what exactly is the CPU you are running on.

I mean, if people didn’t care about these things, those things wouldn’t exist.

Okay, so?

Well, once you start getting to the point that you care about performance that much, then the hypervisor details matter a lot. But surely you jest? Hypervisors make choices, and those choices impact the way your applications run. And if you design an application to run on a particular hypervisor, then the portability of your application is the portability of that hypervisor. If the hypervisor only runs on hardware X or Y, or Z, any performance improvement locks you into that vendor.

To put it differently, if the control path of the hardware is in the hypervisor, and you need to control the hardware, you need to control the hypervisor.

So what? It turns out that the hardware vendors are adding more hardware faster than that integrated solution vendor is sharing. And so, being able to have some degree of portability to run on other hardware will matter a lot.

Because why? Because the more it costs to run a feature, the less money you have and the more money the other person has.

Hoom.

Baremetal! BAREMETAL!

I hear you say. I was cheeky. See, every modern OS is a hypervisor of some kind. When you run on a public cloud, you can’t control the hypervisor.

And in some ways, not to be annoying, running your app in a VM on a cloud is like trying to tune your Java code – optimizations can be done, or you can use C or Rust or C++.

So when I hear bare metal, I don’t hear “no hypervisor”; I hear – – run a hypervisor of your choice on hardware.

BUT BAREMETAL IS HARD. I agree.

So?

Well, here’s where things get very interesting. Suppose you had a portable hypervisor that existed, and you didn’t have to do the hard part of managing the hardware but could control said hypervisor to your heart’s content?

Why then, for those applications where the data path was critical, you wouldn’t be tied to the cloud vendor’s hardware choices.

Data path portability is about being able to run your VM anywhere you want and be able to control the hypervisor.

AHA!

So for the essential thing that can move my costs, I need to move my VM to whatever hardware I want and tweak the hypervisor any way I want.

But then why has the public cloud won so much business?

Well, for the same reason, Java has won over C++.

It’s just got a better control plane for application development. Java makes so many things easier.

And really, that portable hypervisor was painful to use compared to the cloud.

So hypervisor portability was nice, but it was uninteresting if you couldn’t run the control plane. And you couldn’t because it took a lot of time and money to build, and the value of doing that work was marginal. Except when it wasn’t like for Dropbox. Or at Zynga, where we designed our control plane to be portable and had options when Amazon decided to optimize their business at our expense.

But the control planes aren’t tied to the hardware platforms as much.

What does that mean?

Well, it means that for some workloads where the costs matter, using the non-proprietary control planes so I can use the portable hypervisor may be a better choice.

And well, a portable control plane is – to a hypervisor – just another data path.

REPATRIATION!

No.

No software service that exists and increases in value hasn’t been optimized and improved over time. The original S3 relied on MySQL databases, for crying out loud, and Facebook used a single NetApp filer.

Thinking about this as “repatriation” is the wrong mental model. The suitable model is that critical services will get optimized, and at some point, the optimizations will care about the hardware. The existence of portable control planes and portable data planes will be fascinating.

But I have to build my data center!

Nope.

See, those portable hypervisors are pretty much available as a service now.

And that leads me to the final round of thinking.

When application control and data plane portability is possible, will the ability to cram a small number of hardware configurations in data centers be that valuable?

A better physical working environment and the WFH

by Leave a Comment

Watching how many of my coworkers don’t want to go full-time back to work got me thinking about the physical work environment.

Pre-Covid my home office was a laptop with poor ergonomic characteristics on my kitchen table and my car.

My work office, an 8×12 area, shared with a co-worker, had a 32″ monitor, a desk that could move up and down, and a perfect chair and was just a fundamentally more pleasant place to work.

Post-Covid, my home office is a 40×40 room, with a 32″ and 27″ monitor, a desk that can go up and down, a perfect chair, a nice couch, and is quiet.

The effect of Covid was I invested a lot of time and effort to create a great working area at home that no company in Silicon Valley can afford to replicate.

Being forced to come to work in my old office would be going from a really nice office to a worse office – my car and the workspace in my HQ.

If I wasn’t an extrovert, and loud thus irritating my family, I don’t know if I would want to go back to the office.

I am very fortunate in my workspace at work. Some of my coworkers have much smaller and louder workspaces. And those smaller and louder workspaces were created to pack more people into the office because that was just how we did things.

And this got me thinking further.

Pre-COVID, workspaces were a perk that companies could hand out as a way to reward employees. Now employees, having built much better workspaces at home, are rebelling at the idea that they must trade their home workspaces for the qualitatively worse office space.

 

48 architecturalist papers: system efficiency vs features and the multi-cloud debate

by Leave a Comment

Over the years in my career, I have seen the tension between – “MOAR FEATURES” vs. “MOAR EFFICIENCY’.

The more features camp tends to view everything through the prism of – add more features to find more customers. Therefore, every business problem is a feature problem. And any activity that doesn’t result in more features is a waste of time.

The more efficient camp tends to view everything through the prism of – if the software and the construction of the software were more efficient, we could save money and be more productive.

The pro-features folks tend to argue that growth is the solution to every efficiency problem. If you have more money, you can spend your way out of any problem. And so the goal is always to be making more money. The pro-features camp also argues that there are many ways to be more efficient, including hiring cheaper engineers.

This Manichean view of the universe is how we arrive at Product Manager type CEOs arguing that we shouldn’t invest in anything that detracts from feature velocity.

Strategic software architecture must take a different point of view.

Until a product has achieved product-market fit and viability past 18 months isn’t assured, any investment that pays out after 18 months is of questionable value.

Once a product-market fit has been achieved and viability past 18 months is assured, efficiency investments effectively extend the business’s runway, return the margin to the business, and allow the company to generate more cash per dollar spent.

And the point at which you pivot investment is a delicate one.

The pro-feature camp controls the agenda. And they will push hard against any notion that efficiency investment should trump features.

The pro-efficiency camp being overruled for so long will either not exist or have no ability to drive investment decisions.

Worse, the pro-feature camp, by their nature, tends to view any outcome that is further out than 18 months to be very speculative. And the pro-efficiency camp views the value of their work in a 36 month time horizon.

In this battle, the short-term always wins.

In this struggle and debate, my attitude has always been that efficiency must have its own agenda but must sell it to the pro-feature camp. In effect, every efficiency improvement is really a feature.

But to do that, you have to sacrifice the big bang for small incremental wins along the way. In effect, when the business wants a feature, you write code that also improves the efficiency of the system over time.

And, of course, it’s not that simple.

You also have to pick the right technologies. To deliver the biggest wins, you need to be able to use best-of-breed technologies where there is a competitive market, and you need to be able to swap out technologies easily.

Huh?

Let’s take multi-cloud, for example. One argument is that making your application portable is foolish because it detracts from the core business, which is feature delivery. The argument is typically presented in the form – “private cloud very hard, public cloud easy, single cloud easier.”

And there is some truth to that. And the ease comes at a cost. And debates ensue.

I see it very differently. There is this massive competitive market of infrastructure providers. That infrastructure is a commodity except in the rarest of cases. As a technologist, I enable the business to be more efficient by allowing it to take advantage of that commodity infrastructure through my software choices. Customers of infrastructure will choose because of externalities—things like laws, vendor relationships, and personal taste.

Pre-k8s, this argument was slightly harder to make because there was no obvious API layer that would allow you to decouple your application layer from the common infrastructure, and the market wasn’t very competitive.

In fact, in 2015, at VMware, I argued that such a layer would have to come into existence, or AWS would own the world. So our job at VMware was to stay alive until such a layer emerged. I remember saying to folks – this thing would emerge miraculously, and when it did, we would need to jump all over it. I remember folks looking at me as if I was delusional.

Well, k8s arrived and was exactly that layer, and we jumped all over it. K8s was breathtakingly brilliant because it was the right choice for developers and infrastructure providers.

And the reality is that just picking k8s already makes your SaaS system more portable. And it’s an easy choice and gives you portability from day 0.

But it’s more than just k8s. Picking a managed Postgres over a custom DB makes your application more portable.

Incremental choices that over time add up so that when you want to make bigger bets, the cost of making those bets is small.

And the real bet is that over time because the market is now competitive, the infrastructure vendors will create an ecosystem that makes multi-cloud easier and private cloud easier.

In fact, I believe that a single cloud strategy advocates ignore how k8s is creating an API layer that enables innovation and competition in all layers simultaneously. And it’s not the only layer. Amazon, for example,  with i3N metal instances, created a competitive market for hypervisors in AWS.

The anti-multi-cloud folks point to dropbox as a potential failure because of the complexity. On the contrary, I would argue dropbox is an example of how hard multi-cloud is without the right layers creating an opportunity for competition everywhere.

The anti-multi-cloud folks’ argument also turns to – well, java. The argument is that Java attempted to create a sealed abstraction that hid all infrastructure from the applications and failed.

And they have a point. But the goal isn’t the ability to run the application without change on any cloud; the goal is to make the cost of doing that so small that the efficiency gains can be realized. The goal is to make it easy to force the infrastructure vendors to give up some of their margins to the application.


In this picture, you can see what I mean. In the initial state, the application is using only proprietary APIs. Over time, the applications that are being developed have access to standard APIs. The existence of those standard APIs and a competitive market gives choices. The proprietary APIs never go away; they just become a smaller part of the application. And the most important part is that the goal is not to use custom APIs; the goal is not to let custom APIs hold you hostage, to a vendor. And the other point is that it’s multi-cloud; if you want to use custom APIs, use them.

The emergence of that competitive market is creating innovation right now.

And as a result, forward-thinking venture capitalists and forward-thinking architects will choose technologies that will facilitate the emergence of multi-cloud because it is in their selfish interest.