wrong tool

You are finite. Zathras is finite. This is wrong tool.

Bob’s Corner 01 – Authentication tied to the user.

by Leave a Comment

I was fortunate to have three different mentors that made my career. One was Bob English. What we used to say about Bob was – ‘Bob is always right, we just don’t always listen to Bob.’ Bob and I used to talk about technology and I enjoyed those conversations. And I thought it would be fun to have them. Most of them were the result of him writing something I barely understood and didn’t agree with.

Almost 20 years ago, I was working on a streaming media appliance. And Bob was irritated because he felt that there was a business model that could make streaming work, and selling appliances was not it. Ironically he was arguing for NetFlix like streaming models where residuals were a thing about 10+ years before NetFlix became a thing.

The current strike makes me wonder what Bob thinks about AI and media.

Who is Bob?

Robert M English is a computer systems architect with experience in server development, systems internals, and large-scale systems deployment.

ChatGPT and deepfake technologies have generated new concerns for computer security, mostly due to the possibility of much stronger, more personalized phishing attacks conducted at a large scale. Such attacks attempt to get users to divulge the secrets underlying most account security and are effective precisely because authentication is based on such secrets. An alternative approach, tying authentication to the identity of the user, would be far less vulnerable to such attacks and much stronger overall.

Bob wearing a hat with a beard
Robert “Bob” M. English

To see the relative strength of an identity-based model relative to secrets associated with accounts, consider the way we access websites. We contact DNS, which points us to a set of IP addresses, and then we open a secure connection. Passwords do not play a role in this process. Google.com does not have accounts with its users. Instead, it authenticates itself with a certificate, which clients can verify using the website’s public key. The private keys associated with this system are still secrets that must be guarded, but the number of secrets is small and tightly controlled. Successful impersonations of websites are rare and typically limited to the site where the security breach took place.

With account security, the number of secrets that require protection is the product of users and websites, and the secrets themselves are widely spread. Users often choose passwords themselves, so learning a single password on a single site can compromise a user everywhere, making users only as secure as their weakest account. Password managers can prevent the re-use of passwords and help users manage the resulting password explosion, but they rarely work with all sites or applications.

Multi-factor authentication helps by treating the user more as a person with an identity separate from the account and requiring that the user respond through a separate channel, with its own authentication mechanisms, in order for the password to be accepted. Unfortunately, MFA implementations tend to be limited. Some effectively replace passwords entirely, so that the secondary communication channel becomes the true single factor in security. Some create serious account recovery challenges in the event of device loss. Most require additional steps in a typical authentication sequence, creating a poor user experience.

The situation is worse with accounts that predate the web. In many financial systems, access is granted with the account number, a ‘secret’ shared whenever the system is used. Social security numbers are used both as personal identifiers and secrets that can verify identity. Systems sometimes attempt to verify these tokens with additional information, but the information they use is often publicly available.

The recent introduction of passkeys improves security by replacing passwords with public keys similar to those used in website authentication, but the approach remains fragmented, based on shared knowledge at the level of individual accounts. Both accounts and users must track the public/private key pair associated with the user and service, increasing complexity in operation, storage, and recovery.

A better alternative is to base authentication on the identity of the person or entity taking the action. Instead of the knowledge of a secret (however constructed) giving access, the identity of the person gives access. Jane Doe might have access to an email account, an online retail account, a credit card, and a bank account. Steve Smith would not, whether or not he knew a password or account number. Jane could give access to Joe simply by granting Joe access privileges, without the need for an explicit interaction between Joe and the account.

Prior to the arrival of mobile devices, identity was difficult to verify. A person might hold a driver’s license, but a website (for example) would have no way to determine that or to determine who held the card. Today, mobile devices are ubiquitous, able to perform secure protocols to signal the identity of their owners and equipped with layers of mechanisms to determine whether their owner is, in fact, the person using the device. Most authentication today uses these mechanisms to drive access without making them central to the process. Making identification central to authentication would both greatly enhance security and simplify the lives of users.

In digital identification architectures (such as the ones described in this NIST standard), a user establishes credentials via a Credential Service Provider, and those credentials establish the user’s identity. Individual services associate the credentials with the account. The CSP is responsible for verifying that users are, in fact, who they say they are and reflecting to services a level of confidence in that assessment. Users have control over how much information they reveal about themselves to services, allowing privacy to be protected, even when the CSP has access to enough information to make identification very strong.

Users might choose, for example, to allow a CSP to track their locations on a regular basis as a means to enhance security. In such a case, a successful impersonator would have to both defeat whatever mechanisms the CSP used to verify identity and either match the user’s movements or prevent the user from updating the CSP with their new location. Users might be unwilling to share such information with every service they interact with but might be willing to share it with a CSP with strong data protection policies.

Account recovery within such a framework can be greatly simplified. With MFA approaches, for example, a lost device might require an MFA interaction with every affected account. With an identity-focused approach, a user has to reverify with the CSP, and that implicitly re-verifies with all associated accounts. The fact that it can be done once for all accounts allows recovery to be more secure as well as more complete. A user could, for example, have limited access on initial recovery, and only have full access restored by having their identity physically verified at a trusted institution such as a bank.

Identity-based authentication is intrinsically additive. If Google and Facebook both provided identity services, for example, users could verify with both, and account penetration would require simultaneous penetration of both Google and Facebook. A user with particular concerns could require verification through multiple devices simultaneously.

One remarkable aspect of today’s online security is how close we are to this, how many of the pieces of it we have, without reaping the benefits. Our phones verify us with a combination of biometrics and a secret that is never shared, protecting both types of data with high effectiveness. Our phones implement password managers, and to some extent, the sum of those passwords represents a digital identity. Particular accounts, such as Google, may store additional secrets on our phones to enable MFA. The phones have location services and the ability to update servers with our locations, our gait, even our heartbeat.

In spite of all that capability we carry with us every day, someone who gains access to our credit card number can still use it to make fraudulent purchases, harming vendors and service providers if not us directly. Additionally, if we lose our phones, we may have to re-verify with a range of service providers, and the extent of that range is limited only by the amount of inconvenience we’re willing to tolerate in the pursuit of security.

All of the necessary pieces are in place, and all the technologies are known. The primary obstacles aren’t technical issues but business ones. Large tech players haven’t fixed these problems not because they can’t, but because they don’t see it in their private interest to do so.

How corporate culture excludes me, or why I never bothered interviewing at Amazon, Facebook, and Google.

by Leave a Comment

Every so often, I read an article breathlessly talking about the greatness of any particular meeting process and how everyone should aspire to such a meeting culture. And it got me thinking about my career choices.

I never interviewed at Google earlier in my career because they wanted my grades. My grades undersold my abilities because of my neurodiversity. And anyone who thought they measured me accurately was probably a place I never wanted to work at. And then there was this whole class of companies founded by ex-googlers that I associate with that culture and choose not to even consider. I didn’t trust them to judge me based on deeply flawed metrics.

I never interviewed at Facebook because I saw how they had these all-night hack-a-thons, and I was about to be a dad, and I never wanted to work at a company where spending the night hacking was a requirement for my job. In particular, I knew that the people who could spend time would build social relationships that endured, and I would be formally excluded from them or have to choose not to see my kid. There were these youtube videos of the corporate culture and the hacking, and I was like – Nope, not for me. At Zynga, we introduced hack-a-thons, and folks wanted to mimic the all-night nature, and I remember throwing a temper tantrum in an exec meeting, and we agreed that they would not be all-night extravaganzas.

And so similarly, if a recruiter said ex-Facebook, I was like – probably not the kind of place where wanting to be with your family and in your bed will be a place for career growth.

I never interviewed at Amazon and had panic attacks when folks at VMware wanted to introduce Amazon-style meetings because, again, because of my neurodiversity, I knew it would exclude me. But I was always curious whether it was as bad as I thought.

Amazon came to VMware to have a meeting complete with a six-pager, and I was in the meeting because they needed my buy-in to get some technical work done. The meeting was precisely the disaster I expected. I was expected to read a print-out (I use text-to-voice) in some standard font (I use Open Dyslexic). I was expected to read this wall of text in 20 minutes (I scored in the bottom 3% of the population in reading and retaining information) and offer comments.

Did the Amazon employee ask if this works for me? No. Instead, I was expected to endure. So I did my best. What made it worse was that my other coworkers happily did this work. That was humiliating. I felt I was being denied access and treated as a semi-human being. And I realized I could never work with Amazon as a partner or an employee. And companies would copy this style, and I would be pushed out of those companies as well. And that this megalith of a corporation was about to kick me out of the tech industry. Talk about motivation to make VMware win.

And if I was in a charitable mood at the beginning of the meeting, I was livid by the time we got to my chance to talk. I was like, damn if they are this inconsiderate to partners, I can’t begin to imagine how they treat employees, and I thought – nope, not for me. They came to get my support, and their meeting structure did the wrong thing.

In the end, I was determined to ensure Amazon failed. I wanted them to fail. Because if their success meant more of these meetings, by god, I would do everything in my power to fight back.

I will observe their proposal was ridiculous, and it never went anywhere, more or less for the reasons I described in the meeting. I wasted an hour reviewing a ridiculous proposal and being humiliated.

A win, I guess?

Every so often, someone asks why it is such a big deal, and Sheri Byrne-Haber (disabled) explains it oh so well. Because those meetings are designed to make people like me less effective, I wonder if that’s the intent. But I know me, and mine are not welcome, and I will leave.

Now, I will observe that I had to grow up as well. When I got into a position of power, I, too, wanted meetings that only worked for me. And my team blew a gasket and demanded change, and so I changed begrudgingly. But it was only when I got into a shouting match with a senior executive who wanted the text, not a draining meeting, and we argued. I learned that to be inclusive, I had to accommodate his needs as much as he had to accommodate mine. So I used voice-to-text to produce documents, and he used tools to create text I could process via audio systems.

The point is that any rigid system is a rigid system. And its purpose is to force people to conform and, if that conformance doesn’t work for them, to force them out. And if you want to work with all the best and the brightest, then you need to be flexible and creative and not rely on rigid rules of how meetings must work.

https://www.linkedin.com/pulse/why-amazon-six-page-meeting-process-doesnt-work-sheri/

57 architecturalist papers: generated software and created software

by Leave a Comment

Over the years of writing and architecting software systems, I saw the emergence of automatically generated layers of software that were nominally supposed to eliminate a class of software development or even software developers as an employment category.

Borland C++ 3.1

For example, in the 1990s, higher-level programming languages and dynamic compilation of code were supposed to eliminate the need for low-level programming. I remember taking classes in the early 1990s where garbage collection and object-oriented programming would unburden software developers from resource management.

At the same time, we had the emergence of frameworks that promised further productivity gains, like the Distributed Computing Environment and CORBA for distributed systems. As for building Graphical User Interfaces, we had the proliferation of toolkits like OpenView, X-Motif, Borland’s Application Framework, Java Swing, Microsoft .NET Framework, etc.

Over the next 30 years, the story stayed more or less the same. Every year there would be some new technology that would, once again, unburden the software developer from toil.

And, of course, if you happen to be the person paying the software developer, make it possible to have fewer of those developers working for you.

I recall in 1994, a Professor at Brown University remarked to the students in computer science his concern that there would not be enough high-paying jobs in the field. The emergence of OO programming, and the ability to have people work remotely, would reduce the value of a degree in computer science over the long haul.

And he was right, but not quite in the way he anticipated.

All those technologies did reduce the need for highly skilled technology experts to build much of the world’s software. In 2003, I had to write my own time-series database for a product. In 2022, I would spend more time deciding which one to use.

The available software building blocks are more powerful, flexible, and compelling than at any point in the history of computer science.

And that has enabled more people to write software that I could not have conceived of in shocking time frames.

That point was brought home to me in 2009 when a small team of engineers at Zynga leveraging AWS, PhP, and a home-grown NoSQL database built a game on a distributed systems architecture with over 2000 servers. And none of them had a Computer Science degree.

And yet we still invest in software, building new products, and hiring new engineers.

Why?

Competition. It turns out that when everyone can make a scalable game without a CS degree, the magic is not in making the scalable game but in building a scalable game. And when there is a large pool of competing choices, and success only goes to the best game, marginal and incremental improvements result in huge wins.

This brings me to my somewhat non-obvious conclusion: software products’ success is a function of how easy it makes the lives of people who do something useful. The problem is that when anyone can do something, it has no commercial value.

For example, when everyone can build a scalable game, the value of a scalable one is zero. It’s like with special effects; when everyone can have great special effects, the value of special effects in a movie is zero.

So what did the framework and toolkits make easier? They made the expected stuff easier, so you can work on the element differentiating your product or game. You still need to invent a new game mechanic, and that new game mechanic, because it is new, requires new code. And, here’s where it gets fun; eventually, you find new game mechanics that require new hardware, which requires new frameworks, which require…

And if the demand for new games is significant, then the need for new game mechanics is large, and the demand for new software is powerful. We all wonder why we are not more productive while playing games that are more beautiful, engaging, and deeper than ever before.

We live in a world where the amount of software is growing. And more people are writing software than ever before.

And that’s a good thing. But the need for new innovative software remains. And that innovative software is not generated; it is created. And that new software must be created when the boundary between the digital world and the human or physical world changes.

56 architecturalist papers: what if the different people never came?

by Leave a Comment

In my post yesterday, I attempted a rebuttal of Mr. Jassey’s arguments about the superiority of in-place work.

My meta-argument was that a diverse workforce has a diverse set of needs. And that what works for some does not work for others.

A great meta-question is – so what? Does it matter to an organization if some folks are excluded, and some are not? Is cultural homogeneity an asset or a disadvantage?

When a company asserts specific lived experiences are valid and others are not, they explicitly articulate that specific human experiences suit their culture and others are wrong.

Those whose experiences fit into the wrong category self-select against that company. And over time, if it is successful, the company views its choices as correct because those who don’t fit in never show up.

They never bothered to come and talk to you because they saw that you thought they were terrible and that rejection of their existence meant there was no space for them there. But it’s just working from home? Right? Except it isn’t. Where does that end when you assert your lived experience is correct and the other person is wrong? Why should it stop at how we work together? Why not how we dress? How do we address each other?

“We chose to do it this way, and we are successful, therefore we were right.”

What was a choice, a preference, becomes because of the group’s success criteria for articulating what is generally correct and generally incorrect for all human beings.

And again, so what?

Diversity and diverse perspectives matter because otherwise, you suffer from groupthink. If everyone does it one way, and all successful people do it that way, and no one objects to how we are doing it, then it must be the right way to do things. And what happens, invariably, is that someone else figures out a better way to do it, and then the group collapses as the new way triumphs over the old course.

Several years ago, I wrote about Usain Bolt. Usain Bolt was too tall for a sprinter. That was the orthodoxy. But a coach decided to ignore orthodoxy, and guess what? Usain became the greatest sprinter of all time.

Or Ervin Johnson was too tall to be a point guard. But another coach saw his passing and skills and decided that the advantages he brought to the game because of his vision and height and reaching from the point position outweighed the benefits he got to his team as a front-court player.

History is littered with companies, teams, and organizations convinced they were doing things the right way. By asserting that their lived experience was correct over other people’s lived experiences, they drove talent away. That talent then thrived elsewhere.

A rebuttal of some of the points of Mr. Jassy

by 2 Comments

Note I come to work five days a week. I like being in the office. I enjoy being around human beings. And have been leading remote teams for almost 20+ years. In that time, my groups have delivered over 10 billion dollars in actual money, many multiples of that in shareholder value.

Mr. Jassy wrote this memo to his staff, telling them why he would require everyone to come to work.

Update from Andy Jassy on return to office plans (aboutamazon.com)

So he starts with this:

It’s easier to learn, model, practice, and strengthen our culture when we’re in the office together most of the time and surrounded by our colleagues. It’s especially true for new people (and we’ve hired a lot of people in the pandemic); but it’s also true for people of all tenures at Amazon. When you’re in-person, people tend to be more engaged, observant, and attuned to what’s happening in the meetings and the cultural clues being communicated. 

Except if you have ADD and sitting in a meeting where you cannot sit still gets in the way. Or you are deaf, can’t hear what people are saying, and don’t have zoom translate what is being said in real-time.

For those unsure about why something happened or somebody reacted a certain way, it’s easier to ask ad-hoc questions on the way to lunch, in the elevator, or the hallway; whereas when you’re at home, you’re less likely to do so

I am surprised by this one. The emergence of technology like Slack or the DM function in Zoom has democratized communication. Finding time for a senior leader to ask them a question is tricky. And the senior leader in the elevator can walk away from you. But when you send them a message, they tend to take the time to respond because they can do so asynchronously.

In the more productive brainstorm sessions I’ve been a part of over the years, people get excited and blurt out new ideas or improvements to prior proposals, quickly advancing the seed of an idea, and leading to the broader group getting energized and feeling that it’s onto something. This rapid interjecting happens more often in-person because people feel less inhibited about jumping in or even interrupting sometimes. 

I am a cis-gendered white man at the apex of my profession, which worked for me. How many women and non-white and non-cis-gendered men have we trashed over the years because they didn’t speak up?

My first meeting back in person was the same old guys talking with all of the women and others silent.

It was so bad we had to introduce zoom to deal with the problem.

Serendipitous interactions help it, and there are more of those in-person than virtually

This one surprised me. The number of in-person interactions is an order of magnitude smaller than the interactions with people digitally. Just count the volume of people you contact over email, the number of people your email goes out to, etc. Mr. Jassy feels that is low-value communication.

Regardless, Mr. Jassy has made his choices, declared that he believes they are correct over other people’s lived experiences, and can force his employees to choose to work for Amazon or leave.

55 architecturalist papers: Facebook v Zynga Microchannel, the Apple Store, D&D, and killing platforms

by 2 Comments

IBM PS2 60 and 80 side-by-side

In 1981, IBM introduced a revolutionary computer that radically transformed the tech industry, the IBM-PC.

What made the IBM PC revolutionary was its open-system architecture and the royalty-free nature of its use.

Specifically, anyone could create an IBM clone, and anyone could develop software for said IBM clone and make money without paying a dime to IBM.

IBM hated that. So their solution was in 1987 to create the Microchannel PS/2. The intent was to create a new divergent market of PC’s that no one could make clones of.

What happened was that the market split between PS/2 and EISA, and the PS/2 became a failed computer system and a historical artifact and a warning to those who would want to close an open platform.

In June 2008, Apple introduced the App Store. The Apple App Store was the first massive commercial success of a cell phone app store. And it created a standard for how to take royalties from creators. And it was a huge success. So much so that we have had lawsuits that are still going through the court systems to determine the boundaries and limits of the market rules Apple can enforce in the market they created.

In June 2010, Facebook saw the success of the Apple Store and decided they wanted a piece of the Zynga action. Facebook had created an Open Platform, and that Open Platform enabled Zynga to grow like wildfire. But Facebook wasn’t making any money from Zynga. And so, there was a stand-off between Zynga and Facebook. Zynga and Facebook signed a deal that required Zynga to hand over 30% of its revenue to Facebook. That 30% revenue haircut killed Zynga.

Facebook’s thesis was that by creating a new currency, Facebook Credits, that Zynga customers would use, they would increase the total number of people who had digital money, and thus more money would be spent. The thesis failed, and Zynga suffered.

In both Apple and Facebook’s cases, they saw intellectual property creators who used their platforms as free-loaders. In Apple’s case, the tax was declared up-front, so you knew what you were getting into. In Facebook, it was an after-the-fact revision that destroyed a business.

But what happened to said intellectual property creators? As my son recently said – “why does mobile gaming suck?”

Gaming is a hit-driven business. One hit makes all the money, allowing you to make the next game. 30% is a massive tax, restricting the money you have to make further investments. And so the gaming industry has moved back to open platforms, ironically, the Windows PC.

This now brings us to the recent decision by Hasbro to introduce OGL 1.1. A lawyer covered it well here – https://medium.com/@MyLawyerFriend/lets-take-a-minute-to-talk-about-d-d-s-open-gaming-license-ogl-581312d48e2f

If you parse the Lawyer’s responses, it boils down to trying to create an Apple-like App Store for D&D content. Essentially, you hand over your financials and content for a smaller slice of the pie for the right to play.

In effect, it destroys the open ecosystem that D&D had. For example, suppose I have a website with a random generator of D&D content. That random generator is illegal.

Now Hasbro is betting that people play D&D and don’t care about the open content and that the creators will have to suck it up and deal.

Except, and this is a big exception, that isn’t true.

The iPhone was a singular technology with no ability to be replaced. Folks used Facebook because they wanted to connect with friends. Those platforms had value outside of the gaming industry. D&D is a game and a platform.

But it’s an extraordinary game where the player and the GM create content while playing. And the GM can adapt content from other gaming systems to their game. And the GM can adapt rules to their game.

In short, I expect the TTRPG community to discover the power of system-neutral gaming. And that the internet will increase with systems that allow you to convert to the gaming system of your choice. Except for the new restricted one.

My take, and it’s hopeful, is that D&D will continue, but tabletop role-playing will finally escape the long shadow of its creator and his original game.

But going back to software architecture and platforms, it’s always tempting to control a market, and there is a lot of value in doing that. But when you extract a lot of money from a market, you eventually kill the market. And over time, those creators whose businesses are hits will move to open platforms.

54 architecturalist papers: career progression for those of us who are under-represented

by 1 Comment

I intend to write a lot about why you should go and read this https://hackingcapitalism.io/

And then realized I could write very little.

It would be best if you read Kris Nóva’s Hacking Capitalism because it is a highly well-written document by an outsider on how to function and succeed in the alien tech world. It would be best if you read it because Kris Nóva (@nova@hachyderm.io) is a fantastic person who is impressive and distills a lifetime of insight into a tight actionable document.

And that’s the only reason you should read it. And if you need me to tell you why beyond that, here’s my life story.

At NetApp circa 2002, a great manager named Jonathan Crowther took me to lunch to discuss my career. It was the first time any manager had done that.

I was frustrated because my career had stalled out. And I didn’t understand why.

And he smiled and said that the problem was that I only approached people as functional units that needed to solve my problems or I needed to solve theirs. That I never treated people as people.

And I remember staring at him, and then he explained, “when you come in on Monday, you never say – how was your weekend? It’s always – I need this.”

I stared at him. I suspect I am somewhere on the spectrum. And it was a moment of clear revelation.

I spent the entire weekend thinking about this one conversation. A key benefit I had was that I had done a lot of acting, enjoyed D&D, and had an absurdly high executive function. And so I came on Monday morning and acted. I have a youtube video where I talk about this here.

What resonates with Ms. Nóva’s document is that she detaches herself from the reality of understanding why and accepts that things are and that she must devise rules on how to win based on that.

I lack her precision of English, wit, insight, and, to be quite frank, experience. And I wish, when I was 22, she had given me this document.

So who should read it? If you were not born in the United States, you were not a white CIS-gendered man, and you are not privileged, you should read this document.

The United States is an alien country. Its rules are strange. And its culture is derived from 300+ years of interaction between property rights, the evil belief that labor should be enslaved, that extreme Christian Dogmas are fundamental, and a sense of supreme righteousness and contempt.

For example, let’s consider ownership.

A year ago, I had an opportunity to talk to a Chinese woman, from mainland China. And she was talking about ownership and how she was struggling with her career because of that. And I realized she was talking about ownership in a way that sounded like a colorblind person talking about colors.

And then we talked about “what is ownership in the USA.” In the USA, ownership of property is a sacred right. When you own something, you own it. You can do whatever you want to it. And nobody can tell you otherwise. A considerable amount of the conflict in this country is putting boundaries on the ownership of people by others and the limits of ownership of common goods by individuals at the expense of the group. The idea that the country protected property rights from others and the government was eye-opening to her. Her life experience in Xi’s China and property rights was not that. And as we talked about ownership, she said – “oh, so that’s what ownership means.”


I love this country, and as a Greek from a village, a Canadian from Montreal surrounded by more small-town Greeks who fled Greece due to a civil war, a world war, and a junta, the culture of the United States and beliefs frequently leave me perplexed.

Hacking Capitalism is a guide to making this system work for you without understanding the why and just understanding its weak points.

Ubiquity and the ventilator and adding 15 years to everyone’s life

by Leave a Comment

My father is Prof. Charis Roussos. He’s currently 78 years old. And he’s a legendary figure.

To make Prof. Roussos’s impact tangible, let’s compare how set designers imagined the future of hospital medicine.

In 1966, the set designers of the sci-fi show Star Trek imagined the medlab of ~2254 looked like this.


The hospital was a single doctor with a digital chart.

But in 2009, the set designers for the film Star Trek imagined the medlab in 2233 looking like this.

Notice the ventilator, the machines surrounding the hospital bed? Prof. Roussos made them a necessary part of medicine.

But why did this ventilator become so ubiquitous in just 43 years?

Before his research, there was this thing called “shock.” Patients would enter “shock” and die. And the why was a mystery.

But its existence meant that a whole class of things we take for granted are impossible. Surgeries that could save your life but might lead to “shock” were very risky.

But was shock?

He and his teams proved that shock happened when blood pressure dropped.

Let me explain.

Typically you have a specific volume of oxygenated blood pumping through your body. We all know that the heart is a muscle, and we all know about the lungs, but what we didn’t realize was that there were some equally if not more essential muscles, “the lung muscles or the thoracic muscles.”

The body needs oxygen; otherwise, it dies. And without the lung muscles moving, you don’t breathe and die.

But for the lung muscles to move, they need oxygen as well.

Under normal conditions, this is not a problem. There is enough oxygen coming in and enough blood going around for everyone.

But when a patient is under distress, and there is less blood to go around, the brain needs to make some tradeoffs. Well, it can’t stop breathing, so it keeps the lungs working and starves everything else, including itself.

But how much blood do these lungs need? Well, about 25% of the total blood flow.

So what was shock – less blood being pumped through the heart meant less oxygenated blood for everyone, which meant that vital organs started to fail, which led to death. To deal with that, the brain then tells the lung muscles, “WORK HARDER,” and sends even more blood, and the lungs work harder, and vital organs get further starved. . Until they are exhausted, and the brain realize it can do nothing, gives up, and the patient dies.

Shocking.

These lung muscles were the key to keeping people alive!

But how to do that? Well, if the brain only cared about oxygen, what if we fed the patient oxygen or artificially moved the lungs?

And that’s where ventilators come in.

What if we used ventilators to move the lungs artificially, allowing more blood to flow to other parts of the body?

That simple idea saved people who would otherwise have died and enabled surgeries and procedures that in the past were too risky.

You can hear him talk about this here.

52 architecturalist papers: the different flavors of availability

by Leave a Comment

In a recent sequence of discussions on availability at work, I realized that there are different flavors.

But first, a history lesson.

Before virtualization became a thing, the physical data center physically existed. And its destruction, although possible, was extremely rare.

The destruction of an entire physical data center was extremely rare, and we referred to it as a “meteor strike or nuclear war.”

For practical purposes, availability was about dealing with physical component failures. For example, if a server had a disk, and the disk died, could the application on the server keep running?

As hardware components became increasingly resilient and the cost of hardware plummeted, the weak link became the software.

Because rearchitecting software to become intrinsically highly available was impossible, new software frameworks came into existence.

One of the most impressive was vSphere-HA because it said the following

  1. If you have highly available storage and enterprise storage was highly available
  2. If you have shared storage, and enterprise storage was shared
  3. And you capacity connected to that storage, all of your applications will restart after 5 minutes.

vSphere HA and its more boutique solutions more or less solved the availability problem.

And the power of vSphere-HA to systematically solve the HA problem across an entire application fleet greatly expanded the value of VMware’s technology value proposition.

Except, a new availability problem emerged.

In the past, a single software could not destroy a single data center or all data centers. The scope of the blast radius of an operator error was quite limited.

But with virtualized infrastructures and cloud infrastructure, a single button click can destroy an environment in minutes.

For example, at Zynga, an operator error caused by a lousy user-experience design deleted the production Cityville deployment. Cityville had over 10 million Daily Active Users (DAU) and 30 million Monthly Active Users (MAU).

Why was this possible?

Because in the past, you couldn’t delete a server, the applications in the server, the network connections associated with that server, and the data contained in those servers with a single button click in minutes because it would require logging into hundreds of systems. Why? Because the nature of the systems relied on multiple administrators of infrastructure coordinating to either provision or delete infrastructure.

Let’s use an analogy from social media. Before Facebook, I couldn’t share the news with millions of people using a few button clicks to use an analogy. I had to talk to a lot of people. And so, the blast radius of my news was quite limited. Facebook made it easier to share good information like I have a new child very quickly, and sad news like a friend died very quickly. It also made it possible to share fake news and lies very quickly as well. The same technology could be used to do both. And the societal damage of that rapid dissemination of phony information is confirmed.

Virtualization and later cloud moved the entire data center that the application depended on into a single database. The single database allowed agility to grow, shrink, and adjust infrastructure demand to meet the changing application demand. It also made this database the single most significant vulnerability to your environment.

And so software engineers rightfully focused on how to make that database available. But the problem is that no amount of software available can solve operator error, and worse, mistakes that destroy the database in its entirety.

As more and more effort was put into making the central system more robust and available, the overall design became more and more fragile.

But what kinds of errors? Remember, the physical data center rarely goes away in its entirety, but several servers? Those fail all of the time. Worse, as the servers become less available to become cheaper and more disposable because technologies like vSphere HA or cloud-native 12-factor application styles expand, their failure rate increases.

And this cheaper hardware that is more disposable is possible because of the very same centralized databases that when they fail destroy everything. Antifragility makes it clear that the more and more you make everything depend on a single system, the more and more the entire system becomes fragile.

So we have this peculiar phenomenon that the entire virtual infrastructure of the data center resides in a single piece of software that if it fails, everything is gone. And the thing that can cause it to fail is not under the control of software: things like Human error, or data corruptions in parts of the stack that are unknowable, or ransomware, etc.

Without any personal internal knowledge, the Facebook DNS debacle is a great example. The software allowed the network engineering team to make widespread infrastructure changes across Facebook in minutes. And human error combined with technology errors (just another word for human errors) resulted in the entire Facebook infrastructure is unavailable.

IT operators understand this point, as did DevOps teams and the SRE teams. And this is why they refrain from having a single system control their entire infrastructure. The scope of a single error is why those groups like to have more k8s clusters instead of one large one. Or why backup admins have backup software from a different vendor from their primary storage.

Centralizing things allows for unprecedented agility and failure.

What to do?

My take is threefold.

The first is that the era of centralized databases that control the entire infrastructure is closing. Instead, we will have many databases. In the k8s space, the idea of a few large k8s clusters is an anti-pattern. And this is why I am such a huge fan of products like Tanzu Mission Control. They line up with the future.

The second is that those centralized databases will have to be built leveraging blockchain systems that protect against byzantine errors and, in particular, human errors.

The third is that the ability to recover from those database errors in a timely fashion will become a critical differentiator. A once-in-a-decade event is terrible if it happens on a Friday and a company ending event on Black Friday. Software stacks that make that recovery quick and easy will win.