36 architecturalist papers: multi-tenancy and isolation primitives

Over the years, I have struggled with multi-tenancy. The definition of it feels fluid and imprecise.

And so, in a desperate attempt, I wrote some incoherent thoughts down. I’ll take another stab at being precise.

Multi-tenancy has two elements; the first is that there are a set of users who have access to a set of resources. And the users and their rights to those resources are a technical problem. What is the hierarchy of users? What is the hierarchy of permissions? What rights do they have on the resources?

There is a separate problem about the unit of isolation of resources. If two tenants have access to resources, the immediate follow-up question is how much isolation does the resource provide between tenants?

For example, a physical server is a unit of isolation. A virtual machine is a unit of isolation. And so on.

Let’s dig into the notion of isolation a bit. Isolation is used to provide security, performance, and availability isolation. A great example is a black site. A black site is completely isolated for reasons of security from the rest of the world. A dedicated server running a single application is used to isolate that application from any other application that may affect it.

At one extreme level of isolation, each computer system is isolated from another.

There is no multi-tenancy at all.

This model is costly.

And so consumers ask – can I trade off some security, performance, and availability for cost optimization?

In short, can I group a set of applications on a set of resources where the isolation isn’t as strong to get some sharing?

Time-sharing systems were, in many ways, the first such system when they replaced batch systems.

And that’s where things got confusing.

Because the only real isolation is hardware isolation, the minute the hardware is shared, two applications share something.

And although hardware vendors and software vendors have gone to heroic lengths to ensure isolation, the reality is if you are sharing something, you are not isolated.

Now let’s introduce a second person, the hardware resource provider. The hardware resource provider wants to offer resources of varying degrees of isolation. And wants to bill accordingly. The resource provider’s challenge is that the consumer may not be willing to pay for resources if they are too expensive. In other words, the consumer of the resources is willing to trade off some isolation for cost savings.

And now we introduce a third player, the manager of the resources that the consumer of resources uses.

The resource provider gives up some resources, the manager divides them up amongst consumers, and the consumers consume them.

The resource provider provides some resources that are isolated. The manager associates those resources into something that consumers can use, and the consumers consume them.

This simple model assumes that a resource that is isolated can be controlled and offered to a consumer and is at the right granularity.

But what if it isn’t?

And thus, we have the fourth player in our dance, a person who takes isolated resources from resource providers that they then offer to a manager of resources that offers them to consumers. Let’s refer to these resources as “virtual physical resources” (VRP).

And that’s fine, except in one very annoying use-case.

What if two different sets of virtual resources are sharing the same underlying physical resources?

If the VRP’s assume that is not the case, then the entire system falls apart.

Why? Because the guarantees of isolation assume that the underlying hardware has some degree of isolation (remember, the VRP is built on isolated resources).

Huh?

Imagine VRP-1 assumes that a socket is isolated and it controls the resource. VRP-1 can make assumptions about security and performance. If VRP-2 is also using that resource, VRP-1’s is being lied to. And that’s fine until VRP-1 and VRP-2 have conflicting objectives.

At that point, the thing that actually controls the socket has to arbitrate between VRP-1 and VRP-2.

And we’re still good. What if VRP-1 assumed it had control of the socket and gave out virtual sockets, and VRP-2 assumed that it had control of the socket and was giving out a full-time slice of the socket.

The problem is that the isolation unit was a socket, and there was no way to arbitrate between the two VRP’s.

So?

The real challenge in multi-tenant systems is the proliferation of VRP’s. Some VRP’s can not be layered on top of each other because they rely on isolation primitives at a lower level that loses critical information about how the VRP wants the resource to be used.

An obvious solution is to create a new VRP, called VRP-12, the union of VRP-1 and VRP-2 and can effectively share a system between the two users. I worked on such a system at SGI. It was a batch scheduler called Miser.

The problem with that approach is that as the VRP’s proliferate, there is a cost in pushing them down a layer.

What’s the cost?

There are three costs. The first is the system resources for the control plane itself. The second is the complexity of using the control plane. And the third is that as the control plane becomes more complex and supports more resource isolation, the control plane needs isolation.

At some point, it becomes simpler to have different VRP’s operating independently rather than on a single shared system.

But wait, there’s more.

Because a VRP only requires a layer that isolates hardware, and every VRP isolates hardware, VRP’s proliferate and get layered on top of each other. There is no final VRP layer.

So?

What it boils down to, in my mind, that multi-tenancy really means three things:

1. A set of physical resources that need to be shared
2. Some VRP system being used to share the physical resources (1) that uses mechanisms to isolate the hardware and provides isolated resources coupled with a mechanism to assign those resources to an individual group that can use them.
3. Some mechanism for grouping users into hierarchies’.

And what makes multi-tenancy even more complicated is that in any system, there are multiple VRPs.

And so the challenge of multi-tenancy is really about how you enable this ever-expanding hierarchy of VRPs such that they don’t require their own isolated hardware, and each consumer, resource provider, and resource manager can effectively do their job. And understand what guarantees they are getting and providing.

To make this a little bit more concrete, let’s consider k8s.

Some developer wants to deploy a k8s application and request a certain amount of CPU cycles.

To do that, there must be physical CPUs.

Those physical resources are partitioned by some hypervisor that hands out virtual CPUs to an OS.

The OS, in turn, hands out threads that run on the vCPUs.

The k8s system in turns groups threads into things called Pods.

The k8s system is deployed in such a way that each tenant has his own namespace.

Each layer, the hypervisor, the OS, and k8s is providing virtual resource pools. Each layer has a distinct manager.

A multi-tenant system is the combination of those virtual resource pools and the physical hardware resources.