Super Computers

One of the things that has been bothering me when I look at AI is that the advantages of the current foundation model hinge on their use of the most expensive part of computing infrastructure, which is global memory with low latency. What do I mean? If you look at what Fable does, at the core of it, it’s looking at billions of parameters or trillions of parameters, and then trying to infer the next token from those trillions of parameters. That is expensive because the hardware to provide that kind of speed and efficiency and bandwidth is some of the hardest hardware to build.

Historically, there was another kind of computing that had the same problem, and that was scale-up computing, essentially. How do I build a bigger, faster processor that would get better and better over time? What happened was that at some point, we realized we couldn’t scale up processors anymore. Instead, we had to scale them out by having more cores. In parallel, we discovered that trying to scale up applications was incredibly hard, so we had to scale out applications as well. That trend started in 1970 and ended in 2024 when scale-out applications became just the norm of how you build applications. Very few people are doing anything that is pure scale up.

AI is at, let’s say, year 4 of this trend where scale-up applications are what frontier models provide, and then scale-out is what all of the various tricks, techniques, tools, and mechanisms that exist for everything else. Now, the question if you’re an AI fanboy is not whether the foundation models do or do not win. That’s actually orthogonal to my thought, which is that what if the real problem is that the 40-year event of the disruption of scale-up computing suddenly takes not 40 years, but 5 years? The question there is, if it does, what does that mean?

What it means is that the value of the kind of technology that enables the Fable-like systems doesn’t meet the value because the scale-out systems, which rely on less efficient hardware, provide good enough results, or even worse, provide enough good enough results such that the investment necessary to deliver the next high-end model and the hardware necessary to support it are simply not as valuable. This trend disrupted the entire RISC processor market. It disrupted one of my early employers, SGI.

I’m tempted to say this might take 40 years, but I’m also cognizant of the fact that if I look at the total investment in R&D today and compare it to the total investment in 1980, I wouldn’t be surprised to discover that Apple’s R&D budget in 2025 equals the total tech R&D budget of 1980. Maybe I’m off, but it doesn’t feel a ridiculous thing to say.

Over the last 15 years, there have been two useful heuristics for figuring out where computing is going.

When I want to look at how applications are going to be built, I look at games. After all games are at the forefront of creating new kinds of digital experiences, and the need to push the boundaries of how we entertain ourselves is a crucial to create new revenue and sales opportunities.

When I want to look at how infrastructure is going to change, I look at what people want to do in the super computer space.

Two nights ago, I had the marvelous opportunity to hear a talk that was a discussion of Physics and Big Data. As a software infrastructure guy, at the end of the day I like to think about how to build systems that enable applications, I have been wondering if Big Data was going through a bigger-faster-stronger phase or whether there were new intrinsic problems.

And the answer is yes to both.

Clearly we need systems that can do more analysis faster, store more data at cheaper costs, etc.

What was not as obvious was that exponential increase in transistors coupled with the disruptive trend of 3D printing was going to enable:

A proliferation of very sensitive distributed sensors that need to be calibrated and whose data needs to be collected.
The ability to find even weaker signals in the data.

In effect, we were going to be able collect more data faster and because of that we will be able to find things that we could not find before. And solving 1 and 2 on it’s own are very interesting problems that can keep me busy for the next 10 years …

However there are some new problems that come out of that:

We will need to be able to find new ways to explore data and track our exploration through the data.

We will need ways to combine the datasets we create. After all as more and more sensors get created, and sensors get cheaper, the ability to combine data sets will become crucial. And as the scale of the datasets grows, an ETL becomes less realistic.

And then to make this all more interesting, there is some thought that the way we collect data itself may create signals and that meta-analysis of the data will be required. And how you do that is an interesting problem on itself. And how do you create systems can correct for that…

My head has sufficiently exploded. Turns out that just making things go faster isn’t the only problem worth solving…

The Future of AI and Scale-Up Computing

Like this:

Physics and Computer Games and Big Data

Like this:

Share this:

Like this:

Share this:

Like this: