wrong tool

You are finite. Zathras is finite. This is wrong tool.

77 architecuralist: the rapid evolution of AI scale-out

by Leave a Comment

The other day, I remarked that it was possible that Apple’s R&D budget for 2025 equaled the entire R&D budget for tech in 1980. At the time, I thought I was exaggerating, but I then asked Gemini (see below) to go do some research, and what I uncovered was that I was not.

When looking at the current technology trends and moat of a company like Anthropic or OpenAI or any of the AI vendors, it’s easy to assume that they have a very large moat. The problem, though, is a moat is a function of money and resources, and the scale of the resources necessary to turn a scale-up problem into a scale-out problem is staggering.

But when you think about the total amount of dollars that exist in engineering investment and the willingness of pretty much everybody to throw whatever it takes to solve the problem, you realize that it becomes entirely possible to turn what should’ve been a 40-year problem, namely how do we go from scale-up AI to scale-out AI, to a four to five-year problem, and that’s what we’re seeing. Effectively, we’re seeing the attack of scale-out that took 40 years to complete, taking less than five.

What that tells you is that the era of the frontier models may be coming to a premature close. I don’t begrudge the end of that era because as the investment increases and the scale-out increases, the value declines. So, do I think it’ll go to zero? No, but I don’t think it will be where the action is in a year or so.

Here is the budget analysis from Gemini

Comparing tech-only R&D in 1980 to Apple’s 2025 R&D budget makes the contrast even more stark.

Apple’s FY2025 R&D spend was $34.55 billion.

In 1980, total U.S. industrial R&D in the technology sector—which encompassed information technology, mainframe/minicomputers, semiconductors, telecommunications, and office equipment—was roughly $8 billion to $12 billion in nominal terms.

Apple alone in 2025 spent 3 to 4 times the nominal total R&D budget of the entire 1980 tech industry.

Nominal vs. Inflation-Adjusted Comparison

Perspective1980 Total Tech R&DApple FY2025 R&DTakeaway Nominal Dollars~$8 billion – $12 billion$34.55 billionApple’s budget today is 3x–4x larger in raw dollars. Real (2025) Dollars~$32 billion – $48 billion$34.55 billionApple’s single budget roughly matches the purchasing power of the entire 1980 tech sector.

Key Historical Drivers

  • Fewer, smaller players in 1980: In 1980, industrial tech R&D was dominated by a few giants (IBM, AT&T Bell Labs, Texas Instruments, Xerox PARC). Even IBM—the undisputed titan of the era—spent roughly $1.5 billion on R&D in 1980.
  • Apple’s current scale: Apple’s $34.55 billion budget is driven by proprietary silicon, AI infrastructure, custom displays, sensors, and operating systems—demonstrating how single mega-cap tech firms today deploy capital at a scale that exceeds entire national industries four decades ago.

The Future of AI and Scale-Up Computing

by 1 Comment

One of the things that has been bothering me when I look at AI is that the advantages of the current foundation model hinge on their use of the most expensive part of computing infrastructure, which is global memory with low latency. What do I mean? If you look at what Fable does, at the core of it, it’s looking at billions of parameters or trillions of parameters, and then trying to infer the next token from those trillions of parameters. That is expensive because the hardware to provide that kind of speed and efficiency and bandwidth is some of the hardest hardware to build.

Historically, there was another kind of computing that had the same problem, and that was scale-up computing, essentially. How do I build a bigger, faster processor that would get better and better over time? What happened was that at some point, we realized we couldn’t scale up processors anymore. Instead, we had to scale them out by having more cores. In parallel, we discovered that trying to scale up applications was incredibly hard, so we had to scale out applications as well. That trend started in 1970 and ended in 2024 when scale-out applications became just the norm of how you build applications. Very few people are doing anything that is pure scale up.

AI is at, let’s say, year 4 of this trend where scale-up applications are what frontier models provide, and then scale-out is what all of the various tricks, techniques, tools, and mechanisms that exist for everything else. Now, the question if you’re an AI fanboy is not whether the foundation models do or do not win. That’s actually orthogonal to my thought, which is that what if the real problem is that the 40-year event of the disruption of scale-up computing suddenly takes not 40 years, but 5 years? The question there is, if it does, what does that mean?

What it means is that the value of the kind of technology that enables the Fable-like systems doesn’t meet the value because the scale-out systems, which rely on less efficient hardware, provide good enough results, or even worse, provide enough good enough results such that the investment necessary to deliver the next high-end model and the hardware necessary to support it are simply not as valuable. This trend disrupted the entire RISC processor market. It disrupted one of my early employers, SGI.

I’m tempted to say this might take 40 years, but I’m also cognizant of the fact that if I look at the total investment in R&D today and compare it to the total investment in 1980, I wouldn’t be surprised to discover that Apple’s R&D budget in 2025 equals the total tech R&D budget of 1980. Maybe I’m off, but it doesn’t feel a ridiculous thing to say.

Hardware not Software is Eating the World: Len was right.

by 1 Comment

In 1996, Len Widra, a Principal Engineer at SGI, and I got into a heated argument over the importance of software vs. hardware.

I was a kid out of school with an ego to match. Of course, I thought I knew everything. The crux of our debate was whether the software was a relevant technology or important technology.

Len’s observation was that software was irrelevant or something like that. Hardware, he observed, was the important technology.

As a software engineer, this was infuriating. As a computer scientist, this wasn’t very kind. How dare he say that a bunch of silicon was more important than my code?

It’s been almost 18 years, and I’ve learned more.

What I have learned is that new software rarely, if ever, displaces old software unless some new hardware shows up. New hardware shows up, and that new hardware makes the old software irrelevant or obsolete.

There is one interesting caveat. Some software applications are really dependent on the quality of the algorithms, and as the algorithms improve, the software gets obsoleted regardless of the underlying hardware changes. In many cases, the emergence of new algorithms creates new hardware that helps obsolete the old software.

For the vast majority of software systems, however, that’s not the case.

When you are looking for a new opportunity in the technology space, what you need to look for is where new hardware is emerging. If it is sufficiently different, that new hardware will obsolete the old software that was tied to the new hardware creating new opportunities for new software.

A mouthful.

A few examples:

(1) the emergence of x86 servers created the opening for Linux. Before x86 servers were a reality, the UNIX vendors owned the entire software and hardware stack. When x86 became good enough, a new software stack could win because the software used new hardware.

(2) Flash in the storage industry has truly created a massive disruption, enabling many different kinds of software stacks.

(3) Merchant (aka Broadcomm) Silicon is disrupting the networking space that reminds me of the x86 disruption.

(4) ARM processors made mobile computing plausible.

Maybe my favorite example is this picture from TIOBE Software that measures the popularity of programming languages. TIOBE measures popularity – not use or lines of code – and has been doing that analysis for many years:

2014-11-28_0849

You look at the chart, and you realize how slowly programming language popularity changes except for one programming language: Objective-C. The popularity of a single programming language changed dramatically not because it was good or bad but because of a single new hardware platform that enabled new software.

The hardware disrupts because it enables software that was impossible before. The carefully calibrated trade-offs that are baked into a system are tossed into the sea with new hardware. When you want to look for disruptions to your business, never look at software; software is irrelevant; look at the hardware …

 

The systems disruption in networking

by 4 Comments

As I spend more time learning about Networking, I begin to realize that there are significant disruptions happening and they are not as obvious as “SDN” or “Open Flow”.

One of the most significant differences between a networking device and a server is that the data path stays entirely inside  of a custom ASIC instead of running on a general purpose multi-processor.

Why is this important?

When a customer buys a device, what they are buying is the data path, the rest of the product is a second order purchasing decision. In a networking device, the data path is entirely in the hardware, and therefore – although software is important – it’s significantly less important than the ASIC that processes the packets.

A networking company – for all of the discussion about software – is a hardware company that builds an ASIC that processes packets. They live and die by the success of their hardware and by hardware I mean ASIC.

Historically a networking company built it’s chips and sold a device.

The disruption that is happening in the networking space is that – increasingly – a large chunk of the networking ASIC’s are being built for increasingly important segments by vendors that sell to multiple networking companies.

This is very similar to what happened in the late 90’s when Intel started killing off all of the custom microprocessors in the server market.

A simplistic and silly view is that as a result of this disruption is that everyone is going to go and build white-box switches and routers all of a sudden. For some customers wh have the depth of skill and expertise to build their own networking gear this will happen  but for the same reasons I am bullish on storage I am bearish on this happening broadly. The majority of the market will continue to buy gear from networking companies.

The disruption that Intel created in the 1990’s was that companies like SGI that built everything soup-to-nuts had to suddenly go from hardware companies that had software to a systems company.

Customers used to buy SGI hardware to get access to the massive number of processors or the high end graphics engines and they didn’t really care about the software running on them too much. The selling point was the hardware.

As the number of vendors that could deliver the hardware that SGI produced customers went from buying whatever hardware SGI produced running whatever software SGI put on the hardware, to thinking about the whole system and that created more options for the customers.

In effect, the decision criteria went from being what SPECint or SPECfp a MIPS processor had to what is your TPC-C and TPC-D number and whether you were a key partner’s of Oracle.

So what is a systems company then?

A systems company delivers a device that is balanced between hardware, software and packaging. The balance of the three components creates a unique selling proposition that the customer and market is willing to put a premium on.

Systems companies emerge in markets where the hardware is supplied by a small number of vendors and differentiation is created in how you combine the hardware and use the hardware in software and create a package that the customer can buy.

The challenge for systems companies is that sophisticated customers can build the same product from the same components and this creates a pressure to innovate in lots of areas outside of the core components.

The challenge for a hardware company as it makes the shift to a systems company is that the mindset of how you build a product has to change. Whereas in the hardware centric world – the hardware is built and the software guys have to figure out how to make it work, in a systems world there is a fine balancing act between the two. And whereas in a pure hardware world the hardware performance was the be-all of your differentiation, the systems company has a combination of attributes in software, packaging and hardware that create the unique differentiation.

In fact, perversely, how you build software and the easiest way to build software becomes more important over time than any specific hardware platform. And the choice of components becomes more important than any specific component.

The challenge for networking companies is that system design is not where their core focus has been over the last 20 year.s

This change in how you build networking devices from – here’s an ASIC with software to here’s a complete balanced and well integrated system is going to be disruptive to how companies do business.

Now that I said all of that, let me observe, I don’t think my employer, Juniper, or Cisco are in danger of getting disrupted per-se. I just think that the way systems will get built is going to be very different and that that change is going to be very interesting.

And that in my mind is where the real disruption in networking is happening, everything else may just be noise.