Beyond NVIDIA's Moat

The Inference Wars

To be clear, NVIDIA’s moat remains secure for the coming decade. But beyond that horizon, its grip on the AI market is beginning to loosen, and the key measure is the long-term strength of the CUDA moat.

As I explained in my analysis of the Open-Weight Alliance, this was also one of the main forces behind NVIDIA’s push to open up the model market. NVIDIA does not want value and control to concentrate at the model layer. It wants competition there, while the industry remains structurally locked into its infrastructure layer.

In this analysis, I break down the forces beginning to erode NVIDIA’s moat and explain how the evolving structure of the AI industry could gradually weaken its control.

Every economic transaction in AI, when you strip the wrapper off, is denominated in tokens. A token is the fundamental unit — a piece of text, a subword, an image patch, an audio frame — and everything in the industry is a mechanism for producing tokens, storing them, moving them, or charging for them. Model quality is tokens generated per query that are correct. Model speed is tokens per second per user.

Model cost is dollars per million tokens. Training economics is dollars per training-run token. Cloud gross margin is the spread between the cost of producing a token and the price of selling one. Even valuation ends up back at tokens: a $514B backlog is a promise about future tokens.

Follow me on The AI Supercycle as well!

This is not a metaphor. It is the unit. And once you accept that, the whole industry sorts itself into a small number of moving parts: mechanisms that make tokens, mechanisms that gate tokens, and mechanisms that own the customer relationship at the moment a token is priced.

The previous guides in this series — The Fed and the Mint, Kimi K3, Where Google Sits in the Map of AI, Three Tolls on One Road, What the Wallet Sees — each took one slice of the token economy and read it to the ground. This piece pulls them into one map. It is meant to be a reference: the shape of the thing, the players in each region, and the bottlenecks binding each region right now.

The organizing insight is one Andrew Feldman surfaced almost in passing on the Mad Podcast, and once you see it the whole industry looks different: a token has a lifecycle, and the lifecycle has three stages that are as different from each other as manufacturing, warehousing, and retail.

• Training. The weights are made. Sparse linear algebra at planetary scale, one time per model, amortized across every subsequent token that model will ever generate.

• Prefill. The input pass. When you type a prompt, the model has to read it — process every token of context into an internal representation the generation step can use. This step is parallelizable: all the input tokens are known, so the machine can chew through them simultaneously.

• Decode. The output pass. The model generates the answer one token at a time, each new token conditioned on all previous tokens. This step is sequential and cannot be parallelized. Token 500 cannot be computed before token 499. It is a chain, and the chain cannot be broken.

Each of the three has its own physics. Each has its own bottleneck. Each has its own set of players. Each has its own cost curve. And — this is the punchline the rest of the piece is going to unpack — the frontier is moving deeper into the stage where the bottleneck binds hardest, which is decode. Reasoning models spend most of their compute in decode. Agents multiply decode by the number of turns. RL now folds decode into training itself. The memory wall the industry hit in 2024–25 was not a temporary supply crunch. It was the frontier discovering which stage of the token lifecycle it was actually spending its money in.

Let me walk the three stages, then the players, then the economics.

Follow me on The AI Supercycle as well!