Looks accurate — no edits needed.

Opening

Reader, let me start with a piece of news from last week.

Bloomberg reported on August 22, citing sources, that Nvidia had notified some of its major customers that it’s raising prices on servers equipped with AI chips by more than 15%. The increase kicks in for systems shipping in early next year, and it covers both the flagship Grace Blackwell and the next-generation Vera Rubin. The reason was memory prices.

Bloomberg called the situation unusual. Nvidia is considered one of the most profitable companies in the semiconductor industry, and yet here it is, unable to absorb rising costs and passing them on to customers instead.

Add one more number and the picture comes into focus. According to one analysis, for the Vera Rubin, which began shipping this month, 62% of total system cost is memory.

Let me cut to the chase. The center of gravity in AI chip competition is shifting from compute to memory. And that shift is creating a different kind of opportunity for NPUs—chips that have, until now, been left in Nvidia’s shadow.

HBM Now Has a Floor and a Ceiling

Let me sort out the landscape first. Recent brokerage reports have started drawing AI memory as three distinct layers.

HBM is the high-bandwidth memory we already know. It’s the ultra-fast working memory sitting right next to the accelerator — with HBM4, that’s 36GB per stack and bandwidth exceeding 2.8TB/s. It’s the fastest, but also the most expensive, and it’s constrained by capacity and packaging limits.

HBF¹ is the layer below it. Built on NAND flash, its published specs target up to 512GB per package with 0.4–3.0TB/s of bandwidth. The idea is to pair HBM-class speed with a much bigger capacity. But NAND’s inherent latency and durability issues remain unresolved, and this is still an early-stage technology.

SOCAMM2 is the layer above. It repurposes low-power DRAM from smartphones into a server module — compared to RDIMM, the standard server memory module, it delivers more than double the bandwidth and over 75% better energy efficiency. Because it connects via connector rather than solder, it can be pulled out and swapped in.

Here’s a timeline detail worth flagging. Reports commonly cite “up to 256GB,” but 256GB was the spec Micron announced last March and began sampling to customers. The actual mass-production mainstream is 192GB — Samsung Electronics started mass production first in March using its 1b process, and SK hynix followed in April with its 1c process. What’s more interesting, actually, is news going the other direction: as low-power DRAM supply tightened, reports emerged in June that Nvidia was pursuing a plan to cut SOCAMM2 capacity from 192GB down to 96GB. It’s a single-source report, so it can’t be treated as confirmed, but if true, it’s telling — a case of a finished-product spec being scaled back because the components simply weren’t available.

Let’s also look at what goes wrong if you try to get by on HBM alone. Running a large model for inference means loading the entire model weights and conversation history into memory, and at 36GB per stack, you hit the ceiling fast. You might think stacking more layers would solve it, but the higher you stack, the more heat, power draw, and yield loss you get — and the price climbs right along with it. In the end, “more of the fastest memory” is a dead end, both physically and economically. So it becomes rational to split things up: frequently accessed data stays on HBM, while the larger, less urgent chunks move down a layer.

Why is the memory stack growing more layers? Because AI’s center of gravity has shifted from training to inference. Training is a short, intense burst of work, so the fastest memory is the obvious answer. Inference, though, is a different game — running a large model continuously, cheaply, for a long time. What that requires isn’t top speed but the right blend of speed, capacity, and power cost. That’s why a structure once built around a single type of memory has splintered into tiers.

The keynote SK hynix delivered at FMS 2026, the memory industry conference held in the US on August 4, captured this shift exactly in its title: orchestrating AI infrastructure with tiered memory for the agentic AI era.

The Chip Nvidia Folded and the Chip It Bought

But the company that most dramatically confirmed this shift wasn’t a memory maker—it was Nvidia. I think it’s worth placing two events from the past year side by side.

First, the chip it folded. In September 2025, Nvidia unveiled Rubin CPX, an inference-only accelerator. The key detail was that it used GDDR7 instead of HBM. Nvidia claimed it delivered 7.5x the AI performance of the previous generation even without the expensive HBM, and it was built specifically for million-token-scale processing and video generation. It amounted to a declaration: “Inference can run on cheaper memory.”

Then, at GTC 2026 this past March, Nvidia dropped the chip from its roadmap—just 6 months after unveiling it.

Second, the chip it bought. The Vera Rubin platform, also unveiled at GTC 2026, is built from 7 new chips: the Vera CPU, the Rubin GPU, the NVLink 6 switch, the ConnectX-9 SuperNIC, the BlueField-4 DPU, the Spectrum-6 Ethernet switch, and the Groq 3 LPU².

That last name is the crux of the matter. In this architecture, Nvidia split the workload between GPUs and LPUs. GPUs handle large-scale data computation, while LPUs handle ultra-low-latency response. Nvidia said this combination boosts inference throughput for trillion-parameter models by up to 35x. Compared with the previous-generation Blackwell, the numbers were striking: the GPU count needed to train a mixture-of-experts model³ drops to a quarter, inference throughput and performance-per-watt rise 10x, and cost per token falls to a tenth. CEO Jensen Huang told the audience that inference had reached an inflection point.

Put the two events together and here’s how I read them: Nvidia tried to build in-house the answer to “GPU-plus-HBM isn’t the best combination for inference”—and folded. Then it bought that same answer from outside.

Here’s why this matters: up to now, the NPU camp has been the one making this argument. Their logic was that GPUs were originally general-purpose chips built for graphics, so using them for inference creates waste—and that inference-specific designs have an edge in power efficiency and latency. Now the market’s dominant player has effectively conceded that logic, encoding it directly into its own product lineup.

The English draft is accurate, complete, and consistent with the Korean source—no meaning distortions, all numbers present, glossary terms correct, no Hangul, and structural elements match. No edits needed.

So What Happens to NPUs

Here’s the first signal. On August 4, SanDisk and SK Hynix released the first HBF standard specification through the OCP⁴, and Google and Tenstorrent joined the standardization process as consortium members, taking part in technical validation and spec-setting.

Google builds TPUs, and Tenstorrent builds NPUs. That means the room where memory specs get decided now has accelerator companies sitting in it, not just memory companies. Samsung Electronics and Micron are still watching from the sidelines.

The reason is simple. A chip’s performance in inference workloads is no longer determined by the chip alone. Half of it comes down to which memory tier the chip was designed around. So these companies are showing up early, to help write the tier specs that will favor their own chips.

Let’s look at the domestic picture too. Rebellions has launched Rebel Quad, built on HBM3E, and is running field trials in Japan, Saudi Arabia, and the US, while FuriosaAI has landed an adoption case with LG’s EXAONE and set a target of $100 million in revenue this year. Both companies have gone after the inference market armed with power efficiency and low latency.

But what I’m watching closely is which memory these companies actually run on. Rebel Quad uses HBM3E. That’s a sensible choice for securing performance, but it also means a large share of the cost structure is tied to the same components Nvidia uses. That 62% figure from earlier is at work here too. When you’re using the same expensive components as everyone else, there’s little room left to win on price. If HBF or a low-power DRAM tier actually opens up, Korean NPUs would get a chance to build a different cost structure — even before the performance race begins.

That said, I should be honest about the other side of this too. If Nvidia folds inference into its own platform as well, then from a customer’s standpoint, sticking with a familiar ecosystem is the rational move — the cost of learning a new software stack can outweigh whatever you save on power. Korean NPU firms have been targeting inference precisely because competition there was relatively lighter, and now the dominant player has walked straight into that space. Opportunity and crisis are coming out of the same event.

But last week’s price-hike news shakes that balance a little. If server prices climb more than 15%, the cost of sticking with the status quo keeps rising. Switching ecosystems, by contrast, is closer to a fixed cost you pay once. That pulls forward the point where the two cost curves cross. In fact, commentary on this very report suggested that a surge in server prices could accelerate Big Tech’s shift toward in-house chips.

That same commentary flags the other side too, though: whether companies scale up in-house chips or keep using Nvidia, they all still need to secure enough memory from Samsung Electronics, SK Hynix, and Micron. It’s one more confirmation that the center of gravity in this calculation has shifted from chips to memory.

So here’s how I see the future of NPUs. The contest ahead won’t be decided by benchmark scores, but by which layer of this fragmenting hierarchy each player claims. Korean companies, in fact, are already branching out — steering clear of an all-out data-center fight and moving into edge, on-device, and physical AI instead. Just as computing once splintered from mainframe dominance into PCs and mobile, the outlook is that AI semiconductors will similarly fragment into inference, edge, and on-device tiers.

Oswarld’s Lens

There’s one thing I kept confirming while drafting GTM strategies: the point where a later entrant beats the leader is rarely the spec sheet. It’s usually the standard, or the distribution.

You might think hitting harder in the same ring wins the fight, but the side that sets the ring’s size and the number of rounds ends up with the real advantage. That’s why I read Google and Tenstorrent joining the HBF standard body not as a technical-collaboration story, but as a positioning story. They went in early to set the width of the road their own chips will run on.

I think domestic Korean NPU companies need the same kind of move right now. Rather than proving a few percentage points better performance-per-watt than Nvidia, securing a seat at the table where the next generation of memory-hierarchy standards gets decided is the bigger lever in the long run. If they’re going to ask the government for support, I’d argue that’s a more efficient use of funds than R&D subsidies.

One caveat: this isn’t optimism on my part — it’s a conditional observation. Standards participation is just an admission ticket. Whether developers actually use the software that runs on top of it is a completely separate battle.

Closing

If I compress today’s story into three lines, it’s this.

First, the reason Nvidia raised server prices by more than 15%, and the reason 62% of Vera Rubin’s cost is memory, are the same story. The bottleneck has shifted from compute to memory. Second, that’s why memory is splintering from a single HBM standard into a layered structure that includes HBF and SOCAMM2 — and why Nvidia, after shelving its own attempt to build a memory-free inference chip in-house, brought in an LPU from outside instead. Third, in that gap, the battleground for NPUs is shifting from raw chip performance to staking a claim on the memory hierarchy’s standards.

So the next time you read AI chip news, I’d suggest not just looking at the performance multiples — look at what kind of memory the chip was designed to assume. These days, that’s where more of the real information lives.

If you’ve ever tried to cut inference costs in your own work, I’d love to know what got in your way the most. Was it chip prices, electricity bills, or the cost of migrating software? Let me know in the comments — it might become material for the next issue.

This piece analyzes industry structure and is not investment advice. The companies and figures mentioned are based on publicly available data and media reports as of their respective dates; some are drawn from exclusive reporting or industry estimates and are not confirmed facts.

---

💬 Tell me in the comments what ate up the most of your inference costs · 📨 Share this piece with a colleague who follows semiconductors or infrastructure

---

References & Further Reading

Primary sources

• Ian King et al., “Nvidia Customers Notified About AI-Related Price Hikes Above 15%”, Bloomberg, 2026. 8. 22. Link ··· This is where today’s piece starts. If the paywall blocks you, you can get the gist from Korean outlets’ citation coverage. Pay less attention to the size of the hike than to the word “unusual.”
• Sandisk & SK hynix, “Sandisk and SK hynix Advance Global Standardization of High Bandwidth Flash with Release of First OCP Technical Specification”, 2026. 8. 3. Link ··· The original text explicitly names Google and Tenstorrent as consortium participants. This is the single most important line in today’s piece.
• Kim Young-ho, “[GTC 2026] Nvidia Unveils ‘Vera Rubin’ Specs, 3x Compute and 10x Inference Efficiency Leap”, Electronic Times, 2026. 3. 17. Link ··· You can check the list of 7 new chips directly here. Why the Groq 3 LPU is on that list is today’s central question.
• Lee Ki-jong, “Nvidia’s ‘Rubin CPX’ Launch in Doubt, No Memory or Substrate Orders”, TheElec, 2026. 5. 27. Link ··· This is the record of a shelved chip. Coverage of an announced product quietly disappearing is rare, which makes this a valuable source.
• SK hynix Newsroom, “SK hynix Begins Full-Scale Mass Production of SOCAMM2 192GB”, 2026. 4. 20. Link ··· The primary source for the bandwidth and energy-efficiency figures compared against RDIMM.

Background

• Eugene Investment & Securities, AI memory comparison data (graphic via Maeil Business Newspaper’s MK PLUS) ··· The table in section 1 of today’s piece comes from here. Note that the capacity figures are spec ceilings, so you should distinguish them from actual mass-production timing.
• Han Jung-ho, “[On the Ground] Domestic NPUs Take On Nvidia’s Dominance, Betting on Inference and Physical AI”, ZDNet Korea, 2026. 5. 28. Link ··· You can hear the strategies of Rebellions, FuriosaAI, and Mobilint straight from the companies themselves.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. HBF (High Bandwidth Flash): Memory that stacks NAND flash in many layers to approach HBM-level speed while offering far greater capacity. Think of it as a layer filling the gap between HBM and SSDs. It began with an agreement between Sandisk and SK hynix in August 2025, and the first standard specification arrived in August of this year. ↩

2. LPU (Language Processing Unit): A processor specialized in language-model inference, particularly in cutting response latency. It’s made by the US company Groq — spelled and made by an entirely different company from Elon Musk’s xAI chatbot Grok, so don’t mix them up. ↩

3. Mixture of Experts (MoE): Rather than having one giant model answer every question, the model is divided internally into multiple expert modules, only some of which “wake up” for any given query. This cuts computation at the same model size, which is why many recent large models adopt it. ↩

4. OCP (Open Compute Project): A consortium where multiple companies jointly develop open standards for data-center hardware specifications. Once a spec is set here, both component makers and chip companies design their products to match it. ↩

Found this useful? Share it

Copy linkXLinkedInThreadsKakaoTalk네이버 밴드