How a Memory-Saving Algorithm Tanked Memory Stocks

Opening

Subscriber, something curious happened in Wednesday’s U.S. markets. While the Nasdaq 100 climbed, memory semiconductor stocks moved in the opposite direction and fell. SanDisk -5.7%, Western Digital -4.7%, Seagate -4%, Micron -3%. The trigger was a compression algorithm called TurboQuant, published by Google Research. The effect reached the Korean stock market almost immediately too. As of 6 PM, when I’m writing this newsletter, semiconductor-related companies like Samsung Electronics, SK hynix, Hanmi Semiconductor, Leeno Industrial, and Wonik IPS were mostly down.

“A technology that lets AI use less memory” — read just the headline, and it sounds like bad news for memory makers. But what this technology actually shrinks is the temporary working memory on the GPU (the KV cache), not the HBM¹​ or DRAM modules plugged into servers. There’s a gap between the signal the market read and the signal the technology is actually sending — and behind that gap sits a bigger question spanning the entire AI hardware stack. Today I want to walk through both the gap and the question. (Honestly, if you don’t care much about the mechanics, feel free to skip straight to “Why Did the Market React?”)

What TurboQuant Actually Does

Let’s start with the technology. When an AI keeps a conversation going, it needs to remember what was said earlier — and to do that, it stores information in a temporary memory called the KV cache²​. The longer the conversation runs, the more exponentially this memory grows, making it one of the main drivers of AI service costs. TurboQuant is an algorithm that compresses this memory as small as possible while preserving nearly all of its content. No additional training or fine-tuning required.

Here’s an analogy for the core idea. The data an AI normally stores has wildly varying magnitudes — some values are large, some small, and the distribution is uneven. Data like this is hard to compress efficiently. The first stage of TurboQuant, called PolarQuant, applies a random rotation to this data, flattening the distribution of all values into something uniform. It’s like shuffling a pile of unevenly sized luggage until everything comes out roughly the same size — which then packs neatly and efficiently into identically sized boxes. In the paper’s mathematical terms: after rotation, each coordinate follows a beta distribution and becomes nearly independent of the others, which allows optimal scalar quantization³​ to be applied to each coordinate individually.

But there’s another problem here. AI doesn’t just store information — it constantly performs comparisons of similarity between stored pieces of information (inner product operations). The paper proves that a quantizer optimized for MSE (mean squared error) introduces a systematic bias into these comparison values. That’s why a second stage is needed: QJL (Quantized Johnson-Lindenstrauss). After the first stage compresses the bulk, QJL corrects the remaining residual error with just 1 additional bit. This two-stage approach eliminates the comparison bias entirely.

The paper’s experiments show that at 3.5 bits, quality is essentially identical to the original model, and even at a more aggressive 2.5 bits, quality loss is minimal. On Llama-3.1-8B-Instruct, the average LongBench benchmark score was nearly unchanged from the uncompressed version (50.06 vs 50.06), and the model maintained 100% accuracy on a Needle in a Haystack test at 104,000 tokens. The compression ratio exceeds 4.5x.

What’s especially striking is the speed. Traditional product quantization (PQ) took about 240 seconds to index 1,536-dimensional data; TurboQuant took just 0.0013 seconds — roughly a 180,000x difference. That’s because TurboQuant is a data-oblivious, online method that doesn’t need to learn a codebook tailored to the data.

One thing worth flagging, though: the “up to 8x speedup” figure that Google’s blog emphasized was measured for a specific step — attention logit computation — against a JAX baseline. It is not 8x end-to-end inference throughput. And the “6x memory reduction” figure differs subtly between the blog and the paper — the paper is more conservative, stating “4.5x or more.” Numbers get packaged differently depending on the announcement channel, and that’s always worth watching when reading tech news.

Why Did the Market React?

The market’s logic is simple: “If AI only needs one-sixth the memory, doesn’t that mean memory demand drops?” Memory stocks had already run up so much this year (SanDisk is up over 200% year-to-date) that there was also an appetite for an excuse to take profits. But stepping back, KV cache and HBM both use the word “memory,” yet they operate at entirely different layers.

The KV cache is what an LLM temporarily stores in GPU memory to avoid recomputing previously calculated attention data as a conversation continues. When the context window stretches to 1 million tokens, the KV cache alone consumes about 512GB for a Llama 3 70B model — four times larger than the model weights themselves. That’s why KV cache compression is one of the hottest research topics in AI infrastructure right now.

HBM demand, by contrast, comes from bandwidth bottlenecks across the entire training and inference pipeline. TrendForce estimates HBM demand will grow 70% year-over-year in 2026, and Bank of America projects the 2026 HBM market at roughly $54.6 billion (up 58% year-over-year). SK hynix, Samsung, and Micron have all stated that their 2026 HBM production is effectively sold out.

Here’s a simple analogy: TurboQuant improves how you organize notes on your office desk. HBM demand is about needing more offices in the building itself. Organizing your notes more efficiently doesn’t reduce the building’s demand for space. If anything, better-organized notes let a single office get more done — which might make you want to build more offices, not fewer.

Not Just Google’s Story — Nvidia’s KVTC

There’s context here that’s easy to miss. Google isn’t the only one compressing KV caches. At the same ICLR 2026 conference, Nvidia is also presenting a technology called KVTC (KV Cache Transform Coding). It applies the transform coding principles used in JPEG image compression to the KV cache, claiming up to 20x compression — and up to 40x in certain cases. (Seriously, you again?)

Google’s TurboQuant (6x compression, no training needed) and Nvidia’s KVTC (20x compression, requires pre-calibration) take different approaches, but they’re solving the same problem. And Nvidia plans to integrate KVTC into its own inference framework, Dynamo.

Why does this matter? Once KV cache compression technologies are deployed in production, the same GPU can handle longer contexts and more concurrent requests. For AI providers, that means lower inference costs. Does higher efficiency mean lower demand? Not necessarily — think back to the Jevons Paradox⁴​ we covered in a previous issue. But the market is reading this technology from a somewhat different angle.

Oz’s Lens

What interests me more than TurboQuant itself is how the market reacted to this news. Because this isn’t just a story about memory stocks.

Zoom out, and the entire AI hardware stack right now is facing the same question. Nvidia posted unprecedented FY2026 results — $215.9 billion in revenue, over 55% net margin — yet its stock is trading roughly 11% below its October peak. Micron is in the same boat. Two days ago it reported its best-ever quarterly results (revenue of $23.86 billion, gross margin of 74.9%), but the market’s reaction focused on whether it can sustain over $25 billion in capital expenditure. GPUs are falling, DRAM is falling, NAND storage is falling.

The real question the market is asking is “is infrastructure investment at this pace sustainable?” The combined 2026 capex guidance of the Big Four (Microsoft, Meta, Alphabet, Amazon) totals $650-700 billion. That ranks among the largest amounts of private capital ever deployed toward a single purpose in human history. A growing debate among investors is whether the returns on this investment can be justified — and that’s the context behind what we’ve been calling the “AI bubble.”

Here’s the framework I use as a GTM strategist: every technology infrastructure cycle has a “build phase” and an “optimize phase.” In the build phase, the strategy is “deploy first.” In the optimize phase, the strategy shifts to “maximize the efficiency of what’s already deployed.” TurboQuant, Nvidia’s KVTC, and hyperscalers building their own chips (Google’s TPU, Amazon’s Trainium) — all of these are signals of the optimize phase.

Does that make it a bearish signal? I don’t think so. The optimize phase isn’t the end of growth — it’s growth maturing. What changes is simply how the market prices things in. In the build phase, the strategy was “buy everything.” In the optimize phase, you have to figure out who benefits from this efficiency gain and who bears the cost.

The fundamentals of the memory market right now remain strong. HBM supply is sold out through all of 2026, and BofA estimates the market at $54.6 billion (up 58% year-over-year). But once the narrative “algorithms are replacing hardware” lodges itself in investors’ heads, the higher a valuation runs, the more sensitively it reacts. If SanDisk is up more than 200% year-to-date, a single research paper can knock -5.7% off it.

The key is separating time horizons. Software optimizations like TurboQuant affecting the growth rate of hardware demand is a story for 2027 and beyond. The memory supply shortage in 2026 is a matter of physical fab construction and yield — not something algorithms can solve. When the market conflates these two timelines, that confusion is both the opportunity and the risk.

Closing

One more thing before I wrap up. The original TurboQuant paper (arXiv:2504.19874) was posted on April 28, 2025 — almost a year ago. Someone spent a year refining this technology, posting it to the Google Research blog, preparing it for the ICLR 2026 presentation, and finally, it shook the market. It’s interesting to think about what accumulates in the time between an idea leaving the lab and it moving Wall Street’s stock prices.

To sum up: TurboQuant is a meaningful technology that takes AI inference efficiency up a notch. But the reason memory stocks fell today isn’t this one technology alone — it’s because the market has started reading the signal of a broader phase shift, from “build” to “optimize,” across the entire AI hardware stack. If you can understand the different layers of the technology and separate the timelines involved, you can make better judgments amid this kind of volatility.

If a paper from a year ago could shake today’s market, what will the papers coming out of labs right now shake a year from now?

⚠️ This newsletter is not investment advice regarding any specific stock. Please make investment decisions based on your own analysis and consultation with a professional.

References & Further Reading

• Zandieh, A., Daliri, M., Hadian, M., & Mirrokni, V., “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate”, arXiv:2504.19874, 2025. To be presented at ICLR 2026. : The original TurboQuant paper at the heart of today’s newsletter. You can check the mathematical proofs of the two-stage compression structure and the experimental results directly here. Table 1 (LongBench results) and Figure 4 (Needle in a Haystack results) are the key parts.
• Google Research, “TurboQuant: Redefining AI efficiency with extreme compression”, 2026. : Google Research’s blog post summarizing the paper for a general audience. Note that there are subtle differences in how numbers are presented between the blog and the paper, so refer to the paper for precise figures.
• Łańcucki et al., “KV Cache Transform Coding for Compact Storage in LLM Inference”, ICLR 2026. : Nvidia’s KVTC paper. Comparing its approach with TurboQuant’s gives you a fuller picture of the KV cache compression landscape.
• Investing.com, “MU, WDC, SNDK fall: Why Google’s TurboQuant is rattling memory stocks”, 2026.3.25. : An article covering today’s memory stock decline.
• SK hynix, “2026 Market Outlook: SK hynix’s HBM to Fuel AI Memory Boom”, 2026.1. : Helps you understand the structural demand in the 2026 memory market. Covers demand forecasts for HBM3E and HBM4.
• Fortune, “Rampant AI demand for memory is fueling a growing chip crisis”, 2026.2. : An in-depth piece on how AI is reshaping the entire memory market.
• CNBC, “Even a $1 trillion forecast can’t break Nvidia out of a 2026 funk”, 2026.3. : An analysis of why Nvidia’s stock is stuck despite record-breaking earnings. Good for understanding the market psychology of the “build-to-optimize phase transition.”
• VentureBeat, “Nvidia says it can shrink LLM memory 20x without changing model weights”, 2026.3. : Covers Nvidia’s KVTC technology from a practical deployment angle. Good supplementary reading for understanding the differences from Google’s TurboQuant.

The author, Kwangseob Ahn, is a professor of business administration at Sejong University and lead consultant at OBF (Oswarld Boutique Consulting Firm). He teaches statistics and data analysis — business data management and business analytics — while leading GTM and AI strategy consulting in the field, designing the seam between technology and business. He has published academic research on a memory architecture for AI dialogue systems (HEMA) and runs Daily Arxiv, a daily curation of global AI papers. He holds a master’s from Korea University’s Graduate School of Technology Management and a KMBA. He is the author of Homo Brainless: The People Who Outsource Their Thinking.

Footnotes

1. HBM (High Bandwidth Memory): Ultra-fast memory stacked vertically right next to an AI chip (GPU). Because it transfers data several times faster than regular DRAM, it has become a core component in AI training and inference. ↩

2. KV Cache (Key-Value Cache): Temporary memory where an LLM stores previously computed “keys” and “values” during a conversation. Without this cache, the model would need to recompute everything from scratch every time — so the longer a conversation runs, the more GPU memory this cache consumes. ↩

3. Quantization: A technique for reducing precise decimal numbers (e.g., 32-bit) down to fewer bits (e.g., 4-bit, 3-bit). This shrinks file size, but reducing too aggressively can degrade the information. ↩

4. Jevons Paradox: A phenomenon where improving the efficiency of resource use actually increases total consumption rather than decreasing it. Named after 19th-century British economist William Stanley Jevons, who observed that improvements in coal efficiency did not lead to reduced coal consumption. ↩