Opening

Reader, at midnight Wednesday (Korea time), Alibaba’s Qwen team open-sourced the weights for Qwen3.8-Flash-Next. It isn’t a finished flagship — it’s a preview model, releasing the design meant for the next-generation Qwen4 so it can be stress-tested in public first. Total parameters come to 125 billion, but only 6 billion actually activate per token; on top of that sits a 51-billion-parameter N-gram¹ memory that pre-remembers frequently recurring token combinations.

What stopped me in the spec sheet wasn’t a performance number. It was a single line saying that those 51 billion parameters live not inside the GPU but in the server’s ordinary memory. I’ve written twice that “AI’s real bottleneck isn’t the compute chip but HBM”² — and here was an attempt to route around that bottleneck not with a chip, but with a design.

Let me give you the conclusion up front: the real story here isn’t performance, it’s a redesign of the cost structure. The memory tier where parameters live has dropped down a level, every number needs to be read against its own denominator, and the two Flash models released the same night diverged not in price but in licensing.

---

Why Do the 51 Billion Parameters Live Outside the GPU?

Every language model starts by pulling one vector per token out of a dictionary. Qwen bolted on a second dictionary: a table with 20 million slots, keyed by the combination of the previous two or three tokens. It hashes the last few tokens into an address, pulls the corresponding vector out of the table, and mixes it into the second layer’s representation. Because this is a lookup rather than a multiplication, the compute per token barely increases.

The key question is where this table lives. Because the address is deterministic, knowing the token alone tells you exactly which slot to read, so the value can be fetched from CPU memory ahead of time while the GPU is still computing the first layer. This prefetch³ overlaps with the first-layer computation, hiding the transfer latency. You can see the result in vLLM’s deployment recipe: the GPU-side FP8 checkpoint stays at 172.78GiB, and one line gets added on top — “host memory of 51GB or more for N-gram offloading.”

Comparing this to Mixture-of-Experts (MoE)⁴ makes the point clear. MoE also cuts per-token compute, but all 512 experts still have to sit in GPU memory. A lookup table doesn’t. The model card’s line — “efficient for memory-constrained accelerators, more offload-friendly than MoE” — captures exactly this difference.

The idea’s real origin isn’t China — it’s Google. Last February, Google researchers’ SCONE paper opened with the line “main memory and SSDs are far cheaper than accelerator memory,” and proposed keeping N-gram embeddings off the accelerator entirely. By the paper’s own pricing table, system memory runs about $2 per GB versus $0.1 for SSD. Gemma 3n, released the same year, used a similar trick, keeping per-layer embeddings on the CPU. As far as I know, this is the first time that lineage has been scaled up to a 125-billion-parameter model carrying a 51-billion-parameter table on top.

The report is upfront about its own limits, too: growing the N-gram vocabulary keeps pushing training loss down, but downstream accuracy saturates at some point.

Attention that reads only 2,048 tokens out of 1 million

The layer structure of Qwen3.8-Flash-Next. A set of GDNs plus one QSA form a block; the four circles represent the widened residual stream. (Source: Qwen)

The second component targets the cost of long context. Of the 48 layers, 36 are Gated DeltaNet (GDN), a linear-attention layer. As context grows longer, it’s compressed into a single fixed-size state, so cost scales only linearly with length. The remaining 12 layers — one in every four — carry real attention, and that’s where Qwen Sparse Attention (QSA) sits.

In QSA, a lightweight indexer slices the context into small blocks, scores their importance, and picks at most 512 blocks — or 2,048 tokens — per token to attend to in full precision. Even inside a 1-million-token context, each attention layer actually looks at just 2,048 tokens. This belongs to the same lineage as DSA, which DeepSeek released last year, but it cuts the indexer’s cost — which otherwise grows with the square of the length — by compressing the keys. The report’s numbers, measured at a 1-million-token context, show a 7.6x speedup in prefill⁵ and 4.9x in decode, but that’s the speed of a single attention kernel, not the whole model.

The third component is the residual stream⁶, drawn in the diagram as four circles. The conveyor belt connecting layers has been widened from one lane to four; each component draws out only what it needs through a read gate and puts it back through a write gate. The report’s emphasis is on stability: in stress tests run at 4 times the optimal learning rate, the old architecture’s loss spiked frequently while the new one stayed stable, and during actual training there wasn’t a single loss spike.

The fourth piece is the training recipe. They used Muon for matrix weights and AdamW for everything else, and after refitting the scaling laws to raise both batch size and learning rate, they dropped batch warmup entirely — on the grounds that warmup burns 18.8% more optimizer steps.

The Denominator in “One-Ninth”

The official announcement said “one-ninth the training cost of Qwen3.7-Plus,” but the report’s own language is different. Training FLOPs⁷ are roughly one-ninth, and if you break that down, it’s one-third the active parameters multiplied by one-third the training tokens. On base-model benchmarks, it leads in 8 of 14 categories and trails in 6, by as much as 2.6 points. The accurate translation is “comparable performance at one-ninth the FLOPs,” not “outperforms across the board.” FLOPs and GPU-hours are different units, and the report says nothing about whether the actual bill comes out to one-ninth.

The 7.6x figure is kernel speed, and the announcement’s 8.6x figure comes with a condition attached. It means prefill throughput is 8.6x that of Qwen3.7-Plus when the prefix cache⁸ hit rate is 90%. A 90% hit rate describes agentic workloads that read the same system prompt and tool definitions dozens of times over. This isn’t so much a performance boast as an admission of exactly what the model was built to do.

The table most likely to get quoted is the comparison with Claude Opus 4.6: SWE-bench Pro, 62.5 vs. 53.4; CoWorkBench, 73.9 vs. 68.2; JobBench, 55.7 vs. 36.6. But three lines of footnotes come attached. First, CoWorkBench and RecreationBench are benchmarks Qwen built itself. Second, SWE-bench Pro was re-measured only for the Qwen line using a “revised version with problematic tasks fixed,” while the Opus number is the officially published figure — not the same exam. Third, Opus 4.6 is two generations old. Anthropic moved through 4.7 and released 4.8 on May 28, and GLM-5.3-Flash, launched the same night, benchmarked itself against 4.8 instead. On Humanity’s Last Exam (HLE), Opus actually leads, 40.0 to 35.9.

The numbers are all real, and for a model with 6 billion active parameters, these scores are genuinely impressive. But the sentence “it beat Opus” doesn’t hold once you strip away those three footnotes. Even with the footnotes attached, one number still stands: price. The production API version, Qwen3.8-Flash, is slated at $0.16 per million input tokens and $0.47 per million output tokens. Opus 4.8 runs $5 and $25. That’s a 31x gap on input and a 53x gap on output.

Same Night, Same Price, Different Lock

That price wasn’t alone. The same night, Z.ai released GLM-5.3-Flash at $0.15 for input and $0.50 for output. With 18 billion of its 320 billion parameters active, plus a sparse-linear hybrid attention mechanism and an mHC that widens the residual stream, its parts list overlaps with Qwen’s. Z.ai added that “it all runs on Chinese AI chips.” The prices of these made-in-China Flash models converged right around 15 cents and 50 cents. In Artificial Analysis’s independent benchmarking, GLM-5.3-Flash scored an intelligence index of 57 — first among 173 models in the same price bracket — at $0.09 per task. The catch: during evaluation it generated 150 million output tokens, more than double the median. Even at identical per-token pricing, the chattier model runs up the bigger bill.

With prices converging, what’s left is the fine print. Starting with this model, Qwen switched its license from Apache 2.0 to the Qwen Community License 1.0. As recently as a month ago, Qwen3.8-27B still shipped under Apache. The new terms impose two requirements. Any product with over 100 million monthly active users or over $20 million in monthly revenue must display the model’s name on screen, and businesses that resell the model via API (Model as a Service) or offer coding and office AI-assistant products must sign a separate agreement with Qwen before commercial use. Internal use is exempted. Under this wording, yesterday’s issue’s coding tools like Cursor, and hosting companies like Fireworks that run Harvey’s model, would fall within scope. This is my reading, not legal advice.

The blueprint stays open — Qwen has only locked the two businesses that collide with its own API business (QwenCloud) and its own assistant products (Qoder, QwenWork). Evidence for July’s issue, “China Starts Locking Its Doors,” happened to surface this very night, but in a different shape. This isn’t a border export control; it’s a revolving door at the company’s front entrance. And next door, Z.ai opened its own door wider that same night, releasing under MIT. Locking and opening split apart within the same country.

Bypassing the HBM Bottleneck

Last month’s piece “The China That Sanctions Built” and mid-August’s “Intelligence Gone Cheap” both argued that AI’s bottleneck is HBM, and both ended on the same note: Korea sits astride that chokepoint. Today’s announcement adds a footnote to that argument—a bottleneck invites a bypass.

Qwen’s design notes cite only “memory-constrained accelerators” as the rationale—no mention of HBM or export controls. The connection is my own reading, but it’s not baseless. The recommended serving framework includes KTransformers, which offloads experts to CPU memory, and that same night Z.ai announced it runs entirely on Chinese chips. The scarcer HBM gets, the more attractive this design becomes.

51GB is a sliver of a single server’s RAM. What genuinely devours HBM is output tokens that caching can’t spare, and that flow keeps growing regardless of this design. What’s changing isn’t the amount of demand but its direction. Until now, a bigger model meant needing more HBM; in this design, a portion of that scale grows without ever touching HBM. For Korea’s memory industry, that sends a two-pronged signal: the bottleneck’s leverage isn’t permanent, and wherever the bypass leads, it still ends up at DRAM. Products made by the same companies, just carrying different margins.

Oswarld’s Lens

What I kept staring at in this announcement wasn’t the architecture diagram — it was the release sequence. According to TechNode, the stated reason for the disclosure was “to give the developer community time to prepare for the Qwen4 family.” Within a day of the announcement, Unsloth and llama.cpp had already shipped day-zero support, and vLLM and SGLang recipes went up the same day. By the time the real Qwen4 ships, the inference stack already knows how to run its architecture. Z.ai did the same thing in reverse. GLM-5.3-Flash ran for six days starting August 20th on OpenRouter under an anonymous free model called ‘Ox Alpha’, quietly collecting real-world usage. One side opens the blueprint first; the other leaks the model first.

I learned something running Notion’s Korea community: a platform doesn’t spread because its features are good — it spreads when other people are already moving on top of it before launch. If people are building templates before release, launch day stops being a starting line and becomes harvest day. A closed lab can’t run this playbook, because the architecture itself is the trade secret.

The contrast with Korea is stark. LG’s EXAONE 4.0 report explained its choice of sliding window attention by saying “open-source frameworks widely support it,” and the K-EXAONE model card from January this year included a note saying “install our custom version until the library adds official support.” One side picks the architecture the framework already supports; the other tries to make the framework support its own architecture. When evaluating a domestic foundation model, maybe the question to put right next to the benchmark scores should be: “does vLLM run on launch day?”

Closing

To sum up, three things. First, the point of Qwen3.8-Flash-Next isn’t performance — it’s design, and the core of that design is the decision to push 51 billion parameters out of GPU memory and into CPU memory. Second, the 1/9th is FLOPs, the 7.6x is kernels, and the comparison table against Opus carries three lines of footnotes. Strip away the footnotes and what’s left is price — and that price met GLM-5.3-Flash in the same 15-cent territory on the same night, with the license being what set them apart. Third, this design isn’t a rebuttal to the HBM bottleneck thesis — it’s a footnote to it. The bottleneck creates detours, and every detour still ends up at DRAM.

One thing to try this week: open the logs from the agent workloads running in your organization and check the prefix cache hit rate. If it’s close to 90%, this is exactly the kind of workload this model was built for. If it’s below 50%, the 8.6x in the announcement is a number that has nothing to do with you, Reader.

If you’re running long-context agents in production, tell me in the comments what your actual cache hit rate looks like, and whether prefill or decode is the heavier cost. I’ll gather the responses and follow up in the next issue with a piece on “the denominator of agent costs.”

---

💬 Tell me in the comments about the cache hit rate and cost structure of your long-context agents. I’ll factor it into the next issue. 📨 If you have a colleague wrestling with inference infrastructure, please share this piece with them.

---

📎 References & Further Reading

Primary sources

• Qwen Team, “On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability”, Alibaba Group, 2026.8.26. ··· This is today’s primary source. “51B kept off the accelerator,” “1/9th the FLOPs,” “8 of 14,” and the 4x learning-rate stress test are all here. Reading just the abstract and Chapter 1 gives you the design philosophy. Section 2.3 covers the N-gram embeddings.
• Qwen, “Qwen3.8-Flash-Next Model Card”, Hugging Face, 2026.8. ··· Parameter configuration, layer arrangement (GDN 3 : QSA 1), benchmark tables, and footnotes. Footnotes 2 and 5 below the table are the basis for what the body of this piece calls the “revised exam” and the “in-house benchmark.”
• vLLM, “Qwen3.8-Flash-Next Recipe”, 2026.8. ··· This is where “51GB or more of host memory” and the checkpoint size are stated. Also worth checking: the kernel-acceleration figures (10.2x, 6.6x) are listed differently here than in the technical report.
• Qwen, “Official Qwen3.8-Flash-Next Announcement”, X, 2026.8.26. ··· Source for the preview pricing of the full Qwen3.8-Flash release ($0.16 input, $0.47 output) and the “8.6x at 90% cache hit rate” claim.
• TechNode, “Alibaba’s Qwen to open-source Qwen3.8-Flash-Next, previewing Qwen4 architecture”, 2026.8.26. ··· This is where the stated reason for open-sourcing — “giving the developer community a chance to prepare for Qwen4” — comes from.
• Anthropic, “Introducing Claude Opus 4.8”, 2026.5.28. ··· Source confirming the current generation and pricing ($5 input, $25 output) of the comparison model.
• Qwen, “Qwen Community License 1.0”, 2026.8. ··· Only two paragraphs of actual terms. Clause 1 covers the model-name attribution requirement (100 million monthly active users or $20 million in monthly revenue); Clause 2 covers the separate contract requirement for Model as a Service and AI Work Assistant use. Worth reading firsthand. Up through Qwen3.8-27B, licensing was Apache 2.0.
• Z.ai, “Introducing GLM-5.3-Flash”, X, 2026.8.26. ··· Source for the 320 billion / 18 billion active parameter count, the MIT license, the “pre-released as Ox Alpha” claim, “runs entirely on Chinese AI chips,” and pricing ($0.15 input, $0.50 output).
• Z.ai, “GLM-5.3-Flash Model Card”, Hugging Face, 2026.8. ··· This is where I confirmed the adoption of sparse-linear hybrid attention and mHC, and that the comparison model is Opus 4.8. The four recommended serving frameworks are the same as Qwen’s.
• Bloomberg, “China’s Z.ai Made Ox Alpha, Stealth Model That Rivals DeepSeek”, 2026.8.26. ··· The report where Z.ai first acknowledged Ox Alpha as its own model.
• Artificial Analysis, “GLM-5.3-Flash Intelligence, Performance & Price Analysis”, 2026.8. ··· Source for the intelligence index of 57 (#1 among 173 models in the same price tier), the $0.09 cost per task, and the 150 million evaluation output tokens (median 64 million). This is where you can see why price-per-token and cost-per-task diverge.
• Decrypt, “AI Model Ox Alpha Is Free, Beats Claude Fable, and Nobody Knows Who Built It”, 2026.8. ··· The story of how the anonymous August 20 release led the community to identify GLM as the source via its tokenizer. Also covers how the benchmark claims based on 10 tasks changed once all 113 tasks were run.

Background

• Da Yu et al. (Google), “Scaling Embedding Layers in Language Models”, NeurIPS, 2025. ··· The original paper on the design (SCONE) that keeps N-gram embeddings off the accelerator. The unit-cost table in Section 4.3 (roughly $2 per GB of system memory, roughly $0.1 for SSD) is the economics underlying this design.
• InfoQ, “Gemma 3n Introduces Novel Techniques for Enhanced Mobile AI Inference”, 2025.7. ··· An explanation of Gemma 3n, which put per-layer embeddings on the CPU to run an 8 billion parameter model within 4 billion worth of memory. A mobile version of the same idea.
• LG AI Research, “EXAONE 4.0 Technical Report”, arXiv, 2025. ··· Section 2.1 states that sliding window attention was chosen because of “broad support across open-source frameworks.” This is where the contrast drawn in the Lens section comes from.
• LG AI Research, “K-EXAONE-236B-A23B Model Card”, Hugging Face, 2026.1. ··· The requirements section includes a note to “install our version until the library officially supports it.”
• AlternativeTo, “Alibaba releases open-source Qwen3-Next model”, 2025.9. ··· A record of last September’s Qwen3-Next release. You can confirm this followed the same script as the current one.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. N-gram: A grouping of N consecutive tokens. Two tokens, like “machine learning,” is called a bigram; three, a trigram. Memorizing frequently paired combinations helps you read context faster. ↩

2. HBM (High Bandwidth Memory): Memory stacked right next to the GPU, capable of exchanging several terabytes per second. It’s far faster and far more expensive than ordinary server DRAM, and right now it’s hard to secure supply of even with money in hand. ↩

3. Prefetch: A technique for fetching data before it’s needed. Like a chef pulling out ingredients for the next dish in advance, it starts reading from memory before a computation finishes, eliminating wait time. ↩

4. MoE (Mixture of Experts): A structure in which a model contains several small expert networks and selects only a few per token. Total parameters are large, but per-token computation is small. That said, even the experts that aren’t selected still have to sit in GPU memory. ↩

5. Prefill and decode: Prefill is the stage where the entire input context is read in at once; decode is the stage where the answer is generated one token at a time. The longer the input document, the heavier the prefill; the longer the output answer, the heavier the decode. ↩

6. Residual stream: A channel that runs through every layer, carrying information along. Each layer pulls information from this channel, computes on it, and adds the result back in. It’s closer to a conveyor belt. ↩

7. FLOPs: Floating-point operation count. A unit for measuring the computation used in training, but distinct from actual GPU time or cost. The same FLOPs can produce very different bills depending on how continuously the GPU is kept running. ↩

8. Prefix cache: A mechanism that reuses previously computed results when an input with the same leading portion is received again. Hit rates run high in agent workloads that read the same system prompt and tool descriptions over and over. ↩

Found this useful? Share it

Copy linkXLinkedInThreadsKakaoTalk네이버 밴드