926 tok/s on one DGX Spark — the NVFP4 recipes that got me there

▸Replicate the headline in three commands

Everything the 926 tok/s and the giant KV pool come from — no custom container, no patches:

# 1. the image: nightly vLLM main built for GB10 (by eugr)
docker pull eugr/spark-vllm:latest

# 2. serve unsloth's 35B MoE — fp8 KV + MTP; let vLLM pick kernels (auto beats forcing, measured below)
docker run -d --name unsloth35b --gpus all --ipc=host --net=host \
  -e CUTE_DSL_ARCH=sm_121a \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/vllm-cache:/root/.cache/vllm \
  --entrypoint vllm eugr/spark-vllm:latest \
  serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
  --host 0.0.0.0 --port 8888 --trust-remote-code \
  --kv-cache-dtype fp8 --moe-backend auto \
  --gpu-memory-utilization 0.55 --max-model-len 262144 \
  --max-num-seqs 64 --max-num-batched-tokens 32768 \
  --enable-chunked-prefill --async-scheduling --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

# 3. after EVERY boot: send one real request before trusting anything (health lies — see below)
curl -s http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"unsloth/Qwen3.6-35B-A3B-NVFP4-Fast","messages":[{"role":"user","content":"hi"}],"max_tokens":8}'

Notes that save you a day: the ~/vllm-cache mount persists compile/autotune caches across container rebuilds (cold boot 13–16 min → warm ~2 min on some images — h/t walterra). --max-num-seqs is your parallel-request ceiling; 64 was my sweep max. Utilization 0.55 already holds ~10 full-262k contexts; 0.65 held ~21× in an earlier run. Never go above 0.80 on unified memory — the OS lives in the same RAM, and things die ugly (documented below).

Two checkpoints: Qwen3.6-35B-A3B-NVFP4-Fast (MoE, 3B active) and Qwen3.6-27B-NVFP4 (dense). Unsloth's published numbers were not measured on a DGX Spark — they're real figures from other hardware. The point of this table is what changes when the hardware is a GB10.

·Their (non-Spark) claims vs this box

unsloth published (other hardware) vs measured on this GB10 · different benches — direction, not decimals
Figure	unsloth published	Measured on this GB10	Why the gap
35B-A3B decode	295.2 tok/s (cute-DSL)	89–95 tok/s c=1 · 336 agg c=8 · 926 agg c=64	theirs is batch-aggregate on faster-memory hardware; mine crosses it at high concurrency
27B decode	125.9 tok/s (cute-DSL)	33.5–35.5 tok/s	GB10 has 273 GB/s memory bandwidth — a dense 27B physically can't stream weights faster than ~26 tok/s without speculation here
"1.79× faster than FP8"	—	b12x kernels measured: prefill flat, decode −11%	doesn't transfer to GB10 (explained below)
MMLU-Pro 35B / 27B	85.85 / 86.25	(their data, not re-measured)	decides the model choice in section 05

Aggregate @ 64 streams

926 tok/s

35B-A3B, MTP k=3, U=0.55 — no throughput wall found up to 64

KV pool @262k

10.65×

at U=0.55 — and 21.46× recorded at U=0.65. Never exceed 0.80 util

27B w/ DFlash graft

33.5–35.5

tok/s — +35% over stock, fastest 27B-class figure I've seen published for GB10

b12x, engaged properly

−11%

decode, and KV pool cut 10.65× → 3.62×

What I achieved / fixed — the short version

926 tok/s aggregate at 64 concurrent streams — and still climbing at the ceiling I tested
10.65× full-262k KV at U=0.55, up to 21.46× at U=0.65 — fp8 KV works because unsloth ships calibration scales. Hard rule: never above 0.80 util on unified memory
b12x + MTP together — documented nowhere: the speculative drafter needs its own backend ("moe_backend":"triton" inside --speculative-config)
Measured b12x as a net loss on GB10 — with the mechanism, not a shrug
27B +35% via AEON's DFlash drafter grafted onto unsloth weights (hidden size & vocab match)
Prefill measured as a curve, not a number — the single-figure prefill quote is a myth

01The b12x recipe — and what it's for

For DGX Spark, unsloth recommend --moe-backend flashinfer_b12x. Their concern is valid: on their quant scheme, falling back to Marlin kernels costs ~2.5×. But add MTP speculation and the flag kills the boot:

ValueError: moe_backend='flashinfer_b12x' is not supported for unquantized MoE.
Expected one of ['triton', 'flashinfer_trtllm', 'flashinfer_cutlass', 'aiter'].

The MTP draft head ships unquantized and the flag applies to every layer. The fix — implied by r0b0tlab's config, spelled out nowhere:

export CUTE_DSL_ARCH=sm_121a

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
  --moe-backend flashinfer_b12x \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}'

The log confirms Using 'FLASHINFER_B12X' NvFp4 MoE backend. It boots, it answers, it's real.

02Auto vs b12x — vLLM's own picks win on this box

To be precise about what's being compared: both columns are unsloth's checkpoint. The left column lets vLLM auto-select kernels (it picks Cutlass — on my image the Marlin fallback unsloth warn about never happens). The right column is unsloth's Spark recommendation (b12x), made bootable with my drafter fix. Same box, same benchmark, same hour:

35B-A3B · U=0.55 · N=8 · fp8 KV — my recipe (auto) vs the b12x path
Metric	auto → Cutlass + MTP k=3 (mine)	b12x + Triton draft + MTP k=2
Prefill pp8192	6,694 / 6,488	6,645 / 6,775
Prefill pp32768	4,824 / 4,850	4,968 / 4,993
Decode c=1	86.0 / 89.6	75.2 / 81.8
Decode c=8	322 / 336	184* / 337
KV pool @262k	10.65×	3.62×

Prefill flat, decode −11%, KV pool cut to a third (b12x's workspace buffers eat it). This isn't unsloth being wrong — it's hardware context. The boot log shows why: Using Triton/FLA GDN prefill kernel. This model is a GDN hybrid, so prompt processing never touches the MoE math b12x accelerates, and generation speed on GB10 is pinned by memory bandwidth, not compute. b12x's 1.79× is real where MoE compute is the bottleneck. On a GB10 with this architecture, it isn't — so vLLM's auto selection is the optimal Spark config, and now that's a measurement.

03What GPU utilization actually buys

gpu-memory-utilization is how much of the 121 GiB the engine reserves. People treat it as a speed dial. It isn't one: generation speed doesn't change with it at all. What it buys is KV cache — the working memory that holds conversations — which means more parallel sessions and longer contexts:

KV pool vs utilization — decode identical in every row · "10.65×" = ten full 262k-token conversations at once
Model	util	KV @262k	Decode c=1
35B-A3B (fp8 KV)	0.55	10.65×	86–90
35B-A3B (fp8 KV)	0.65	21.46×	same
27B dense (fp8 KV)	0.55	1.90×	30–33
27B dense (fp8 KV)	0.74	4.24×	same

Two hard ceilings before you crank it: the operating system lives in the same 121 GiB (push past ~0.85 and the OS can't even spawn processes — SSH hangs, ports never open), and speculative-decoding scratch memory allocates outside this setting entirely (the AEON tab has the full story — it cost me a power-cycle). House rule: 0.80 max, and leave more if anything else runs on the box.

04Prefill is a curve, not a number

35B-A3B (MoE, fp8 KV) 27B dense (fp8 KV)

Single request, prompt caching defeated. My own docs had "~7,000 tok/s" written down as the prefill number; it was an 8k-only figure.

35B-A3B (MoE) 27B dense

TTFT is the number you feel — and it spans 300× on the same box. A short question answers in 0.10 s on the 35B; the same model needs 6.8 s at 32k, and the dense 27B needs 30 s — paid on every message, before a single word appears. No setting moves it; only prefill throughput and prompt caching do. (In real agent use with vLLM's prefix cache at ~86% hits, measured mean TTFT was 2.2 s — the cache is why agent workloads survive.)

05Dense vs MoE — read this before picking "the smart one"

Speed measured here · accuracy from unsloth's published figures
	35B-A3B	27B dense	ratio
Prefill 8k / 32k	6,600 / 4,850	1,600 / 1,085	4.1× / 4.5×
Decode c=1 / c=8	87.8 / 329	30.5 / 137	2.9× / 2.4×
KV @262k	10.65×	1.89×	5.6×
MMLU-Pro	85.85	86.25	−0.4

On paper: 3–4× the speed for 0.4 benchmark points. In practice, one honest nuance from daily use: for coding specifically, the dense 27B still writes noticeably better code than that 0.4-point gap suggests — benchmarks and real work don't fully agree. My split: the dense 27B stays as the dedicated coding model on quality grounds; the MoE 35B does everything else, where its speed and 5.6× KV pool win outright. Pick per workload, not per leaderboard.

Three uncensored checkpoints by AEON-7 — Ornith 35B, heretic 35B, and the 27B MTP — on his own sm_121a vLLM image with DFlash speculative decoding. Unlike unsloth's, AEON's numbers ARE measured on a DGX Spark, so they compare directly. It's the best uncensored setup I've found for this box.

·His published Spark figures vs this box

AEON published (from his model cards / repos, measured on GB10) vs measured here · his bench is category-based & non-streaming; mine streams and reads ~15–25% lower — direction, not decimals
Figure	AEON published	Measured here
heretic 35B, single stream	75.2–123.6 tok/s by category (avg ~97)	76.8 mean (streaming) · 82–90 house bench — inside his range
heretic 35B, aggregate @ c=64	739.8 tok/s (DFlash k=11, U=0.85)	not replicated — I only ran heretic to c=4; his figure stands. (My own c=64 measurement, 926 tok/s, is the unsloth 35B with MTP — see the other tab; different checkpoint & speculation, so not a head-to-head)
Ornith 35B, single stream	70.3–119 by category (avg 93.3)	92.7 coding / 91.0 chat (house bench) — matches
27B (DFlash, Spark)	median 38.5 · peak 71.3 tok/s (at k=15)	by category (at k=10): math 44.6 · json 36.7 · coding 32.6 · prose 27.3 — same territory once category is matched

Decoding his "median 38.5 / peak 71.3": those are category statistics, single-stream — the median across six prompt types and the best of them, 200-token outputs, measured at DFlash k=15. Not concurrency. Speculative decoding's speed depends on how predictable the output is: math and JSON accept many drafted tokens, prose accepts few. My first "28–33" figure was one generic prompt with a streaming penalty; measured per-category at his output length, the numbers line up. I also tried his k=15 draft length on my setup: net-flat with worse variance and a smaller KV pool — the optimal k belongs to the checkpoint+drafter pair, not the method. Stacks are compared only against themselves on this page — a cross-stack row would compare checkpoints, speculation methods, and workloads all at once, and mean nothing.

27B MTP on new image

+22%

decode 25.0 → 30.4 tok/s, image 0.24 → 0.25.0

heretic on new image

+8.5%

decode 70.8 → 76.8, KV 2.69× → 2.88×

ornith on new image

±0%

dead heat — the checkpoint I tested first

Marlin alternative

none

forcing Cutlass: rejected — Marlin is the only backend for this quant scheme

What I achieved / fixed — the short version

Migrated all three checkpoints to image 0.25.0 after a full A/B: +22% (27B MTP), +8.5% (heretic), ±0 (ornith)
Proved Marlin isn't a mistake here: forcing Cutlass rejects this quant scheme — no speed was left on the table
Diagnosed the fp8-KV zombie boot: these checkpoints ship without fp8 calibration data, so the server says healthy and dies on the first message. BF16 KV is mandatory
Sized the 27B MTP safely: it needs 27 GiB of cache per full-length conversation — recipe below, learned via one crashed machine
Fixed the nightly restart: DFlash slows down over ~20 h and crashes ~23 h, and docker restart can never fix it on this hardware

01The image A/B I almost called wrong

AEON shipped image 0.25.0 this week. My first pass tested one checkpoint, generation speed only — saw a dead heat, and nearly concluded "no gain." Then I tested all three, on prompt processing and generation, three runs each. Per model:

Ornith 35B · 0.24 → 0.25.0
	0.24	0.25.0
Prefill 8k	4,578	4,761
Prefill 32k	4,207	4,286
TTFT @32k	7.8 s	7.6 s
Decode c=1	66.3	63.4
KV @262k	2.68×	2.69×
Verdict	dead heat (runs varied ±15%)

Heretic 35B · 0.24 → 0.25.0
	0.24	0.25.0
Prefill 8k	5,116	5,186
Prefill 32k	4,418	4,585
TTFT @32k	7.4 s	7.1 s
Decode c=1	70.8	76.8
KV @262k	2.69×	2.88×
Verdict	+8.5% decode, +7% KV

27B MTP · 0.24 → 0.25.0
	0.24	0.25.0
Prefill 8k	1,757	1,885
Prefill 32k	1,322	1,456
TTFT @32k	24.8 s	22.5 s
Decode c=1	25.0	30.4
DFlash accept	27.8%	31.3%
Verdict	+22% decode

The checkpoint I happened to test first (ornith) was the only one of three that showed nothing. Stopping there would have discarded a 22% win and deleted the wrong 45 GB image. Where a win is claimed, the three runs on each side don't even overlap; ornith's own runs disagreed with each other by 15%, which is what "dead heat" means here.

02Marlin is the ceiling — and that's now proven, not assumed

Every AEON checkpoint auto-selects Marlin kernels. Unsloth flag Marlin as a 2.5× slowdown for their models, so I forced the faster Cutlass path on ornith and heretic:

ValueError: NvFp4 MoE backend 'FLASHINFER_CUTLASS' does not support the deployment
configuration since kernel does not support quantization scheme
QuantKey(u8, scale(f8e4m3fn, static, GroupShape(row=1, col=16)), ...)

Rejected on both. AEON's checkpoints use a different weight format than unsloth's, and in this build only Marlin implements it. Translation: the "slow path" warning doesn't apply across formats, nobody is leaving speed on the table, and this stack's numbers are what this stack does.

03Sizing the 27B MTP without killing the box

Plain-language version first. Every conversation the model holds open needs "working memory" (the KV cache) — and how much depends on the model. This 27B needs 27 GiB for one full-length (262k-token) conversation — about five times what ornith needs. The gpu-memory-utilization number decides how much of the 121 GiB the engine may reserve; the cache has to fit inside that.

what happens at each setting · N = parallel conversations allowed
Setting	What happened	Use it?
util 0.55	engine refuses to start — needs 27 GiB for one conversation, only 21.5 available	no
util 0.70 · N=2	boots clean · 1.47 conversations of cache · 30.4 tok/s	yes — the safe recipe
util 0.80 · N=4	machine hard-locked — no SSH, no ping, power button	never

Why the last row killed the machine — the memory nobody counts

DFlash speculation runs a small helper model that guesses several words ahead, and each parallel conversation gets its own scratch space for those guesses. Here's the trap: that scratch space is not included in what gpu-memory-utilization accounts for. So at 0.80, the engine reserved ~97 of 121 GiB, every pre-boot check passed — and then four conversations' worth of guess-buffers allocated on top, in memory that also belongs to the operating system. On a normal PC that crashes a program; on unified memory it takes down the whole computer. Rule of thumb: whatever's left outside the engine must fit the OS plus (parallel conversations × drafter scratch). Lower the utilization or the conversation count — they multiply.

04The 20-hour clock

One more thing this stack demands: DFlash's guess-acceptance degrades over ~20 hours of uptime, and the engine crashes around hour 23. A nightly restart isn't hygiene here — it's survival. And the obvious way to do it, docker restart, can never work on this hardware: stopping the engine leaves ~90 GiB of "ghost" memory the driver hasn't returned yet, and the instantly-restarting container walks straight into it and dies. Mine failed silently every night until I caught the box dead one morning.

The sequence that works — with one term defined: "voice" below means the speech containers (Qwen3 speech-to-text and text-to-speech) I run beside the LLM for voice chat; they hold memory the LLM needs back while it reboots:

detect which LLM container is running
→ pause the voice (speech) containers        # they hold unified memory
→ docker stop the LLM
→ sync; echo 3 > /proc/sys/vm/drop_caches    # releases the ghost memory (~4 s)
→ docker start the LLM                       # same container = exact same config
→ wait for /health
→ send ONE REAL CHAT REQUEST                 # /health lies; the engine can be dead behind it
→ restart the voice containers

05Why keep this stack at all

Because if you need uncensored models, this is the best-engineered option I've found for a Spark — and it's not slow: heretic hits 123.6 tok/s on math single-stream (his measurement; category matters a lot with speculation), ornith matches its published ~93 average on my box, and the whole family holds 262k contexts. The unsloth stack wins on capacity and raw aggregate; the AEON stack is what exists when censored won't do — and after this week's image bump, its 27B is 22% faster than it was on Monday.

AWhose shoulders

Nearly everything above started as someone else's work. Two of the biggest wins were just reading what they'd published, carefully.

r0b0tlab

Published a working sm_121 config listing "B12X routed experts · MTP K=2 · Triton draft MoE." That phrase is the entire b12x+MTP fix. He had it before I did.

unsloth

The NVFP4 checkpoints, CUTE_DSL_ARCH=sm_121a, the b12x availability probe, and the accuracy figures. Their Spark advice didn't pan out on my workload — their checkpoints absolutely did.

AEON-7

The sm_121a images, the DFlash drafters (one of which I grafted onto unsloth's 27B for +35%), the uncensored checkpoints, and properly Spark-measured model cards — the only vendor numbers here that compare directly.

walterra.dev

The persistent /root/.cache/vllm mount (cold 13–16 min → warm ~2 min) and the fork-starvation warning at high utilization — which I then reproduced the hard way.

eugr

The nightly vLLM-main image the unsloth stack runs on, and the honest README line — "NVFP4 performance on Spark not fully optimized in vLLM (any build)" — my data independently confirms it.

BHow to read my numbers (and everyone else's)

Three things you need to know before comparing any figure on this page to a figure from anywhere else.

1. Benchmarks measure the measuring tool, too

My benchmark asks the server to stream its answer word-by-word, the way a chat app does. Most benchmarks ask for the whole answer in one block. Streaming has overhead — and I measured exactly how much: the same model, on the same machine, on the same day, scores 68.6–71.9 tok/s with my streaming tool and 92.7 with my older non-streaming one. Nothing about the model changed; only the measuring method did. That's a 25% gap from methodology alone. So: numbers from the same tool can be compared; numbers from different tools can't — including mine against unsloth's or AEON's. Where this page puts them side by side anyway, it says "direction, not decimals" and means it.

2. I published my mistakes on purpose

In four days I reached two confident, wrong conclusions and nearly a third:

Wrong #1: "The new AEON image is 20% slower." It wasn't — I had switched benchmarking tools between the measurements (see above). Running both images through the same tool showed them identical. Wrong #2: "The b12x kernels can't work on this GPU." vLLM's own check said they were available the whole time; the actual blocker was one unquantized layer needing its own setting — the fix in section 01. Near-miss #3: I almost judged the new AEON image on a single model — the one model (of three) that happened to show zero difference. Testing the rest revealed +22%.

3. The habits that caught all three

Change one thing at a time. Re-measure the old setup with the new tool before declaring anything slower. After every server start, send one real chat message — the health check is a different program from the engine and will happily say "ready" over a corpse. And when two runs disagree with each other by more than the difference you're trying to detect, run a third instead of publishing. None of this is clever. All of it is the reason this page isn't three falsehoods with confident formatting.