th3noloengineering notes← All notes
GLM-5.2vLLMBlackwellSM120GCPNCCLMoE

Serving GLM-5.2-504B on 8× RTX PRO 6000 GPUs

Manuel Parra2026-07-0414 min read

I set out to serve the 504-billion-parameter GLM-5.2 checkpoint on eight RTX PRO 6000 Blackwell GPUs in a GCP g4-standard-384 Spot VM. The model loaded, but every answer quickly collapsed into repetition. The sparse-attention kernel looked broken on SM120. It wasn't: the checkpoint left index_topk_pattern null, so vLLM ran the sparse indexer on layers that should have shared it. Deriving that pattern from indexer_types fixed the output.

This write-up follows the investigation: the tests that ruled out decoding, the dense fallback that isolated attention, the unrelated NCCL failure, and the final sparse configuration that served cold prompts up to 240K tokens.

The setup

  • Model: 0xSero/GLM-5.2-504B, architecture glm_moe_dsa — Multi-head Latent Attention (MLA) plus a DeepSeek-V3.2 sparse path where a small "lightning indexer" (index_n_heads=32, index_topk=2048) picks which keys each query attends to. 78 layers, first 3 dense, 168 routed experts in this REAP-pruned checkpoint, NVFP4 weights (~288 GB on disk).
  • Hardware: 8× RTX PRO 6000 Blackwell, SM120, tensor-parallel size 8.
  • Serving: vLLM, weights streamed from a GCS bucket, on a Spot VM that deletes itself when idle.

Most published GLM-5.2 / DeepSeek-V3.2 sparse-attention results run on B200 (SM100) or H100/H200 (SM90). The RTX PRO 6000 is Blackwell but a different SM target (SM120), and kernel coverage is not the same across SM100 and SM120 — which is where the trouble starts.

Failure: deterministic repetition

The first working boot used a prebuilt vLLM image built for SM120 (the "b12x" image, below). It served, but every generation collapsed into a loop. A prompt about France returned Paris. France France France France and kept going. Math prompts never reached an answer. The collapse was deterministic — it happened at temperature 0 and at every prompt length — which rules out sampling noise.

What I ruled out

The repetition looked like a decoding bug, so each decoding-side cause was changed on its own and tested. Each one was a full reload of a 288 GB checkpoint, so this was not cheap:

  • NCCL version — swapped, no change.
  • Decode context parallelism (DCP) — disabled, no change.
  • KV-cache dtype — fp8 vs auto, no change.
  • CUDA-graph capture — --enforce-eager to disable it, no change.
  • Multi-token prediction (MTP) — disabled, no change.
  • Tensor-parallel degree — 4 vs 8, no change.

None moved the output. That pointed away from decoding and toward attention. The wrong lesson to draw here — the one this write-up originally drew — is "the sparse kernel is broken on SM120." The right lesson is narrower: attention is misconfigured. The difference turned out to be one config field.

A separate NCCL failure

Switching images to test a dense fallback surfaced a second, unrelated failure: NCCL aborted during startup inside graph/search.cc, before the model could load across the 8 GPUs. The cause was an environment variable baked into the image: NCCL_GRAPH_FILE="". NCCL read the empty string as a path to a topology graph file, tried to open it, and failed. A variable set to empty is not the same as an unset variable.

# Baked into the image, breaks NCCL graph search:
#   NCCL_GRAPH_FILE=
# Fix — remove it entirely, do not set it to "":
unset NCCL_GRAPH_FILE

The failure looked like a Blackwell fabric problem, but removing the empty NCCL_GRAPH_FILE variable allowed NCCL to build its own topology and complete the 8-GPU initialization.

The dense fallback isolated attention

Before finding the real cause, one path already produced correct output: force the model onto vLLM's dense Triton MLA kernel, which runs on SM120. Three source edits to stock vllm/vllm-openai:v0.23.0 disable the sparse indexer:

# 1) vllm/platforms/cuda.py — SM100-only Blackwell check excludes SM120 (major 12)
#    before: if device_capability.major == 10:
if device_capability.major >= 10:

# 2) vllm/model_executor/models/deepseek_v2.py — stop treating glm_moe_dsa as v3.2-sparse
self.is_v32 = hasattr(config, "index_topk") and \
    getattr(config, "model_type", None) != "glm_moe_dsa"

# 3) same file, weight loop — the disabled indexer's weights have nowhere to go
for name, loaded_weight in weights:
    if ".indexer" in name:
        continue

Served with VLLM_ATTENTION_BACKEND=TRITON_MLA --enforce-eager, the prompts that had looped returned correct text: 17 × 23 = 391 and coherent prose. Dense attends to every key instead of the top 2048, so it is slower and gives up the long-context advantage of the sparse indexer, and --enforce-eager (needed here because CUDA-graph capture was flaky on this path) costs more at decode. Correct output on the dense path showed that the checkpoint, dequantization path, and MoE routing could work together. The remaining fault was specific to sparse attention.

Root cause: a missing index_topk_pattern

The sparse kernel was never broken on SM120. It was being told to build a full indexer on every layer.

GLM-5.2's config carries an indexer_types array — 78 entries, one per layer, marking each layer "full" or "shared". But vLLM's sparse path does not read indexer_types. It reads a separate field, index_topk_pattern — a 78-character F/S string (here 21 F, 57 S) that says which layers run the indexer and which skip it. In this checkpoint that field ships as null. With it null, vLLM builds a full indexer on all 78 layers, and the 57 layers meant to skip corrupt attention. Fluent locally, looping globally — exactly the symptom.

The fix is to derive the pattern from the checkpoint's own indexer_types and pass it in:

--hf-overrides '{"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'   # 78 chars, 21 F / 57 S

With the pattern supplied, the same b12x sparse kernel on the same 8 SM120 GPUs produces correct output — and vLLM logs confirm it now skips the indexer on the "S" layers instead of running all 78. The looping runs never passed this field; 0xSero's SM120 recipe does, which is why theirs worked and the first attempts did not.

Derive the string per checkpoint — do not copy the one above onto a different model. Read indexer_types from the model's config.json (78 entries here) and map each in order: full → F, shared → S. A different GLM-5.2 prune has a different array and a different string. This is arguably a packaging/ergonomics gap more than a kernel bug: the checkpoint already ships indexer_types, but vLLM reads only index_topk_pattern — which the config leaves null — and does not fall back to deriving it. Closest tracked report: vllm-project/vllm#46726.

unset NCCL_GRAPH_FILE
vllm serve /models/glm-5.2-504b \
  --tensor-parallel-size 8 --quantization modelopt_fp4 \
  --attention-backend B12X_MLA_SPARSE --moe-backend b12x \
  --kv-cache-dtype fp8 --load-format fastsafetensors \
  --hf-overrides '{"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'

Validation on sparse attention

Validated on the 8-GPU SM120 box, all on the sparse path:

  • Greedy France prompt → The capital of France is Paris. (the exact prompt that had looped).
  • 17 × 23 → 391.
  • A financial-analysis task — read twelve quarterly income statements out of a ~6–30K-token document, fill an empty JSON with full-year revenue, net income, gross-margin %, and year-over-year growth — returned every figure exactly, including a deep single-value lookup.
  • A needle placed in the middle of a filler document and retrieved correctly from a single cold prompt — velvet-thunder-7788 at both 128K and 240K tokens, no prefix cache. Cold long context works, but only with the DCP config in the ceiling section below; the minimal command above still tops out around 32K.
  • Decode about 43–45 tokens/second — batch 1, steady-state, at short-to-medium context (a few hundred up to ~30K tokens including prefill). Long-context decode is slower; do not read 45 tok/s as a 240K figure.

Reaching 240K context with DCP

The minimal sparse command above has a ceiling, and it is easy to hit. The b12x sparse indexer allocates a scratch workspace during startup profiling and locks it (about 708 MB). A single fresh prompt whose per-step indexer allocation needs more asserts and kills the engine:

AssertionError: Workspace is locked but allocation from _run_b12x_paged_topk
requires 1412 MB, current size 708 MB. Workspace growth is not allowed after locking.

The wrong instinct — mine, first — is to make the workspace bigger by raising the prefill batch. That fails twice: a large batch (131072) breaks torch.compile, and warmup length does not size the workspace either (warming at 120K still locked at 708 MB, and the warmup itself crashed). On that path a single cold prompt tops out around 32K, and it is tempting to write the limitation down and move on.

The fix runs the opposite direction: do not grow the workspace — bound the work that hits it, at any total context. Two flags do it:

  • --enable-chunked-prefill --max-num-batched-tokens 8192 — counterintuitively small. A 240K prompt prefills in ~30 chunks of 8192, so the indexer only ever sees an 8192-token query slice per step. That bounds the query dimension.
  • --decode-context-parallel-size 8 — shards the KV along the sequence dimension across the 8 GPUs, so each rank's indexer scores only its 1/8 of the keys. That bounds the key dimension.

Together the per-step allocation stays under the 708 MB lock no matter how long the prompt is. It also needs a few b12x environment variables — the load-bearing ones are VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, which stops the profiler mis-locking the reservation, and clearing VLLM_B12X_MLA_EXTEND_MAX_CHUNKS.

One reproducibility note that cost a boot to learn: the NCCL_P2P_DISABLE=1 and VLLM_ENABLE_PCIE_ALLREDUCE=0 flags this recipe ships are load-bearing on GCP's virtualized PCIe, not bare-metal leftovers. Dropping them to chase faster collectives enables a custom PCIe all-reduce that crashes with custom_all_reduce.cuh:455 'invalid argument' and crash-loops the engine. Leave them on.

With that config, cold single prompts of 48K, 128K, and 240K all served with no assert, and the needle came back correctly from a fresh 240K prompt. The cost is prefill throughput: those small 8192 chunks serialize a long prefill (~1000 tok/s; a cold 240K prompt takes about 256 seconds before the first token). Prefix caching hides most of it in practice — the long document is prefilled once and every follow-up question reuses the KV — but a cold, unique long prompt pays the full ingest. I validated cold prompts up to 240K tokens, with more memory headroom on eight GPUs than on the four-GPU reference configuration.

This is 0xSero's mechanism, not mine: borrowing only their index_topk_pattern fixed correctness but left this ceiling, and it took their full launch config to see the answer was smaller chunks plus DCP, not a bigger workspace.

Cutting the 288GB load time to four minutes

Weights stream from a GCS bucket over a gcsfuse mount. Stock vLLM reads the checkpoint one shard at a time — about 106 seconds per shard, roughly 20 minutes before the server accepts a request, all of it billed at 8-GPU Spot rates. fastsafetensors reads shards in parallel and brings the full load to about 4 minutes:

pip install fastsafetensors   # then: vllm serve ... --load-format fastsafetensors

Scale-to-zero deployment

An 8-GPU Blackwell VM is expensive to leave running, so it does not. The serving VM runs a watchdog that reads vLLM's own /metrics; when running and waiting request counts hit zero and the token counters stop moving for a set window, the VM deletes itself through the Compute API. The cold path includes VM provisioning and the roughly four-minute model load described above. The model exists while it is answering plus a short idle tail, not 24 hours a day.

What I would reuse

The specific bug was a null config field. The reusable parts are the method:

  • A deterministic failure that survives every decoding-side toggle is not a decoding bug — it is attention. But before concluding a kernel is broken, read the model's own config fields: the layer pattern that drove this was sitting in indexer_types, unused, while vLLM read a different field that shipped null.
  • A compute-capability check written for one Blackwell part (== 10) silently excludes a newer one (SM120, major 12). Widen it.
  • A variable set to an empty string is not unset. NCCL_GRAPH_FILE="" fails where no variable at all succeeds.
  • Correct beats fast: the dense workaround was the cheapest way to prove the weights were fine and isolate the fault to attention, even though the sparse path is the one worth shipping.

Credits

This stands on other people's work:

  • 0xSero — the REAP-pruned GLM-5.2-504B NVFP4 checkpoint, and the glm-5.2-sm120 recipe that carries the correct index_topk_pattern. Matching that pattern is what turned looping into correct output.
  • voipmonitor — the b12x vLLM image, built for sm_120a, which is what runs the sparse kernel on the RTX PRO 6000 at all. Full tag: voipmonitor/vllm:black-benediction-b12xpr11-vllmbb6c5b7-b12xd90d89c-fi3395b41aa8d-dg324aced12c-cu132-20260608.
  • Cerebras — REAP (Router-weighted Expert Activation Pruning), the method behind the 504B pruned checkpoint (arXiv:2510.13999).