Serving GLM-5.2-504B on RTX PRO 6000: the vLLM sparse attention fix on SM120

I served the 504-billion-parameter GLM-5.2 checkpoint on eight RTX PRO 6000 Blackwell GPUs in a GCP g4-standard-384 Spot VM. The model loaded, but its output became repetitive after a few tokens. The sparse-attention kernel looked broken on SM120. It was not. The checkpoint had a null index_topk_pattern, so vLLM ran the sparse indexer on layers that should have skipped it. Deriving the pattern from indexer_types fixed the output.

This note follows the investigation: the tests that ruled out decoding, the dense fallback that isolated attention, the unrelated NCCL failure, and the final sparse configuration that served cold prompts up to 240K tokens.

The setup

Most published GLM-5.2 and DeepSeek-V3.2 sparse-attention results use B200 (SM100) or H100/H200 (SM90) GPUs. The RTX PRO 6000 uses Blackwell with a different SM target, SM120. Kernel coverage differs between SM100 and SM120.

Failure: deterministic repetition

The first working boot used a prebuilt vLLM image built for SM120 (the "b12x" image, below). It served, but every generation collapsed into a loop. A prompt about France returned Paris. France France France France and kept going. Math prompts never reached an answer. The collapse was deterministic — it happened at temperature 0 and at every prompt length — which rules out sampling noise.

What I ruled out

The repetition looked like a decoding bug, so each decoding-side cause was changed on its own and tested. Each one was a full reload of a 288 GB checkpoint, so this was not cheap:

None of these changes affected the output. That pointed away from decoding and toward attention. The sparse kernel was not broken on SM120. Attention was misconfigured. The difference was one configuration field.

A separate NCCL failure

The dense fallback isolated attention

Before finding the real cause, one path already produced correct output: force the model onto vLLM's dense Triton MLA kernel, which runs on SM120. Three source edits to stock vllm/vllm-openai:v0.23.0 disable the sparse indexer:

# 1) vllm/platforms/cuda.py — SM100-only Blackwell check excludes SM120 (major 12)
#    before: if device_capability.major == 10:
if device_capability.major >= 10:

# 2) vllm/model_executor/models/deepseek_v2.py — stop treating glm_moe_dsa as v3.2-sparse
self.is_v32 = hasattr(config, "index_topk") and \
    getattr(config, "model_type", None) != "glm_moe_dsa"

# 3) same file, weight loop — the disabled indexer's weights have nowhere to go
for name, loaded_weight in weights:
    if ".indexer" in name:
        continue

Served with VLLM_ATTENTION_BACKEND=TRITON_MLA --enforce-eager, the prompts that had looped returned correct text: 17 × 23 = 391 and coherent prose. Dense attends to every key instead of the top 2048, so it is slower and gives up the long-context advantage of the sparse indexer, and --enforce-eager (needed here because CUDA-graph capture was flaky on this path) costs more at decode. Correct output on the dense path showed that the checkpoint, dequantization path, and MoE routing could work together. The remaining fault was specific to sparse attention.

Root cause: a missing index_topk_pattern

The sparse kernel was never broken on SM120. It was being told to build a full indexer on every layer.

Derive the string per checkpoint — do not copy the one above onto a different model. Read indexer_types from the model's config.json (78 entries here) and map each in order: full → F, shared → S. A different GLM-5.2 prune has a different array and a different string. This is arguably a packaging/ergonomics gap more than a kernel bug: the checkpoint already ships indexer_types, but vLLM reads only index_topk_pattern — which the config leaves null — and does not fall back to deriving it. Closest tracked report: vllm-project/vllm#46726.

unset NCCL_GRAPH_FILE
vllm serve /models/glm-5.2-504b \
  --tensor-parallel-size 8 --quantization modelopt_fp4 \
  --attention-backend B12X_MLA_SPARSE --moe-backend b12x \
  --kv-cache-dtype fp8 --load-format fastsafetensors \
  --hf-overrides '{"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'

Validation on sparse attention

Validated on the 8-GPU SM120 box, all on the sparse path:

Reaching 240K context with DCP

The minimal sparse command above has a ceiling, and it is easy to hit. The b12x sparse indexer allocates a scratch workspace during startup profiling and locks it (about 708 MB). A single fresh prompt whose per-step indexer allocation needs more asserts and kills the engine:

AssertionError: Workspace is locked but allocation from _run_b12x_paged_topk
requires 1412 MB, current size 708 MB. Workspace growth is not allowed after locking.

Together the per-step allocation stays under the 708 MB lock no matter how long the prompt is. It also needs several b12x environment variables. The required settings include VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, which prevents the profiler from locking the wrong reservation, and a cleared VLLM_B12X_MLA_EXTEND_MAX_CHUNKS value.

One reproducibility note that cost a boot to learn: the NCCL_P2P_DISABLE=1 and VLLM_ENABLE_PCIE_ALLREDUCE=0 flags this recipe ships are load-bearing on GCP's virtualized PCIe, not bare-metal leftovers. Dropping them to chase faster collectives enables a custom PCIe all-reduce that crashes with custom_all_reduce.cuh:455 'invalid argument' and crash-loops the engine. Leave them on.

With that config, cold single prompts of 48K, 128K, and 240K all served with no assert, and the needle came back correctly from a fresh 240K prompt. The cost is prefill throughput: those small 8192 chunks serialize a long prefill (~1000 tok/s; a cold 240K prompt takes about 256 seconds before the first token). Prefix caching hides most of it in practice — the long document is prefilled once and every follow-up question reuses the KV — but a cold, unique long prompt pays the full ingest.

This is 0xSero's mechanism, not mine: borrowing only their index_topk_pattern fixed correctness but left this ceiling, and it took their full launch config to see the answer was smaller chunks plus DCP, not a bigger workspace.

Cutting the 288GB load time to four minutes

Weights stream from a GCS bucket over a gcsfuse mount. Stock vLLM reads the checkpoint one shard at a time — about 106 seconds per shard, roughly 20 minutes before the server accepts a request, all of it billed at 8-GPU Spot rates.

pip install fastsafetensors   # then: vllm serve ... --load-format fastsafetensors

Scale-to-zero deployment

The serving VM is deleted when it becomes idle. A watchdog reads vLLM's /metrics. When running and waiting request counts reach zero and token counters stop moving for a set period, the watchdog deletes the VM through the Compute API. The cold path includes VM provisioning and the roughly four-minute model load. The model runs while it serves requests and for a short idle period after them.

What I would reuse

The specific bug was a null configuration field. The reusable method is:

• A deterministic failure that survives every decoding-side toggle is not a decoding bug — it is attention. But before concluding a kernel is broken, read the model's own config fields: the layer pattern that drove this was sitting in indexer_types, unused, while vLLM read a different field that shipped null.
• A compute-capability check written for one Blackwell part (== 10) silently excludes a newer one (SM120, major 12). Widen it.
• A variable set to an empty string is not unset. NCCL_GRAPH_FILE="" fails where no variable at all succeeds.
• Correct beats fast: the dense workaround was the cheapest way to prove the weights were fine and isolate the fault to attention, even though the sparse path is the one worth shipping.

Credits

This stands on other people's work:

Related work

Read the GLM-5.2 inference case study for the deployment constraints, validation scope, and result. For my related engineering background, see the AI and inference CV (PDF).