GLM-5.2 sparse attention on Blackwell
A 504-billion-parameter checkpoint loaded on eight RTX PRO 6000 GPUs. Output then repeated because the sparse indexer ran on the wrong layers.
Context
Problem
The model loaded on eight SM120 GPUs, but every generation became repetitive after a few tokens.
Constraints
- Each hypothesis required a full reload of a roughly 288 GB checkpoint.
- Published sparse-attention results mostly targeted different GPU architectures.
- A separate empty NCCL_GRAPH_FILE variable caused an unrelated startup failure.
System
Architecture
- 01
Checkpoint configuration
- 02
vLLM model configuration
- 03
Dense or sparse MLA
- 04
Layer indexer
- 05
CUDA runtime
- 06
Eight-GPU execution
My role
Ran the serving experiments, changed one variable at a time, inspected the checkpoint and vLLM source, and validated the corrected sparse path.
Decisions
- Used a dense attention fallback to separate checkpoint and MoE correctness from sparse-indexer behavior.
- Derived index_topk_pattern from the checkpoint's indexer_types instead of copying a pattern from another prune.
- Bounded per-step work with 8,192-token chunked prefill and decode-context parallelism.
Validation
What failed
- Changing NCCL, DCP, KV-cache dtype, CUDA graphs, MTP, and tensor parallelism did not change the repetitive output.
- Increasing the prefill workspace failed because the runtime locked the workspace after profiling.
Tests and checks
- Re-ran the original France and multiplication prompts on the corrected sparse path.
- Validated exact financial extraction from a long prompt.
- Retrieved a planted value from cold 128K-token and 240K-token prompts.
Outcome
Result
Deriving the missing layer pattern restored correct output. Chunked prefill and decode-context parallelism supported cold prompts with up to 240K-token context.
Limits
The reported 43 to 45 tokens per second applied to short-to-medium context, not the 240K-token run.