Patched vLLM for Qwen3.8-Flash-Next NVFP4 on one B200 (linux/amd64), experimental
152
Patched vLLM image for serving Qwen3.8-Flash-Next NVFP4 (starkweatherdigital/qwen3.8-flash-next-nvfp4, 109 GB) on one NVIDIA B200 (linux/amd64, CUDA 13.0.1).
Status: experimental. The image builds and imports on a CPU host, but it has not been run on a GPU yet. Whether the NVFP4 kernels work on a B200 is unverified.
linux/amd64 manifest of vllm/vllm-openai:qwen38-flash-next (vLLM 0.1.dev20073+g8e685d198, torch 2.13.0+cu130), pinned by digest.17c58984: 20 (4-bit PLE loader), 30 (graph output buffer), 35 (PLE
NVFP4 mmap), 40 (mamba eagle drop), 41 (mamba state seed). Patch 10 (sm121/DGX Spark workaround) is skipped on purpose.serve-b200, which builds the vllm serve command from environment variables.The model is not in the image. vLLM downloads it from Hugging Face onto a persistent volume on the first start.
| Variable | Default | Meaning |
|---|---|---|
MODEL | starkweatherdigital/qwen3.8-flash-next-nvfp4 | HF id, or a local directory |
SERVED_MODEL_NAME | qwen3.8-flash-next | name in the OpenAI API |
CTX | 131072 | max context length |
GPU_MEM | 0.90 | GPU memory utilization |
SEQS | 16 | max concurrent sequences |
MTP | 1 | speculative tokens (0 = off) |
CACHE | 1 | prefix caching |
MMAP / PREWARM | 0 / 0 | PLE mmap (needs MODEL to be a local directory) |
VLLM_API_KEY | none | set it, otherwise the API is open |
The main KV cache stays BF16 on purpose: the recipe reports that the attention layer rejects an FP8 main KV cache.
docker run --gpus all -p 8000:8000 \
-e VLLM_API_KEY=... -e HF_HOME=/data/hf -v /data:/data \
pt9912/vllm-qwen38-b200:1
Pin by digest for reproducibility:
pt9912/vllm-qwen38-b200@sha256:6f13de2fb100914c5ea5d381bc8cdb98a9b9ceeb9df7d9435197e7fb92d974dd
For RunPod (create, verify, start and stop a Pod through the REST API v2) see pt9912/runpod-qwen38-flash-next.
About 19.9 GB unpacked, 8.7 GB compressed.
The model and the recipe are subject to the Qwen Community License 1.0 (inherited from the base model). Check it before redistributing.
Content type
Image
Digest
sha256:f4ffbb293…
Size
8.1 GB
Last updated
2 days ago
docker pull pt9912/vllm-qwen38-b200:4