azampatti/Qwen3.8-Flash-Next-FP4-Local-AZ

Qwen3.8-Flash-Next FP4-Local AZ

NVIDIA's quantization-aware-distilled FP4 checkpoint, nvidia/Qwen3.8-Flash-Next-FP4-Local, running 5 routed experts per token instead of 10, with the healed shared expert from azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound plugged in. No retraining was done for this checkpoint: the shared expert is the one healed on the INT4 model, converted to NVIDIA's format.

One self-contained folder, ~102 GB, same layout and file names as NVIDIA's.

Engine support. This checkpoint needs an engine that reads the routing width and shared-expert width from config.json. vLLM does (see Run it). TensorFold v1.1.4 does not yet: its Flash Next CUDA path is compiled for exactly 10 experts and a 640-wide shared expert and refuses this config at load. What it would need is listed under For TensorFold.

What is different from NVIDIA's checkpoint

	NVIDIA FP4-Local	This checkpoint
Routed experts per token (num_experts_per_tok)	10	5
Shared expert (shared_expert_intermediate_size)	640, MXFP8	1280, MXFP8 (healed)
Shared-expert gate	bf16	bf16 (healed)
MTP layer's shared expert	640, bf16	1280, bf16: NVIDIA's 640 plus 640 zero rows (output unchanged)
Everything else		byte-identical (NVFP4 routed experts, MXFP8 attention, bf16 head, FP8/NVFP4 n-gram table, vision tower, MTP)

Only tensors/model-00015-of-00021.safetensors and config.json differ from NVIDIA's files. The other 20 shards are NVIDIA's, unchanged.

The shared expert. On the INT4 model, cutting routing from 10 to 5 experts was healed by training only the shared expert (widened 640 → 1280, plus its gate) with KL self-distillation against the same model at 10 experts. That BF16 master (V8, epoch 2, step 1675; the one shipped in the INT4 repository) is quantized here straight to NVIDIA's MXFP8 layout: e4m3 values with one power-of-two scale per 32 weights, e = ceil(log2(amax / 448)). That is the rule NVIDIA's own shared-expert tensors follow. Error against the BF16 master: 2.66 %.

The MTP layer keeps NVIDIA's weights. Its 640-wide bf16 shared expert is zero-padded to 1280 so that one config describes both the trunk and the MTP layer: the extra gate/up rows give silu(0) * 0 = 0, and the extra down columns are zero, so the MTP layer's output is bit-for-bit the same as NVIDIA's.

How it compares

Capability on our own test harness, on one DGX Spark (GB10), with a 262,144-token window. Each number is a trimmed average over 6 runs (best and worst dropped).

	NVIDIA FP4-Local (10 experts)	This checkpoint (5 experts)
Capability (trimmed average, 6 runs)	49.4	51.7

The two numbers come from different engines. The original ran on TensorFold v1.1.4, which is deterministic at temperature 0, so its six runs were identical. This checkpoint ran on vLLM (the myllmbox recipe v5.3.1, its settings except the window, 262,144 without YaRN), because TensorFold cannot load it yet. Read the table as: the cut to 5 experts costs no measurable capability.

Run it (vLLM)

Any vLLM build that serves NVIDIA's FP4-Local checkpoint serves this one: the format is the same ModelOpt MIXED_PRECISION checkpoint, and the widths come from config.json. With the myllmbox recipe, point model: at this repository (or a local copy under ./models/).

MTP at 10 experts (optional). The numbers above use the checkpoint as it is, with the drafter at 5 experts. The MTP layer was trained at 10. To draft at 10 while the model routes 5, give vLLM's speculative config a draft folder: a copy of this folder's config.json with text_config.num_experts_per_tok set to 10, an index listing only the mtp.*, lm_head.* and embed_tokens tensors, and links to the shards they live in (1, 14, 15). Then add "model": "<draft folder>" to --speculative-config. The shared-expert width stays 1280, because the MTP layer's is padded to match.

For TensorFold

TensorFold v1.1.4's Flash Next CUDA engine (zig/src/families/flashnext/) checks config.json against compile-time constants and refuses this checkpoint. Two things would have to change:

1. Routing width. cuda_prompt_wt.zig has slots = 11 (10 routed experts + the shared one), and cuda_config.zig requires num_experts_per_tok == slots - 1. This checkpoint needs 5 routed experts (slots = 6). The MTP layer uses the same constant, so it would route 5 as well, unless it gets its own.
2. Shared-expert width. cuda_config.zig requires shared_expert_intermediate_size == 640 + tensorfold_learned_neurons, with the Living Weights pocket capped at 128. This checkpoint's shared expert is 1280 wide, and so is the MTP layer's (zero-padded). Two ways to get there:
   • a shared-expert width of its own, used by the shared-expert buffers and kernels;
   • a pocket of up to 640 (tensorfold_learned_neurons: 640). Rows 0-639 of each shared expert started from the original weights, and rows 640-1279 are the widening the heal trained, so a 640 + 640 split matches how it was built.

Everything else passes TensorFold's ModelOpt checks unchanged: NVFP4 group-16 routed experts with CSF scales, MXFP8 group-32 shared expert and attention projections, and NVIDIA's other settings. Every tensor appears exactly once, which is what TensorFold's loader expects (it reads every shard, and the first copy of a name wins). These are the changes the source calls for. No TensorFold patch has been written or tested against this checkpoint.

Files

The same file names as NVIDIA's repository: tensors/model-000NN-of-00021.safetensors, model.safetensors.index.json, config.json, tokenizer, chat template, and processor configs. Checks run before upload:

• every tensor name and its shard match NVIDIA's index;
• the 74,901 tensors of shard 15 that are not the shared expert are byte-identical to NVIDIA's;
• the 144 MXFP8 matrices and their scales are exactly the quantization of the BF16 master;
• the 48 gates are bit-identical to the master;
• the MTP shared expert's original part is bit-identical to NVIDIA's, and its padding is all zeros.

License

Built from NVIDIA's Qwen3.8-Flash-Next-FP4-Local; its weights are governed by the NVIDIA Open Model Agreement. The base model, Qwen3.8-Flash-Next, is governed by the Qwen Community License 1.0. Modified by azampatti: top-5 routing and a replaced shared expert, as described above.

Credits

• NVIDIA for the QAD FP4-Local checkpoint this is built on.
• Qwen for Qwen3.8-Flash-Next.
• myllmbox for the vLLM recipe and image the comparison ran on.
• TensorFold and Mia's AI Lab, whose single-Spark TensorFold recipe this checkpoint is meant to drop into once the engine allows it.