amd/Mistral-Large-4-1T-A52B-Quark-MXFP4

Model Overview

• Model Name: amd/Mistral-Large-4-1T-A52B-Quark-MXFP4
• Model Architecture: PixtralForConditionalGeneration
  • Input: Text, Image
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350/MI355
• ROCm: 10.0 (HIP runtime 7.15.26333)
• PyTorch: 2.13.0+rocm10.0.0
• Transformers: 5.20.2
• Operating System(s): Linux
• Inference Engine: vLLM
• Container base image: This model was tested using the docker container vllm/vllm-openai-rocm:nightly-rocm100-00b6c4c565a8732c131e1f452d4a17803bb0842b (vLLM nightly 26/10/2026) on a development branch for Mistral Large 4.
• Model Optimizer: upcoming AMD Quark 0.14
  • Routed experts weight quantization: OCP MXFP4, Static
  • Routed experts activation quantization: OCP MXFP4, Dynamic
  • All other layers: unchanged precision compared to base mistralai/Mistral-Large-4-1T-A52B

Model deployment

Please refer to vLLM recipe and SGLang cookbook.

This model was tested with vLLM on 4x MI355X. The Mistral tokenizer (tekken.json v15, with low / medium / high reasoning efforts) requires mistral-common 1.13.0 or later, which supports these reasoning effort levels:

pip install "mistral-common[image]>=1.13.0"

Example vLLM server:

OMP_NUM_THREADS=1 VLLM_ROCM_USE_AITER=1 \
vllm serve amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 \
    --tokenizer-mode mistral \
    --config-format mistral \
    --load-format mistral \
    --tensor-parallel-size 4 \
    --monokernel \
    --kv-cache-dtype fp8 \
    --max-model-len 262144 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.85 \
    --reasoning-parser mistral \
    --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'

Notes:

• --monokernel is only supported on MI350X / MI355X (gfx950, 256 CUs) at tensor parallelism TP=4. It supports dspark speculative decoding, and fp8 and bf16 KV cache.
• --speculative-config enables DSpark speculative decoding with the checkpoint's draft head, which is kept out of the quantization (see mtp_dspark.* in EXCLUDE). num_speculative_tokens must be 3, the depth of the draft block; drop the flag for a non-speculative baseline.

Model Quantization

The model was quantized from mistralai/Mistral-Large-4-1T-A52B with AMD-Quark, through the file-to-file flow: the safetensors shards are quantized one at a time, so the 1T-parameter model is never materialized.

from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import QConfig
from quark.torch.quantization.config.template import MXFP4Scheme

SRC = "/path/to/Mistral-Large-4-1T-A52B"
OUT = "/path/to/Mistral-Large-4-1T-A52B-Quark-MXFP4"

# Pre-quantized layers kept in their source format. Native Mistral module names,
# which differ from the Hugging Face layout (layers.N.attention.wq rather than
# model.layers.N.self_attn.q_proj).
EXCLUDE = [
    "*attention.*",
    "*shared_experts.*",
    "*feed_forward.*",
    "*.gate",
    "output",
    "vision_encoder.*",
    "patch_merger.*",
    "vision_language_adapter.*",
    "mtp_dspark.*",
]

quant_config = QConfig(global_quant_config=MXFP4Scheme().config, exclude=EXCLUDE)
ModelQuantizer(quant_config).direct_quantize_checkpoint(
    pretrained_model_path=SRC,
    save_path=OUT,
    keep_excluded_layers_as_original_model_state=True,
    device="cuda",
)

Evaluation

Both checkpoints are evaluated with exactly the same tasks, sampling parameters, maximum generation lengths and number of repeats, on the same host, so the scores are directly comparable; only the serving setup differs (see Evaluation settings).

Summary

mean@K: accuracy averaged over K independent samples, K being given per benchmark in Evaluation settings. Recovery: MXFP4 score divided by the baseline score.

Benchmark	mistralai/Mistral-Large-4-1T-A52B baseline	This model (MXFP4)	Recovery
GPQA Diamond (mean@8)	88.89	88.13	99.15%
AIME26 (mean@8)	98.75	98.75	100.00%
MMMU Pro, Standard (10 options) (mean@2)	77.66	77.37	99.63%
MMMU Pro, Vision (mean@2)	74.60	74.22	99.49%
SciCode-Verified, main problem pass rate (mean@4)	77.34	77.34	100.00%
AA-LCR (mean@16)	79.31	78.44	98.90%
IFBench, prompt_level_loose (mean@5)	75.27	74.87	99.47%
MMLU-Pro, chat (mean@1)	85.40	85.04	99.58%

Note: Results may vary from run to run.

Detailed results

Benchmark	Metric	mistralai/Mistral-Large-4-1T-A52B baseline	This model (MXFP4)
GPQA Diamond	mean@8	88.89	88.13
	maj@8	89.39	89.90
	Mean output tokens per answer	10,389	10,505
AIME26	mean@8	98.75	98.75
	maj@8	100.00	100.00
	Mean output tokens per answer	12,032	12,574
MMMU Pro, Standard (10 options)	mean@2 (run 1 / run 2)	77.66 (77.57 / 77.75)	77.37 (77.40 / 77.34)
	Mean output tokens per answer	4,162	3,728
MMMU Pro, Vision	mean@2 (run 1 / run 2)	74.60 (74.10 / 75.09)	74.22 (74.16 / 74.28)
	Mean output tokens per answer	5,713	5,016
SciCode-Verified (with background)	Main problem pass rate, mean@4	77.34 (71.88-81.25)	77.34 (73.44-79.69)
	Subproblem pass rate, mean@4	88.41 (84.67-91.99)	90.16 (88.85-90.94)
	Mean output tokens per step	5,464	5,354
	Mean output tokens per problem	24,502	24,008
	Steps hitting the grading time limit (1,800 s per environment), rated incorrect	4	7
AA-LCR	mean@16 (per-repeat range)	79.31 (75.00-84.00)	78.44 (75.00-83.00)
	Mean output tokens per answer	910	1,026
IFBench	prompt_level_loose, mean@5 (per-repeat range)	75.27 (74.00-76.33)	74.87 (71.67-77.00)
	prompt_level_strict, mean@5	72.73	72.07
	inst_level_loose / inst_level_strict, mean@5	77.67 / 75.12	77.33 / 74.77
	Mean output tokens per answer	4,894	4,874
MMLU-Pro, chat	mean@1	85.40	85.04
	Mean output tokens per answer	2,147	2,098

SciCode-Verified, AA-LCR and IFBench: the range over the repeats is given in brackets.

Paired per-question comparison (MXFP4 minus baseline), over the units that differ (problems and subproblems for SciCode-Verified):

Benchmark	Baseline better / MXFP4 better
GPQA Diamond (mean@8)	22 / 15 (of 37)
AIME26 (mean@8)	2 / 1 (of 3)
MMMU Pro, Standard (mean@2)	108 / 100 (of 208)
MMMU Pro, Vision (mean@2)	125 / 113 (of 238)
SciCode-Verified, main problem pass rate (mean@4)	13 / 13 (of 26)
SciCode-Verified, subproblem pass rate (mean@4)	33 / 37 (of 70)
AA-LCR (mean@16)	23 / 15 (of 38)
IFBench, prompt_level_loose (mean@5)	55 / 54 (of 109)
MMLU-Pro, chat (mean@1)	331 / 288 (of 619)

Truncated and unparseable answers are rare and are scored incorrect for both models alike: across benchmarks, at most 1.07% of the answers reach the maximum generation length (1.14% with the fp8_ds_mla KV cache below), and at most 0.09% of the MMMU Pro answers carry no extractable answer — all multi-letter replies to badly-posed questions with a single gold letter, a dataset defect that hits both models alike.

Evaluation settings

Benchmark	Harness	Task	Dataset	Size	Maximum generation length	Repeats
GPQA Diamond	lm-evaluation-harness	gpqa_diamond_cot_zeroshot	Idavidrein/gpqa (gpqa_diamond)	198 questions	96,000 tokens	8
AIME26	lm-evaluation-harness	aime26	math-ai/aime26	30 problems	130,000 tokens	8
MMMU Pro	evalscope	mmmu_pro, variants standard (10 options) and vision	MMMU/MMMU_Pro (0d7426df)	1,730 questions per variant	131,072 tokens	2
SciCode-Verified	evalscope	scicode_verified, with scientist background	shhu2001/SciCode-Verified (eea11a86)	64 problems, 287 graded steps	65,536 tokens per step	4
AA-LCR	evalscope	aa_lcr, LLM judge	ArtificialAnalysis/AA-LCR (bdae010b)	100 questions	131,072 tokens	16
IFBench	evalscope	ifbench	allenai/IFBench_test (ed7c1245)	300 prompts	131,072 tokens	5
MMLU-Pro (chat)	lm-evaluation-harness	mmlu_pro_chat, 5-shot as multi-turn chat	TIGER-Lab/MMLU-Pro	14 subjects, 12,032 questions	16,384 tokens	1

mistralai/Mistral-Large-4-1T-A52B is evaluated with tensor parallelism TP=8 and DSpark speculative decoding (--speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'), with BF16 KV cache.

amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 is evaluated with tensor parallelism TP=4, with --monokernel and DSpark speculative decoding (--speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'), with FP8 KV cache (--kv-cache-dtype fp8).

Sampling, for both models and all tasks: temperature=1.0, top_p=0.95, no top_k, no fixed seed, reasoning_effort medium; these are the default sampling parameters of the base checkpoint. The DSA indexer key cache is always FP8.

GPQA Diamond, AIME26 and MMLU-Pro (chat)

lm-evaluation-harness tasks, branch lm-eval-fixes, commit 503ad16:

lm_eval --model local-chat-completions \
    --model_args "model=<server-model-id>,base_url=http://<server-ip>/v1/chat/completions,num_concurrent=16,timeout=99999,tokenized_requests=False,think_end_token='[/THINK]',tokenizer=<tokenizer-path>,tokenizer_backend=huggingface,server_sampling_defaults=True" \
    --tasks gpqa_diamond_cot_zeroshot,aime26,mmlu_pro_chat \
    --gen_kwargs 'reasoning_effort=medium,temperature=1.0,top_p=0.95,until=["[INST]"]' \
    --log_samples --output_path results \
    --apply_chat_template \
    --max_gen_toks_per_task gpqa_diamond_cot_zeroshot=96000 aime26=130000 mmlu_pro_chat=16384 \
    --repeats_per_task gpqa_diamond_cot_zeroshot=8 aime26=8 mmlu_pro_chat=1 \
    --generation-length

One amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 GPQA Diamond answer (fp8 KV cache) was manually re-scored as correct: its final line is The answer is (Bs)., the right letter with a stray character, which the extractor does not parse. There are no such cases for mistralai/Mistral-Large-4-1T-A52B, for AIME26, or with the fp8_ds_mla KV cache.

SciCode-Verified

evalscope datasets, branch scicode-verified-pr (draft PR modelscope/evalscope#1978), commit 701e8ea2:

evalscope eval --model <server-model-id> --eval-type openai_api --api-url http://<server-ip>/v1 --api-key EMPTY \
    --datasets scicode_verified \
    --dataset-args '{"scicode_verified": {"hf_dataset_revision": "eea11a866be6860725258702b39ef8651ed26abd", "extra_params": {"provide_background": true, "prompt_mode": "official", "grading_envs": "or"}}}' \
    --generation-config '{"max_tokens": 65536, "timeout": 5400, "stream": true, "temperature": 1.0, "top_p": 0.95, "reasoning_effort": "medium"}' \
    --repeats 4 --eval-batch-size 128 \
    --sandbox '{"enabled": true, "engine": "docker", "pool_size": 128, "default_config": {"network_enabled": false}}'

A step (subproblem) is rated correct if its tests pass in either the 2024 (numpy 1.26 / scipy 1.13) or the 2025 (numpy 2.4 / scipy 1.17) environment, as in the SciCode-Verified paper, with a 1,800 s limit per environment.

MMMU Pro

The variant is selected with dataset_format, which must sit under extra_params (otherwise evalscope silently runs the standard variant with 4 options). evalscope's vision prompt does not repeat the options as text, so the absolute scores are not directly comparable with NVIDIA or Artificial Analysis numbers; the comparison between the two checkpoints is unaffected.

# Run the command twice per variant (mean@2).
FMT="vision"   # or "standard (10 options)"
evalscope eval --model <server-model-id> --eval-type openai_api --api-url http://<server-ip>/v1 --api-key EMPTY \
    --datasets mmmu_pro --dataset-hub huggingface \
    --dataset-args "{\"mmmu_pro\": {\"hf_dataset_revision\": \"0d7426df4ccb3d8704a992fe6850abee7c262127\", \"extra_params\": {\"dataset_format\": \"$FMT\"}}}" \
    --generation-config '{"max_tokens": 131072, "timeout": 7200, "stream": true, "temperature": 1.0, "top_p": 0.95, "reasoning_effort": "medium"}' \
    --repeats 1 --eval-batch-size 96

AA-LCR

The judge needs room for its reasoning: with max_tokens 8192, 3 / 800 verdicts came back empty because the judge's reasoning reached the limit, hence 32768. The dataset is AA-LCR v1.0.0; the judge alone flips about 1.1% of the verdicts between two judgings, which is part of the noise on the AA-LCR delta.

evalscope eval --model <server-model-id> --eval-type openai_api --api-url http://<server-ip>/v1 --api-key EMPTY \
    --datasets aa_lcr --dataset-hub huggingface \
    --dataset-args '{"aa_lcr": {"hf_dataset_revision": "bdae010bbce259820c0e34c1d7cce210d966fb75"}}' \
    --generation-config '{"max_tokens": 131072, "timeout": 7200, "stream": true, "temperature": 1.0, "top_p": 0.95, "reasoning_effort": "medium"}' \
    --repeats 16 --eval-batch-size 16 \
    --judge "$(cat aa-lcr-judge-dsv41.json)"

with the judge configuration aa-lcr-judge-dsv41.json (<judge-host>: the judge server, deepseek-ai/DeepSeek-V4.1-Flash-1017 on vLLM). Both checkpoints are judged by the same judge server:

{"strategy": "llm", "models": [{
  "model_id": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "api_url": "http://<judge-host>:8000/v1", "api_key": "EMPTY",
  "eval_type": "openai_api",
  "generation_config": {"reasoning_effort": "high", "max_tokens": 32768, "timeout": 7200}
}]}

IFBench

Metric prompt_level_loose reported; published IFBench results usually use a higher reasoning effort (max or xhigh), here it is medium. Both models had 1 / 1,500 IFBench answers run to the maximum generation length, each on an adversarial word- or syllable-level constraint (model behaviour, scored as failed).

evalscope eval --model <server-model-id> --eval-type openai_api --api-url http://<server-ip>/v1 --api-key EMPTY \
    --datasets ifbench --dataset-hub huggingface \
    --dataset-args '{"ifbench": {"hf_dataset_revision": "ed7c12452594b7028b15807b86d3938150b663ae", "primary_metric": "prompt_level_loose"}}' \
    --generation-config '{"max_tokens": 131072, "timeout": 7200, "stream": true, "temperature": 1.0, "top_p": 0.95, "reasoning_effort": "medium"}' \
    --repeats 5 --eval-batch-size 128

Common settings

• The evaluation servers run without --reasoning-parser mistral for lm-eval, which reads the reasoning trace from content, and with --reasoning-parser mistral for evalscope (with a fix for reasoning leaking into content in multi-turn streaming, to be upstreamed).

The correctness of the evaluation results was checked by an agent.

Scoring notes:

• A generation that hits the maximum generation length, never closes its reasoning ([/THINK]), or gives an empty final answer is scored incorrect and does not vote in maj@8.
• maj@8 ties (no single most frequent answer among the 8 repeats) are broken by the answer that appears first in repeat order. On GPQA Diamond, 0 of 198 questions are ties for mistralai/Mistral-Large-4-1T-A52B and 1 of 198 for amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 (3 of 198 with the fp8_ds_mla KV cache below), so maj@8 can move by up to about 0.5 point for these two models (1.5 points with the fp8_ds_mla KV cache) depending on the tie rule. AIME26 has no ties.
• Mean output tokens per answer: mean generation length over all generations, reasoning trace included.

Reference only: results with fp8_ds_mla KV cache

Evaluation results with fp8_ds_mla KV cache (unreleased on Instinct, reference only):

Model	Task	KV cache dtype	Repeats	mean	maj@8	Mean output tokens per answer
This model (MXFP4)	GPQA Diamond	fp8_ds_mla	8	87.50	88.38	10,410
This model (MXFP4)	AIME26	fp8_ds_mla	8	99.17	100.00	11,637
This model (MXFP4)	MMLU-Pro (chat)	fp8_ds_mla	1	85.26	N/A	2,087

These results use TP=4 with DSpark speculative decoding and without --monokernel.

Difference with --kv-cache-dtype fp8:

• fp8 on AMD Instinct: the latent KV cache (512 latent + 64 RoPE values per token, 576 bytes) is stored as plain FP8 E4M3 with a single per-tensor scale of 1.0 (the checkpoint ships no KV scales).
• fp8_ds_mla: reference only, using the DeepSeek-V3.2 / FlashMLA fp8_ds_mla KV cache layout: 656 bytes per token, the 512 latent values in FP8 E4M3 with one FP32 power-of-two scale per block of 128 values, and the 64 RoPE values in BF16. The per-block scales keep small-magnitude latents (e.g. in the first layers) more precise than a single per-tensor scale.

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.