LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM prebuilt-adac974b · GPU · decode 22.1 tok/s · prefill 338 tok/s · TTFT 3.07 s · all 1210 ops delegated (2026-10-05); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 6.8 tok/s · prefill 8 tok/s · TTFT 1.68 s · all 1187 ops delegated (2026-08-17); Mac Studio M4 Max · LiteRT-LM 0.17.1 · GPU · decode 129.5 tok/s · prefill 1340 tok/s · TTFT 201 ms · all 1210 ops delegated (2026-10-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/ministral3-3b/CARD.md

Ministral-3-3B-Instruct-2512 — LiteRT-LM (blockwise int4)

mistralai/Ministral-3-3B-Instruct-2512 converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Text-only conversion (the Ministral-3 text decoder; the Pixtral vision tower is dropped).


File	`Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm` (~2.3 GB; embedding externalized so every section is <2 GiB → loads on iOS)
Quantization	int4 weights — blockwise (block 32), symmetric, OCTAV clipping; tied embedding/lm_head at INT8
Compute	integer
Context (KV cache)	4096
Base model	mistralai/Ministral-3-3B-Instruct-2512 (Apache-2.0)

2026-10-06: GPU graph rewrite; the weights are unchanged. The previous file was exported without litert-torch's --apply_gpu_composites. Its graph wrote the KV cache with two DYNAMIC_UPDATE_SLICE ops per layer and ran attention with BATCH_MATMUL. The new graph has the shape that flag exports: one odml.cache_update composite per layer writes the KV cache, odml.runtime_bmm composites compute the attention products, and a signature input param_tensor (INT32[1,1,1,7]) is added for the runtime to fill. The attention mask stays FLOAT32 and is added to the logits by broadcast. The file also gains a 16-token prefill signature, copied from prefill_128. The copy shares every weight buffer, so the file grows by graph structure only. Every other bundle section and every weight byte is identical to the previous file (checked per section and per buffer). The new file is 2,342,866,928 bytes, sha256 13c47bcbaf305e4337f280b27c991168b0c624a6d60e00df10f0f23ebd28ad83 (previous: 2,340,982,768 bytes, sha256 57daeeb4…). On the CPU, the 16-token signature's output is bit-identical to prefill_128's on the same tokens (2 of 2 cases). The answers to 8 questions and 1 long prompt are byte-identical with and without the new signature, both on the CPU and on the Mac GPU with fp32 activations. Without the new signature, the rewritten graph also answers the 8 questions byte-identically to the previous file (Mac GPU, fp32 activations). With the default fp16 activations on the Mac GPU, the new file gets all 8 final answers right and passes the long prompt. On 30 GSM8K questions (greedy, up to 2048 tokens) it gets 26 right, as the previous file does; one question differs in each direction. The Raspberry Pi 5, Pixel 8a and iPhone 17 Pro figures further down were measured on the previous file and not re-measured.

Apple M4 Max, litert-lm 0.17.1, `benchmark --cache no`	previous file	new file	new / previous
GPU decode, 256 tokens	92.4 tok/s	129.5 tok/s	×1.401
GPU prefill, 256 tokens	1215 tok/s	1340 tok/s	×1.103
GPU TTFT, 16-token prompt	0.121 s	0.038 s	×0.310
CPU decode, 256 tokens	22.1 tok/s	22.5 tok/s	×1.014
CPU TTFT, 16-token prompt	1.478 s	0.589 s	×0.398
CPU init	2.2 s	3.1 s	×1.416

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm \
  --backend gpu \
  --input_prompt "Explain on-device AI in one sentence."

The .litertlm bundle carries the tokenizer and the prompt template (Ministral's native Mistral [INST] … [/INST] format, stop token </s>), so no separate tokenizer files are needed. This is a direct-answering instruct model (no <think> block) and terminates cleanly at </s>.

Run on Android

The easiest way to try this model on a phone is the official Google AI Edge Gallery app — it runs .litertlm models fully on-device and can import your own:

Install a recent Gallery (package com.google.ai.edge.gallery, APK from the repo's releases — 1.0.15+ supports .litertlm). Older 1.0.x builds (package com.google.aiedge.gallery) only accept the legacy MediaPipe .task format and reject .litertlm.
Download Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm from this repo and push it to the device:
```
adb push Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm /sdcard/Download/
```
In the app, tap the + button (bottom-right), pick the file, and choose the GPU backend (CPU also works).
Chat. Nothing else to configure — the .litertlm bundle already carries the tokenizer and prompt template, so the model uses its native Mistral [INST] chat format automatically.

Device RAM (important — this is a ~2.7 GB / 3B model): GPU on Android needs roughly 2× the model size (the weights plus the ML Drift GPU weight cache), so GPU is only offered on ~12 GB+ devices. On an 8 GB phone (e.g. Pixel 8a) only CPU is selectable, and you must free RAM first (close apps / reboot → ~4 GB free) or the app is OOM-killed on load. For smaller phones, prefer a 1–2B model (e.g. a Qwen3-1.7B .litertlm), which runs comfortably and can use the GPU.

What the runtime can do, measured (added 2026-08-17, previous file): the paragraph above is about what the Gallery app offers — its accelerator choice is fixed at import time from device RAM. The model itself is not the limit: driving litert_lm_main directly on a Pixel 8a (Tensor G3, Mali-G715, 8 GB), the previous file ran entirely on the OpenCL delegate — 1187/1187 nodes in the 128-token prefill graph and 1087/1087 in decode, zero rejected ops — and answered correctly. So on an 8 GB phone the GPU is out of reach through the Gallery, not out of reach for the model. The new file was not re-measured on this phone.

See the Gallery Importing Local Models guide for details. To embed the model in your own Android app instead, use the LiteRT-LM Kotlin API (Gradle artifact com.google.ai.edge.litertlm:litertlm-android, getting started).

Performance

The three speed tables below were measured on the new file (the 2026-10-06 upload).

Apple M4 Max

Mac Studio (M4 Max), litert-lm 0.17.1 (pip CLI), litert-lm benchmark --cache no: 256 prompt and 256 decode tokens, median of 3 processes on the GPU (WebGPU on Metal, fp16 activations) and of 2 on the CPU (XNNPACK, no weight cache). TTFT (16) comes from a separate run with a 16-token prompt and 32 decode tokens, same flags.

Backend	Prefill (256)	Decode	TTFT (256)	TTFT (16)	Init
GPU (WebGPU on Metal)	1340 tok/s	129.5 tok/s	0.20 s	0.038 s	2.8 s
CPU (XNNPACK)	125 tok/s	22.5 tok/s	2.51 s	0.589 s	3.1 s

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Galaxy S26

Samsung Galaxy S26 (SM-S942Q, Android 16), litert_lm_advanced_main built from LiteRT-LM main (2026-09-18 build) with libLiteRtOpenClAccelerator.so. GPU: OpenCL, bundle default fp16 activations, --disable_cache=true (cold init). CPU: XNNPACK, 4 threads, with the weight cache (the init writes it). Prefill, Decode and TTFT (1024): a 1024-token prompt and 256 decode tokens, 3 iterations in one process on the GPU and 2 on the CPU. TTFT (16): a 16-token prompt and 32 decode tokens, 3 iterations in one process. A range is the lowest and highest iteration. In the 1024-token GPU run, decode was lowest on the third iteration; TTFT (16) was highest on the first iteration. Peak is the process high-water mark (VmHWM) of the 1024-token run.

Backend	Prefill (1024)	Decode	TTFT (1024)	TTFT (16)	Init	Peak
GPU (OpenCL)	315.6–338.8 tok/s	21.85–22.16 tok/s	3.07–3.29 s	0.11–0.13 s	10.64 s	1.40 GB
CPU (XNNPACK)	43.7–61.0 tok/s	8.24–8.73 tok/s	16.91–23.55 s	0.28–1.33 s	4.25 s	3.68 GB

iPhone 18 Pro

iPhone 18 Pro (iPhone19,2, iOS 27.0), LiteRT-LM v0.17.1 xcframework (same bytes as v0.17.0) through a self-built harness app (C API / Swift wrapper), default memory entitlement (~3,369 MB). GPU: Metal. CPU: XNNPACK with the disk weight cache. Each row is 2 timed runs, each after a 6-minute rest; the thermal state was nominal at the start and end of every run.

Backend	Prompt	Prefill	Decode	TTFT	Init
GPU (Metal)	128 tokens	277.6 tok/s	44.0 tok/s	0.48 s	6.54 s
GPU (Metal)	16 tokens	337.1 tok/s	43.7 tok/s	0.074 s	6.67 s
CPU (XNNPACK)	128 tokens	99.0 tok/s	18.1 tok/s	2.44 s	2.24 s
CPU (XNNPACK)	16 tokens	81.5 tok/s	15.5 tok/s	0.599 s	2.28 s

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for #### <n>, identical prompt and answer-extraction for both rows so the only variable is the on-device quantization:

Configuration	GSM8K
bf16 (reference)	89.0%
This model — LiteRT int4	85.0%

LiteRT int4 is at parity: −4 pt vs bf16 with no reasoning collapse. The model also passes the local quality gate 8/8 (non-degenerate, clean stop at </s>). 85% is a strong on-device GSM8K for a 3B and far above a naive min-max int4 of the same model (blockwise-32 + OCTAV optimal-clipping is what preserves the accuracy).

Galaxy S26 — GPU backend

On the Galaxy S26 GPU, every transformer signature of the new file is fully delegated: one partition each on the OpenCL delegate (LITERT_CL), with the same node counts as on the Mac.

file	`prefill_128`	`prefill_16`	`decode`
`Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm`	1210 / 1210 nodes	1210 / 1210	1112 / 1112

Measured on a Samsung Galaxy S26 (SM-S942Q, Android 16) with litert_lm_advanced_main built from LiteRT-LM main (2026-09-18 build) and libLiteRtOpenClAccelerator.so; GPU activations are the bundle default fp16. Speed and peak memory are in the Performance tables above.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with litert-torch using a blockwise int4 recipe (INT4 weights, block size 32, symmetric, OCTAV optimal-clipping) with the tied embedding/lm_head kept at INT8, KV cache 4096, and Ministral's native Mistral chat template. Ministral-3 is a standard dense decoder (Ministral3ForCausalLM, YaRN RoPE), so it rides the existing converter and runtime with no custom graph code; only the text decoder is exported (the vision tower is dropped first).

externalize_embedder=True (required for iPhone). This 3B's weights would otherwise be a single ~2.55 GiB TFLite section, which exceeds the ~2 GiB single-section mmap limit on iOS — engine creation fails with "Failed to map section: Cannot allocate memory". Externalizing the (tied) embedding into its own section drops the main weights section to ~1.8 GiB (and dedups the tied matrix, ~2.74 GB → 2.34 GB total), so the model loads on iPhone (Metal GPU) as well as Android/desktop. Same weights, so GSM8K parity is unchanged. Verified on-device (previous file): iPhone 17 Pro decodes at 14–18 tok/s on the Metal GPU (prefill ~41 tok/s, TTFT 0.40 s, ~1.45 GB footprint) — the previously-failing ">2 GiB section / Cannot allocate memory" mmap error no longer occurs.

Template note (important for any Mistral/Ministral): the model must be exported with its native Mistral [INST] … [/INST] template and real EOS </s> — not ChatML. Mistral's tekken tokenizer has no <|im_end|> token, so under a ChatML template the int4 model never hits a registered stop token and runs away after the correct answer. With the Mistral template it stops cleanly.

Reproduce

Built with litert-torch and a blockwise-32 + OCTAV int4 recipe, forcing the simple Mistral [INST] chat template (the model's full jinja template doesn't render in the runtime's minimal jinja engine, so the structured [INST] prefixes are extracted instead):

EXTERNALIZE_EMBEDDER=1 CACHE=4096 python scripts/export_simple_template.py \
    src_models/ministral3-3b-text \
    out/ministral3-3b-boctav4 \
    templates/mistral_simple.jinja \
    BOCTAV4   # blockwise-32 int4 + OCTAV, int8 embeddings

The equivalent ai_edge_quantizer recipe is included as ministral3_int4_block32_octav.json. The text decoder is extracted from the multimodal checkpoint with scripts/extract_ministral3_text.py (drops the vision tower; loads with missing=0/unexpected=0).

Graph rewrite (2026-10-06). The file was rebuilt from the previous file with two tools from tools/gpu_graph/ in https://github.com/john-rocky/hf-to-litertlm. gpu_graph_retrofit.py turns the two DYNAMIC_UPDATE_SLICE KV-cache writes per layer into one odml.cache_update composite, turns the BATCH_MATMUL attention products into odml.runtime_bmm, adds the param_tensor input, adds the FLOAT32 attention mask to the logits by broadcast (--mask add_bcast), and keeps every original weight buffer's bytes. prefill_bucket_clone.py copies an existing prefill signature at a new length; the copy shares every weight buffer, and every existing subgraph and SignatureDef keeps its index and bytes.

python tools/gpu_graph/gpu_graph_retrofit.py previous.litertlm retrofit.litertlm --decomp exporter --mask add_bcast
python tools/gpu_graph/prefill_bucket_clone.py build retrofit.litertlm new.litertlm --lengths 16 --source prefill_128 --report bucket.json

A rebuild from the previous file gives every bundle section byte-identical to the shipped file. Only the bundle header (uuid and timestamp written by litert-lm pack) differs, so the whole-file sha256 differs.

Training data & PII

This is a weights-exact format conversion of mistralai/Ministral-3-3B-Instruct-2512; no new training was performed. Ministral was trained by Mistral AI on a large-scale, undisclosed proprietary mixture of web, code, and multilingual text. As the corpus is web-derived it may incidentally contain personal data/PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for available details.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: You are Ministral-3-3B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris. You power an AI assistant called Le Chat. Your knowledge base was last updated on 2023-10…. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm now emits the block exactly once when no system message is given. In Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm, the block is not emitted when you pass a system message. The restored block adds 523 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

Raspberry Pi 5 (CPU) (previous file)

These rows were measured before the 2026-10-06 graph rewrite, on the previous graph (the file it replaced: sha256 57daeeb4…), and were not re-measured; the weights are byte-identical and the new file's Mac CPU prefill, decode and TTFT are ×0.995–×1.039 of the previous file's, so the speed columns are expected to hold.

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
`Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm`	13.7 (13.7–13.8)	1.9 (1.9–1.9)	21.3 s	3.7 GB

Known runtime issues

External weights (since 2026-10-30). In Ministral-3-3B-Instruct-2512_q4_block32_ekv4096.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier file with the weights inside the graph stays at revision d27c2af.

From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.