Instructions to use mlx-community/Qwen3.8-27B-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/Qwen3.8-27B-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("mlx-community/Qwen3.8-27B-4bit") config = load_config("mlx-community/Qwen3.8-27B-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use mlx-community/Qwen3.8-27B-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "mlx-community/Qwen3.8-27B-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "mlx-community/Qwen3.8-27B-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use mlx-community/Qwen3.8-27B-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "mlx-community/Qwen3.8-27B-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default mlx-community/Qwen3.8-27B-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use mlx-community/Qwen3.8-27B-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "mlx-community/Qwen3.8-27B-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "mlx-community/Qwen3.8-27B-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
M6 Mac mini 24GB benchmark: context limits, prefill and MTP results
Sharing hardware measurements for people considering this model on a 24GB M6 Mac mini.
I tested Qwen3.8-27B Q4 across context sizes, including generation with and without the MTP head and prompt-cache reuse. I measured approximately 270 tok/s average prefill, 9.8 tok/s ordinary decoding, and up to 17.5 tok/s with MTP, with lower generation speeds at longer contexts.
Raising the macOS GPU wired-memory limit to 20GB gave roughly 30K usable context in my setup. Around 40K was possible with a 21GB limit in stress testing, but that leaves little headroom for macOS and other applications.
These are measurements on the 24GB machine. The larger context capacity discussed for a 32GB configuration is an estimate, not a tested result.
Code and data: https://github.com/lpalbou/medium/tree/main/qwen-mtp-bench
Methodology, charts and discussion: https://medium.com/@lpalbou/mac-mini-m6-qwen3-8-27b-a-glimpse-at-apples-local-ai-future-1b2d36aa2c16
Comparable runs on an M5 Pro Mac mini would be particularly useful, ideally with the same quantization, context sizes and cache settings.
useful, thanks. running the same 4-bit on a 64GB M5 Pro mini this weekend, will add my numbers here so people can compare 24 vs 64
my numbers from the 64GB M5 Pro mini, same mlx-community 4-bit in mlx-lm, stock wired memory limit. short prompts (~220 tokens in), thinking off, no MTP: about 340 tok/s prefill, 17.8 tok/s decode, 16.2 GB peak memory. didn't test long context or MTP yet so can't match your context sweep. raw data here: https://huggingface.co/datasets/pratikgx/1000-rewrites-later
So M5 pro about 25% faster in prefill than M6 and in decode - due to memory bandwidth -, about twice as fast. It remains a nice "cheap" solution compared to a M5 pro, but if budget is not an issue and form factor is a must, the M5 mini pro is definitely a small beast.
fyi, created a small app to automatically benchmark machines: https://github.com/lpalbou/wattever + GH page: https://www.lpalbou.info/Wattever/ where we can just PR our results into the repo to share it more broadly. eg a benchmark I am doing on my rtx 5060ti: runs