Instructions to use Pinkstack/DistilGPT-OSS-qwen3-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Pinkstack/DistilGPT-OSS-qwen3-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Pinkstack/DistilGPT-OSS-qwen3-4B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Pinkstack/DistilGPT-OSS-qwen3-4B")
model = AutoModelForCausalLM.from_pretrained("Pinkstack/DistilGPT-OSS-qwen3-4B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Pinkstack/DistilGPT-OSS-qwen3-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Pinkstack/DistilGPT-OSS-qwen3-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pinkstack/DistilGPT-OSS-qwen3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Pinkstack/DistilGPT-OSS-qwen3-4B

SGLang

How to use Pinkstack/DistilGPT-OSS-qwen3-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Pinkstack/DistilGPT-OSS-qwen3-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pinkstack/DistilGPT-OSS-qwen3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Pinkstack/DistilGPT-OSS-qwen3-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pinkstack/DistilGPT-OSS-qwen3-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Pinkstack/DistilGPT-OSS-qwen3-4B with Docker Model Runner:
```
docker model run hf.co/Pinkstack/DistilGPT-OSS-qwen3-4B
```

Other models?

by twoxfh - opened Sep 25, 2025

Discussion

twoxfh

Sep 25, 2025

First, thank you for this distillation, its been a lot of fun working with it.

Curious to know if you plan on opening up the entire process and dataset so other architecture models could be distilled in a similar way? LFM2 2.6B is reporting performance close to 4B models and it would be interesting to see the same process implemented with it to greater enhance the reach to CPU bound enthusiasts.

Thanks for reading, appreciate your work.

Pinkstack

Owner Sep 26, 2025

•

edited Sep 26, 2025

Thank you for your support! 💜

While currently not planned, opening the dataset could be indeed very helpful to some, probably without the mixed in gpt OSS 120b outputs though. README will be updated soon but a large portion of the dataset is based on andyrdt/gpt-oss-20b-rollouts but highly filtered and reformatted.

Quite a bit of information was already shared in the README, training was done using unsloth for more optimized fine tuning, other settings do not matter as much as it changes between model to model and what kind of system you are using for fine tuning.

As for other smaller models, it is theoretically possible and will be looked into, not sure how LFM would perform, though I did test it a little and it was okay. Other unrelated models are currently planned but after they are done I will try to do a smaller run on lfm2 2.6b.

Edit:
Just to add, tests on other architectures was done, we continually pretrained and then did sft on a custom 6b moe too but it performed very poorly compared to this model.

Pinkstack

Owner Sep 30, 2025

dataset uploaded

BingoBird

Oct 5, 2025

Still amazes me that behavior patterns can be distilled and transferred.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment