Instructions to use RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds

SGLang

How to use RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds with Docker Model Runner:
```
docker model run hf.co/RedHatAI/MiniChat-1.5-3B-pruned50-quant-ds
```

MiniChat-1.5-3B-pruned50-quant-ds / README.md

mgoin

Update README.md

325940c over 2 years ago

preview code

Raw

History Blame Contribute Delete

3.7 kB

	---
	base_model: GeneZC/MiniChat-1.5-3B
	inference: false
	model_type: llama
	prompt_template: \|
	<s> [\|User\|]\n
	{prompt}</s>
	[\|Assistant\|]\n
	quantized_by: mwitiderrick
	tags:
	- deepsparse
	---
	# MiniChat-1.5-3B - DeepSparse
	This repo contains model files for [MiniChat-1.5-3B](https://huggingface.co/GeneZC/MiniChat-1.5-3B) optimized for [DeepSparse](https://github.com/neuralmagic/deepsparse), a CPU inference runtime for sparse models.

	This model was quantized and pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
	## Inference
	Install [DeepSparse LLM](https://github.com/neuralmagic/deepsparse) for fast inference on CPUs:
	```bash
	pip install deepsparse-nightly[llm]
	```
	Run in a [Python pipeline](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md):
	```python
	from deepsparse import TextGeneration

	prompt = "How to get in a good university?"
	formatted_prompt = f"<s> [\|User\|]\n{prompt}</s>[\|Assistant\|]\n"

	model = TextGeneration(model_path="hf:neuralmagic/MiniChat-1.5-3B-pruned50-quant-ds")

	print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
	"""
	As an AI, I don't have personal experiences or opinions, but I can provide you with some general advice on how to get into a good university. Here are some tips to consider:

	1. Academic performance: A good university requires good academic performance. This means you need to maintain a good GPA (grade point average) and achieve high marks in your courses. To do this, you need to put in the effort to learn and understand the course material.

	2. Pursue a diverse range of courses: A good university student should not limit themselves to just one area of study. They should take courses in various fields that interest them. This will help them develop a wide range of skills and knowledge.

	3. Networking: A good university student should network with others in their courses and beyond. This can be done through attending events like guest lectures, group meetings, and social events.

	4. Be proactive
	"""
	```

	## Prompt template
	```

	<s> [\|User\|]\n
	{prompt}
	</s>[\|Assistant\|]\n
	```
	## Sparsification
	For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.

	```bash
	git clone https://github.com/neuralmagic/sparseml
	pip install -e "sparseml[transformers]"
	python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-1.5-3B open_platypus --recipe recipe.yaml --save True
	python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
	cp deployment/model.onnx deployment/model-orig.onnx
	```
	Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
	```python
	import os
	import onnx
	from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
	input_file = "deployment/model-orig.onnx"
	output_file = "deployment/model.onnx"
	model = onnx.load(input_file, load_external_data=False)
	model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
	onnx.save(model, output_file)
	print(f"Modified model saved to: {output_file}")
	```
	Follow the instructions on our [One Shot With SparseML](https://github.com/neuralmagic/sparseml/tree/main/src/sparseml/transformers/sparsification/obcq) page for a step-by-step guide for performing one-shot quantization of large language models.
	## Slack

	For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)