TinyBalls-110M-V1

A 113M-parameter LLaMA-style instruct model, trained from scratch on 2B tokens and then fine-tuned for chat and tool use.

The base (non-instruct) model is a separate repo: igidn/TinyBalls-v1-Base.

Architecture

params	113.3M (151.0M counting the tied head twice)
layers	12
hidden size	768
attention	12 query heads / 4 KV heads (GQA), head dim 64
FFN	SwiGLU, intermediate 2048, pre-activations clamped at 10.0
vocab	49,154 (SmolLM2 byte-level BPE + ChatML and tool specials)
context	32,768 (8,192 used in practice, see below)
tied embeddings	yes
RoPE theta	500,000

Weights are float32 and load as a standard transformers LlamaForCausalLM.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("igidn/TinyBalls-110M-V1")
model = AutoModelForCausalLM.from_pretrained(
    "igidn/TinyBalls-110M-V1", torch_dtype="float32")

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=128,
                     do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:]))

Generation stops at <|im_end|> (id 2), which is the configured EOS. The sampling settings in generation_config.json (temperature=0.8, top_p=0.95) are the ones to use — greedy decoding is prone to repetition loops on this checkpoint. Every number below was measured under sampling.

Chat format

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{content}<|im_end|>
<|im_start|>tool
<|tool_result|>{result}<|im_end|>

An assistant turn is optional <think>...</think> reasoning, then text, then zero or more <|tool_call|>{...} segments, then <|im_end|>. There is no newline after the closing <|im_end|> — the next turn's <|im_start|> follows immediately.

Tool calling

Tool schemas go in the system message as full JSON, and the model replies with <|tool_call|>{"name": "...", "arguments": {...}} and nothing after it:

<|im_start|>system
You are a helpful assistant with access to tools. Use the provided tools when
they can help answer the user's request. If a tool call is needed, put any
reasoning in <think>...</think> first.
Available tools:
[{"type": "function", "function": {"name": "get_weather", ...}}]
To call a tool, reply with <|tool_call|>{"name": "...", "arguments": {...}} and nothing after it.<|im_end|>

Use the chat_template in tokenizer_config.json as the format of record.

Training

Three stages, each from igidn/tinyballs-v1:

1. Pretraining, 2.01B tokens (1.5B broad + 0.5B quality anneal) on an English-only mix of web, books, Wikipedia, code, math, papers and forums. Sequence length 2048, RoPE theta 10,000.
2. SFT on 90,804 conversations / 161M tokens of distilled chat, tool-calling and agentic data, in three phases. RoPE theta is raised 10k → 50k → 500k alongside context 2048 → 8192 → 32768.
3. Tool-call repair, 630 steps / 41.3M tokens on a targeted mix for schema binding, abstention, error recovery and grounded answers. This is the released checkpoint.

Files

file
model.safetensors	fp32 LlamaForCausalLM weights
config.json	architecture, rope_theta 500000, 32k context
tokenizer.json	the exact tokenizer these weights were trained with
tokenizer_config.json	specials + ChatML chat_template
generation_config.json	sampling defaults, EOS = `<
Modelfile	Ollama serving template

License

MIT.

-igidn