GPT3-dev-125m-0610-instruct

An experimental English assistant with 125 million parameters, built with my custom GPT3Dev implementation of a GPT-3-style decoder. This release combines supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and a chosen-response likelihood anchor (RPO).

The main improvement is more stable, shorter assistant responses and more consistent stopping. Capabilities are still limited: this is a small research model for learning, local experiments, and further fine-tuning.

Project code: krll-corp/GPT3.

Base model: GPT3-dev-125m-1005.

Model details

Property	Value
Parameters	125,231,616
Architecture	Decoder-only GPT3Dev with pre-LayerNorm
Layers / attention heads	12 / 12
Hidden / feed-forward size	768 / 3,072
Attention	Alternating dense and local causal attention; local window setting 256
Context length	2,048 tokens, shared by prompt and generated response
Position embeddings	Learned absolute embeddings
Vocabulary	50,264 tokens; GPT-2 byte-level BPE with added Harmony chat tokens
Weights	Full model in Safetensors, FP32; approximately 501 MB
EOS / padding	<|end|> (token ID 50258)
License	MIT

Post-training preserved the supplied model architecture, tensor shapes, tokenizer vocabulary, and chat template. The included modeling_gpt3dev.py is byte-identical to the base model. Loading uses the standard Transformers auto API with trust_remote_code=True.

Usage

The example uses CPU and FP32. The reference environment is Transformers 4.52.4 and PyTorch 2.9.1; use the pinned Transformers version for this example.

pip install torch "transformers==4.52.4" safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "k050506koch/GPT3-dev-125m-0610-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
)
model.eval()

messages = [{"role": "user", "content": "Explain overfitting simply."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        num_beams=1,
        no_repeat_ngram_size=4,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

Set model_id to the downloaded repository directory to load locally. Keep the prompt and generation budget within the 2,048-token context limit. EOS and padding are already configured in the repository.

The example uses deterministic greedy decoding with no_repeat_ngram_size=4 to reduce repetition. Remove that argument for raw greedy decoding. The two settings are reported separately below. In my chat tests, num_beams=5 improved response quality. This beam-search setting is separate from the sampled Pass@5 metric below.

Chat format

Use tokenizer.apply_chat_template() with add_generation_prompt=True. The included template formats a single exchange as:

<|start|>user<|message|>Your question.<|end|><|start|>assistant<|channel|>final<|message|>The answer.<|end|>

The generation prompt ends at <|start|>assistant<|channel|>final<|message|>. This checkpoint uses the bundled Harmony-style template; use it when adapting examples from earlier releases.

Post-training

1. Broad completion-only SFT: 6,000 filtered examples from SmolTalk and Smol-SmolTalk, covering short conversations and instruction constraints. Loss was applied to assistant completions. Evaluation prompts and user feedback prompts were excluded from this stage.
2. Targeted SFT: a reviewed 90-example curriculum covering concise causal explanations, faithful messages, and short stories, with a separate 30-example validation split.
3. Preference optimization: conservative sigmoid DPO with a 0.1 chosen-response NLL anchor (RPO), at learning rate 3e-7. The selected parent was step 18 of a 27-step targeted DPO run.
4. Final SFT correction: four optimizer steps at learning rate 2e-6, using branch-aligned targets and balanced replay. This corrected greedy token choices after preference optimization.

The broad SFT datasets were pinned to revisions 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa (SmolTalk) and f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc (Smol-SmolTalk). Targeted examples and preference pairs were constructed and reviewed for this experiment.

MMLU evaluation

Evaluated locally on the MMLU test splits across all 57 subjects. The reported averages give equal weight to each subject. Accuracy and Pass@k values are fractions between 0 and 1.

Metric	Subject average
Greedy accuracy	0.2636 (26.36%)
Sampled Pass@1	0.2625 (26.25%)
Sampled Pass@5	0.3040 (30.40%)
Perplexity	394.4300

The custom evaluator scores each question + choice string by its summed next-token cross-entropy over the full padded sequence, including the question and padding. Greedy accuracy selects the lowest-loss choice. Sampled Pass@k draws k choices with replacement from softmax(-loss) after probability filtering, and counts a question as correct if any draw matches the answer. No few-shot examples or chat template are used.

The evaluation uses CPU, batch size 8, a 512-token limit, seed 1234, unscaled logits, top_p=0.95, and a probability cutoff of min_p=0.05. Perplexity is computed over the padded question–choice sequences for all four options, then averaged across subjects. These results depend on the evaluator's scoring and padding convention; direct comparisons require the same method.

MMLU results by subject (57 subjects)

Subject	Sampled Pass@1	Sampled Pass@5	Greedy accuracy	Perplexity
abstract_algebra	0.3400	0.4000	0.3100	264.3209
anatomy	0.2889	0.3333	0.2815	421.6698
astronomy	0.2697	0.3421	0.2829	193.8157
business_ethics	0.2000	0.2200	0.2000	366.9200
clinical_knowledge	0.2453	0.2906	0.2264	195.2811
college_biology	0.2361	0.2569	0.2222	665.2739
college_chemistry	0.2400	0.3100	0.2600	483.5550
college_computer_science	0.2600	0.2800	0.2500	497.7610
college_mathematics	0.2400	0.2900	0.2300	303.0469
college_medicine	0.2601	0.3064	0.2659	168.4599
college_physics	0.2745	0.3431	0.3039	254.3922
computer_security	0.1600	0.1900	0.1800	1477.3218
conceptual_physics	0.1915	0.2681	0.2000	789.0711
econometrics	0.2281	0.2281	0.2281	746.7415
electrical_engineering	0.3034	0.3655	0.3310	745.9070
elementary_mathematics	0.2540	0.3254	0.2698	212.0642
formal_logic	0.2381	0.2937	0.2460	227.5306
global_facts	0.2500	0.3300	0.2400	395.9895
high_school_biology	0.2419	0.2710	0.2452	224.4375
high_school_chemistry	0.3103	0.3300	0.2956	479.8817
high_school_computer_science	0.2700	0.3000	0.2700	421.5334
high_school_european_history	0.2788	0.3212	0.2848	82.8736
high_school_geography	0.2576	0.3182	0.2677	267.0929
high_school_government_and_politics	0.2487	0.2902	0.2642	376.1947
high_school_macroeconomics	0.2590	0.3154	0.2641	313.0468
high_school_mathematics	0.2926	0.3481	0.2889	377.4047
high_school_microeconomics	0.2731	0.3445	0.2899	214.0582
high_school_physics	0.2781	0.3245	0.2781	117.3412
high_school_psychology	0.2917	0.3119	0.2954	540.3541
high_school_statistics	0.2593	0.3056	0.2685	86.3663
high_school_us_history	0.2892	0.3284	0.3039	63.8813
high_school_world_history	0.2363	0.2869	0.2489	58.5576
human_aging	0.2108	0.2332	0.1928	916.5548
human_sexuality	0.2595	0.2824	0.2672	745.4502
international_law	0.4711	0.4793	0.4463	764.2953
jurisprudence	0.1852	0.2222	0.1667	179.4332
logical_fallacies	0.2945	0.3190	0.2883	979.5042
machine_learning	0.2500	0.2857	0.2589	471.1254
management	0.3107	0.3398	0.3107	475.1721
marketing	0.2863	0.3462	0.2735	85.6438
medical_genetics	0.3600	0.4500	0.3600	741.9251
miscellaneous	0.2797	0.3103	0.2708	271.5844
moral_disputes	0.2283	0.2688	0.2399	235.3917
moral_scenarios	0.2369	0.3173	0.2413	73.4469
nutrition	0.2647	0.2908	0.2647	567.7101
philosophy	0.2476	0.2926	0.2476	290.7084
prehistory	0.2377	0.2809	0.2562	177.9805
professional_accounting	0.2376	0.2943	0.2518	98.1543
professional_law	0.2549	0.2803	0.2542	15.3536
professional_medicine	0.2610	0.2721	0.2574	148.1690
professional_psychology	0.2271	0.2467	0.2255	410.0942
public_relations	0.2818	0.3091	0.2636	398.9774
security_studies	0.2980	0.3061	0.2939	65.9797
sociology	0.2388	0.2637	0.2388	942.6319
us_foreign_policy	0.3100	0.3200	0.3000	283.4713
virology	0.2410	0.2530	0.2410	791.5547
world_religions	0.2222	0.2924	0.2222	320.0519

A little bit about the research | evals

These are recorded post-training results from a custom local evaluator. The comparison is against the earlier internal candidate final-assistant (v1). These checks assess instruction constraints, response content, termination, and repetition; they are not a standard academic benchmark or a general factual-accuracy score.

Raw means greedy decoding without repetition controls. nr4 adds only no_repeat_ngram_size=4. Both use one beam, with no sampling, reranking, token suppression, or response trimming.

Check	Earlier candidate (v1)	This release (v2)
Clean checks passed, raw	20/230	24/230
Clean checks passed, nr4	28/230	28/230
Responses ending with EOS, raw	75.2%	95.2%
Responses reaching the generation cap, raw	24.8%	4.8%
Responses with detected repeated 4-grams, raw	56.5%	26.5%
Responses with repeated sentences, raw	17.0%	0.9%
Responses reaching the generation cap, nr4	13.0%	1.3%
Responses with detected repeated 4-grams, nr4	14.3%	2.6%
Frozen proxy checks passed, raw	0/9	1/9
Frozen proxy checks passed, nr4	0/9	0/9

The 230-prompt set excluded the targeted training prompts. The nine-prompt proxy set was opened after the checkpoint was frozen, with no subsequent training. Exact formatting and lexical checks do not replace human assessment of answer quality.

On the three training-exposed feedback prompts, the release passed 3/3 checks under both decoding settings. The short story still reads as simplistic and repetitive despite passing those automated checks.

The clearest gain is response stability. Transfer to unseen factual explanations and paraphrased email constraints remains weak. MMLU and HellaSwag scores from earlier public checkpoints do not describe this release.

Limitations

• Factual errors, incoherence, repetition, and missed instructions remain common.
• Reasoning, arithmetic, coding, and strict output formats are unreliable.
• English is the evaluated language; multilingual quality is unverified.
• Sustained multi-turn conversation and long-context quality have not been established by these checks.
• Outputs can reflect biases or inappropriate content from training data; safety behavior has not been systematically evaluated.

This checkpoint is intended for research and education. Review generated content before using it in an application.

Artifact integrity

SHA-256 checksums for the released artifacts:

model.safetensors   2485b6b7e187a508e617f0c17795199b2e4f3dce96399f0d8254fb2688e420d0
modeling_gpt3dev.py  826a05d28508146f037fbabd19240f2bf468081d6311ad58821412a0643ca2e5

License and acknowledgements

MIT; see LICENSE. Thanks to OpenAI, Hugging Face, and PyTorch for the research and tools behind this project.

References: GPT-3 paper, Transformers, PyTorch.