Thaki-BanCode-KoEn-v2 — code-block detector for LLM guardrails (mmBERT-small, 140M)

🇰🇷 한국어 카드: MODEL_CARD.ko.md

A Korean/English text classifier that decides whether a prompt or a model output contains a program code block. It is built as the model behind an LLM Guard style BanCode scanner. On public documentation that was never used in training, it keeps the detection rate of the original BanCode model while removing almost all of its false alarms on normal Korean and English prose.

Licenses differ by part (details in LICENSE.md): weights and config CC BY-SA 4.0, wrapper code MIT. This repository does not contain the tokenizer. Load it from jhu-clsp/mmBERT-small at the pinned revision below; it keeps its own terms (see NOTICE.md). Because the tokenizer is not here, the hosted inference widget does not work; use the code below.

0. How to use

Pinned tokenizer: jhu-clsp/mmBERT-small revision abc32620dd4f6ab06f5fbe905dc25f310618e09f. Tested with Python 3.11, torch 2.9.0 (CPU), transformers 5.14.1, tokenizers 0.22.2. transformers 4.x cannot load this ModernBERT config and tokenizer combination; other versions were not tested.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-small",
                                    revision="abc32620dd4f6ab06f5fbe905dc25f310618e09f")
model = AutoModelForSequenceClassification.from_pretrained("ThakiCloud/Thaki-BanCode-KoEn-v2").eval()
CODE = model.config.label2id["code"]          # id2label = {0: "code", 1: "no_code"}

def p_code(texts):
    enc = tok(texts, truncation=True, max_length=256, padding=True, return_tensors="pt")
    with torch.inference_mode():
        return torch.softmax(model(**enc).logits.float(), -1)[:, CODE].tolist()

p = p_code(["Run this:\n```bash\npip install requests\n```"])[0]
is_code = p > 0.5                              # frozen threshold, no rounding

Or use the wrapper in this repository, which reads the pinned tokenizer revision from the profile:

from huggingface_hub import snapshot_download
import sys
d = snapshot_download("ThakiCloud/Thaki-BanCode-KoEn-v2")
sys.path.insert(0, d)
from bancode import BanCodeV2
scanner = BanCodeV2(d)
print(scanner.scan("Run this:\n```bash\npip install requests\n```"))   # (prompt, is_valid=False, risk_score)

Operating profile (bancode_v2_profile.json, frozen at adoption, do not retune): score = P(code), flag when score > 0.5, max_length 256 (later tokens are dropped), raw text, no preprocessing. Scanner contract (sanitized_prompt, is_valid, risk_score) with LLM Guard's calculate_risk_score(score, 0.5) rule.

⚠️ Not supported: using this model as a drop-in for the stock llm_guard.input_scanners.BanCode class of llm-guard 0.3.16. That class strips markdown code fences and digits before classifying (this model was trained on raw text, where the fence is the signal), compares the predicted label with the string "CODE" (this model's label is lowercase code), and pins transformers 4.51.3, which cannot load this model. Use the wrapper and profile above.

1. Independent evaluation (sources never used in training)

One preregistered scoring run on a frozen cell of 13,814 items built from public documentation and sentence corpora that are disjoint from all training, model-selection and test sources (source-level exclusion, plus 1 exact and 46 near-duplicate removals). Labels follow a deterministic markup rule: a fenced block with a language tag = code; paragraphs outside blocks, including prose with inline code spans, = not code. All three models were scored by us with the same code on the same cell. Sources: Kubernetes, React, FastAPI, es-toolkit and Docker documentation, a Korean chatbot corpus, and UD Korean-Kaist / English-EWT sentences.

Language (code / normal)	Model	False positives on normal text (95% CP upper)	Code detection rate	AUROC
Korean (2,482 / 4,200)	this model (threshold 0.5)	0.00% (0 / 4,200; ≤ 0.07%)	99.3%	0.9999
	previous in-house model, synthetic data only (0.5)	3.45%	93.6%	0.9867
	original LLM Guard BanCode vishnun/codenlbert-sm (0.97)	91.3%	98.2%	0.9025
English (3,010 / 3,600)	this model (threshold 0.5)	0.72% (26 / 3,600; ≤ 1.00%)	99.8%	0.9999
	previous in-house model, synthetic data only (0.5)	4.94%	93.6%	0.9825
	original LLM Guard BanCode vishnun/codenlbert-sm (0.97)	60.7%	95.1%	0.8445

Weakest slices of this model (n ≥ 50):

Slice	n	This model	Previous in-house	Original
Korean markup blocks (HTML/XML etc.), detection	66	93.9%	93.9%	98.5%
Korean code-only snippets without surrounding prose, detection	340	95.9%	98.2%	87.9%
English general sentences (UD English-EWT), false positives	1,200	1.83%	1.42%	16.6%
Korean prose with inline code, false positives	1,200	0.00%	11.1%	99.9%
English prose with inline code, false positives	1,200	0.08%	11.6%	86.3%

2. Development-stage numbers (same sources as training — not independent)

These come from a held-out split of the training sources (Wikipedia, Wikibooks, Stack Exchange) that was sealed before training and used once for the preregistered adoption decision. They are in-distribution and are reported only to document that decision; section 1 is the evidence for generalisation.

Language	Model	False positives	Detection
Korean (678 code / 2,541 normal)	this model	0.24%	99.9%
	previous in-house model	4.37%	97.2%
English (4,120 code / 8,723 normal)	this model	0.39%	99.0%
	previous in-house model	5.49%	95.2%

3. Intended use

• Detecting program code (source code, shell commands, config files, markup) in Korean or English chat prompts and model outputs, as one signal in a guardrail pipeline (for example a "no code in this channel" policy).
• Recommended start: log or warn mode. Treat a block decision as a policy choice made on your own traffic.

Out of scope: judging whether code is malicious, detecting secrets, languages other than Korean and English, documents longer than 256 tokens (only the start is read).

4. Limitations and release context (measured or known, not hidden)

• Real user inputs were not evaluated. The independent cell is public technical documentation plus general sentence corpora; real chat prompts may differ. Q&A-style sources are absent from it, because Stack Exchange is a training source.
• Log blocks were too rare to evaluate (Korean 3, English 4 items; report only). Blocks without a language tag (Korean 251, English 264) and Korean config blocks with Hangul inside the code (39) were report only.
• Labels are deterministic from markup; code written in prose without a fence counts as "not code" by construction.
• The mmBERT backbone was pretrained on web data that may include the same public documentation (not measurable).
• One normal Korean public-prose sentence out of 91 (KLUE-MRC) was flagged where the previous model flagged none — a known 1-item difference recorded at adoption.
• Latency and CPU cost were not measured for this release; no speed claim is made.
• This model is one component of an internal LLM-guard research program. It is released on its own; other components of the same program (for example the URL phishing classifier) did not pass independent evaluation and are not released. The previous in-house BanCode model (synthetic data only) stays unreleased.
• No ONNX export is included in this repository.

5. Model details

• Backbone jhu-clsp/mmBERT-small (ModernBERT, 22 layers, hidden 384, vocab 256k), sequence classification head, labels {0: code, 1: no_code}.
• Fine-tuning: 3 epochs, batch 32, max length 256, lr 5e-5, fp32, sqrt class weights, seed 42; epoch picked by development macro-F1. Seeds 43 and 44 were trained only to check stability.
• model.safetensors sha256 44c72235ea8c1cfb679b5dc5e59acdb7ddae4fe203182ce3b7e45114acbb0348, byte-identical to the adopted checkpoint. Every file of this repository is pinned in SHA256SUMS.txt.

6. Tokenizer equivalence check

The adopted checkpoint was saved with a re-serialized copy of the mmBERT tokenizer. Before publishing, we checked in one job that loading the tokenizer from the pinned upstream revision gives the same model behaviour on 66 hand-written Korean/English probe texts (not taken from any training or evaluation set; includes texts longer than 256 tokens, leading spaces, tabs, CRLF, emoji, NFD Hangul): identical token ids, identical logits and identical decisions compared with the original checkpoint package (tolerance: max |Δlogit| ≤ 1e-6; publishing was gated on this check). The check is repeated after publishing on files downloaded anonymously from this repository.

7. Training data

The data is not redistributed; only attribution is given here and in NOTICE.md.

Source	License	Rows used
Korean and English Wikipedia and Wikibooks (MediaWiki API, revision ids recorded)	CC BY-SA 4.0	together with Stack Exchange: 47,448
Stack Exchange askubuntu / devops / dba / codereview via HuggingFaceH4/stack-exchange-preferences (rev c7bda740)	CC BY-SA 4.0 (posts: CC BY-SA 2.5–4.0 by date)	(included above)
Synthetic Korean/English code and prose, generated by a self-hosted quantized copy of Qwen/Qwen3.8-27B (Apache-2.0); prompts were topic and format instructions only, no seed data, no external API	in-house generation	1,802

Epoch selection also used a development split with 237 public items from KLUE-MRC (CC BY-SA 4.0), SQuAD (CC BY-SA 4.0) and HumanEval (MIT); these were not used for gradient updates. No AI-Hub data was used.

8. Licenses

Part	Terms
Fine-tuned weights and config (model.safetensors, config.json)	CC BY-SA 4.0
Wrapper code and profile (bancode.py, bancode_v2_profile.json)	MIT
Base model jhu-clsp/mmBERT-small	MIT; the authors' permission notice is kept in NOTICE.md
Training text (Wikipedia, Wikibooks, Stack Exchange)	CC BY-SA, credited in NOTICE.md (data not redistributed)
Tokenizer	not included; loaded from jhu-clsp/mmBERT-small @ abc32620dd4f6ab06f5fbe905dc25f310618e09f under that repository's own terms

The CC BY-SA 4.0 label of the weights does not cover the tokenizer. We do not claim that training-data licenses pass to the weights automatically; CC BY-SA 4.0 was chosen for the weights as the conservative option.

9. Citation

Base model: mmBERT (Marone et al., 2025, arXiv:2509.06888). Original scanner: LLM Guard by Protect AI.