rootfs/modernbert-dissat-binary-lora

ModernBERT Dissatisfaction Binary Classifier (LoRA)

A lightweight LoRA adapter for detecting user dissatisfaction in conversational AI systems. This model classifies user follow-up messages as either SAT (satisfied) or DISSAT (dissatisfied).

Model Description

This is a LoRA (Low-Rank Adaptation) adapter trained on top of answerdotai/ModernBERT-base for binary classification of user satisfaction in multi-turn conversations.

• Base Model: answerdotai/ModernBERT-base
• Task: Binary Classification (SAT vs DISSAT)
• Adapter Size: 102 KB (vs 571 MB for full fine-tuning)
• Trainable Parameters: 26,114 (0.017% of base model)

Intended Use

Detect when users are dissatisfied with AI responses in:

• Chatbots and virtual assistants
• Search clarification systems
• Task-oriented dialogue systems
• Semantic routing pipelines

Performance

Metric	Value
Accuracy	93.97%
F1 Macro	93.74%

Note: A full fine-tuned version achieves 99.84% accuracy. This LoRA adapter trades ~6% accuracy for 5,600x smaller size.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForSequenceClassification.from_pretrained(
    "answerdotai/ModernBERT-base",
    num_labels=2
)
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "rootfs/modernbert-dissat-binary-lora")
model.eval()

# Classify
text = "[USER ORIGINAL QUERY] Book a table for 2\n[SYSTEM ANSWER] I found a nice Italian restaurant\n[USER FOLLOW-UP] No, I wanted Chinese food"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = outputs.logits.argmax(dim=-1).item()

labels = {0: "SAT", 1: "DISSAT"}
print(f"Prediction: {labels[prediction]}")  # Output: DISSAT

Training Details

Training Data

Combined dataset from multiple sources:

• SGD (Schema-Guided Dialogue): 117,946 examples with dialogue act labels
• INSCIT: 18,989 examples with clarification signals

Total: 136,935 training examples

LoRA Configuration

Parameter	Value
LoRA Rank (r)	16
LoRA Alpha	32
Dropout	0.1
Target Modules	query, key, value, dense
Training Epochs	3
Learning Rate	2e-5
Batch Size	32

Label Mapping

• SAT (0): User is satisfied, continuing normally
• DISSAT (1): User shows dissatisfaction signals (corrections, rejections, clarification requests)

Hierarchical Pipeline

This model is designed as Stage 1 of a 2-stage hierarchical pipeline:

1. Stage 1 (This Model): SAT vs DISSAT classification
2. Stage 2: If DISSAT, use modernbert-dissat-explainer-lora to explain why (NEED_CLARIFICATION, WRONG_ANSWER, WANT_DIFFERENT)

Limitations

• Trained primarily on English dialogue data
• Performance may vary on domains significantly different from training data
• LoRA version has ~6% accuracy gap compared to full fine-tuning

Citation

If you use this model, please cite:

@misc{modernbert-dissat-binary-lora,
  author = {rootfs},
  title = {ModernBERT Dissatisfaction Binary Classifier (LoRA)},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/rootfs/modernbert-dissat-binary-lora}
}

License

Apache 2.0