Vibrato-ai/vibrato-v1

Model Card: Vibrato

Superseded by vibrato-v2. A post-release audit found that v1's evaluation was contaminated: the model trained on 18 of VocalSet's 20 singers with no singer-disjoint split, so the accuracy figures below are effectively training-set numbers. Against measured acoustic labels (Praat CPPS, jitter/shimmer, f0 deviation), v1's five quality outputs all have negative R² — worse than predicting a constant. v2 fixes the corpus coverage, roster, labels, and splits, and reports genuinely held-out numbers. Use v2; v1 remains available for reproducibility.

Model Details

• Name: Vibrato
• Version: 1.0 (formerly ToneyMultiTask v2.0)
• Type: Multi-task 1D CNN for vocal analysis
• Framework: PyTorch (training), CoreML (deployment)
• License: Apache-2.0
• Repository: vibrato-ai/vibrato-v1

Project

Vibrato is an open vocal-AI model for the singing community, maintained by Anycompany LLC. It powers Toney, a free iOS vocal-training app, and is published for anyone to use, study, and improve. The training pipeline (model.py, CoreML export) is included in this repository.

Roadmap: v1 (this release) — VocalSet-trained baseline. v2 — expanded training data (augmentation + additional public datasets). v3 — federated learning: on-device fine-tuning with privacy-preserving aggregated updates (voices never leave the phone).

Intended Use

Real-time vocal analysis for singing practice and coaching. Designed to run on-device on iPhones via CoreML Neural Engine.

Primary use: Analyze vocal recordings to classify voice type, singing technique, vowel production, and vocal quality metrics.

Out of scope: Speech recognition, language identification, speaker identification, medical diagnosis of vocal disorders.

Model Architecture

Input: [1, 4000] raw audio (0.25s @ 16kHz)
  -> SharedEncoder (4x ConvBlock with BatchNorm, stride-2 downsampling, AdaptiveAvgPool)
  -> 256-dim feature vector
  -> Voice Head (FC 256->128->6, logits — apply softmax at inference)
  -> Technique Head (FC 256->128->7, logits — apply softmax at inference)
  -> Vowel Head (FC 256->128->6, logits — apply softmax at inference)
  -> Quality Head (FC 256->128->5, sigmoid applied in-model)

Training Data

Dataset	License	Size	Use
VocalSet	CC BY 4.0	10.1 hrs, 20 singers (18 used in v1 — see below)	All four heads (technique, vowel, voice type; quality labels heuristically derived from the audio)

Coverage in v1: Training used 1,562 of VocalSet's 3,613 files (43%). The remainder were dropped because their technique folders were not in the technique map (slow_piano, slow_forte, fast_forte, fast_piano, pp, forte, messa, and the misspelled "vibrado" folder). Additionally, a gender-table erratum in the singer roster (the code listed 11 female / 9 male singers, while VocalSet actually contains 9 female / 11 male) caused singers male10 and male11 (364 files) to be silently skipped — so 18 of the 20 singers contributed training data. Both issues are fixed in the v2 pipeline plan; this card describes the shipped v1 model as trained.

Planned for v2 (not used in v1): Annotated-VocalSet (F0 annotations for pitch-stability labels), SVQTD (human vocal-quality labels).

Attribution: This model was trained on VocalSet, used under CC BY 4.0: Julia Wilkins, Prem Seetharaman, Alison Wiss, Bryan Pardo, "VocalSet: A Singing Voice Dataset", Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), 2018. Zenodo DOI: 10.5281/zenodo.1193957. Changes made to the original data for training: audio resampled to 16 kHz, segmented into 0.25 s windows, technique folder labels remapped to the model's 7-class scheme, and heuristic quality labels added.

Output Labels

Voice Types (6)

soprano, mezzo-soprano, alto, tenor, baritone, bass

Techniques (7)

belt, falsetto, vibrato, straight, breathy, nasal, mixed

Vowels (6)

a, e, i, o, u, schwa

Quality Scores (5, continuous 0-1)

brightness, breathiness, strain, power, stability

Per-class caveats (v1)

• falsetto: 0 training samples reached this class in v1 (the folder-mapping bug above dropped them), so its logit is effectively untrained — do not rely on falsetto predictions.
• nasal: trained on VocalSet's vocal_fry recordings (a labeling shortcut), so it actually detects vocal fry, not nasality.
• mixed: trained on lip_trill recordings, so it actually detects lip trills, not mixed voice.
• Voice types: VocalSet publishes no fach/voice-type metadata; the soprano-through-bass labels are heuristic assignments made in this pipeline, not ground truth.
• strain: the heuristic pseudo-label saturates at 1.0 on ~83% of training clips (the zero-crossing "jitter" estimate is not true pitch-period jitter), so the strain output has limited dynamic range.

Evaluation

All numbers are on the held-out validation split (random 20% of windows) at epoch 49. "Production condition" applies the peak normalization the iOS app performs before inference — a preprocessing mismatch with training (which used raw amplitudes), which measurably degrades accuracy; v2 will train and evaluate under matched normalization.

Metric	Val (reported)	Val (production condition)	Majority-class baseline
Voice type accuracy	47.9%	—	~22.6%
Technique accuracy	48.2%	30.4%	25.6%
Vowel accuracy	57.8%	48.6%	20.8%
Quality MSE	0.0292	—	—

On the two singers never seen in training (male10/male11, excluded by the roster bug): technique 48.9% / vowel 64.4% raw, dropping to 31.0% / 53.4% under production normalization. The quality head reaches R-squared of approximately 0.32 against its own deterministic heuristic targets.

Note that the validation split is random over 0.25 s windows, so windows from the same recording (and singer) appear in both train and val — these numbers are optimistic relative to true singer-held-out generalization.

Limitations

• Trained on ~10 hrs from professional singers (VocalSet; 18 of its 20 singers due to the roster bug above) — limited demographic and stylistic coverage; expect degraded accuracy on voice types, languages, and styles outside that distribution.
• 0.25 s context window — no long-range phrase analysis.
• Quality scores are heuristic-derived labels, not clinician annotations.
• Not for speaker ID, speech recognition, or medical use.

Ethical Considerations

• The model is not designed or evaluated for speaker identification
• No personally identifiable information is stored or inferred
• All inference runs on-device; no audio data is transmitted
• The model is NOT a medical device and should not be used for vocal health diagnosis