cafoscari-nlp/LIBE

LIBE: Language Identification through Representation Learning

LIBE identifies languages by embedding input texts and natural-language descriptions of languages into a shared semantic space and retrieving the most similar descriptions.

This checkpoint is based on LaBSE and was trained on 10 million sentences from GlotLID-C. It implements the bi-encoder component of LIBE and supports language identification over a candidate inventory covering more than 1,800 languages.

• Interactive app: Try LIBE on Hugging Face Spaces
• Code and data: Ca’ Foscari NLP / LIBE
• Paper: Rocco Tripodi. Representation Learning Enables Language Identification in Zero- and Few-Shot Settings. Transactions of the Association for Computational Linguistics, 2026, to appear. See Citation.
• Language inventory: language_sampling.tsv

Model description

Language identification is a key component of multilingual NLP pipelines and multilingual corpus construction. Incorrect predictions can create spurious associations between text and languages, reducing the quality of downstream models. These errors are particularly consequential for low-resource languages and for fine-grained distinctions between closely related languages and dialects.

LIBE formulates language identification as similarity search. A bi-encoder maps both input sentences and natural-language descriptions of languages into the same vector space. At inference time, the model ranks candidate languages by the cosine similarity between the input embedding and each description embedding.

The approach supports data-efficient learning and allows additional candidate languages to be introduced through their descriptions. Adding a description does not require changing a fixed classification head, although reliable identification of a newly added language must still be evaluated.

The associated paper also introduces an optional cross-encoder re-ranking stage to improve discrimination in ambiguous or low-resource cases. This checkpoint and the example below use the bi-encoder; they do not include that optional re-ranking stage.

Abstract

Language Identification (LID) is a key component of multilingual NLP pipelines and multilingual corpus construction. LID errors can create spurious text–language associations, harming downstream model quality, especially for low-resource languages and fine-grained taxonomies involving closely related languages and dialects.

We propose LIBE, a representation-based approach that embeds input texts and natural-language descriptions into a shared semantic space with a bi-encoder, enabling LID via similarity search. This formulation supports data-efficient learning and the addition of new languages through their descriptions. To improve discrimination in ambiguous or low-resource cases, we also introduce an optional cross-encoder re-ranking stage.

Across multiple benchmarks, LIBE achieves competitive performance with strong LID systems when initialized from pretrained multilingual encoders, while requiring substantially less task-specific LID supervision in several adaptation settings. The hybrid bi-encoder–cross-encoder formulation further improves robustness in challenging open-set scenarios, achieving up to 90% F1 with as few as five labeled examples.

Code and data are available in the LIBE project repository.

Model and training details

Property	Description
Model type	Sentence-transformer bi-encoder
Initialization	LaBSE
Task-specific training data	10 million sentences from GlotLID-C
Inputs	Text to identify and natural-language descriptions of candidate languages
Model output	Dense embeddings
LID decision	Candidate language with the highest cosine similarity
Language coverage	More than 1,800 candidate languages; see the inventory
Software	Sentence Transformers

For background on the encoder, see the LaBSE model page. Training hyperparameters, sampling details, and benchmark protocols should be consulted in the associated paper and project materials. This card does not specify unreported checkpoint-specific hyperparameters.

Language coverage and training-data sampling

The per-language sampling table documents how sentences were sampled from GlotLID-C to construct LIBE’s 10-million-sentence training set. It reports the available and sampled sentence counts for each language, together with their proportions before and after sampling.

Column	Description
lang	Language identifier in the source corpus.
n_sents	Number of sentences available for the language before sampling.
pi	Proportion of the source sentence pool belonging to the language: n_sents / sum(n_sents).
n_sampled	Number of sentences selected for the language in the training sample.
qi	Proportion of the sampled training set belonging to the language: n_sampled / sum(n_sampled).

The table describes training-data representation. Training coverage, prediction-candidate coverage, and evaluation coverage are distinct: a language’s presence in the table does not imply equal identification accuracy or inclusion in every evaluation setting. The languages available for prediction depend on the description inventory used at inference time.

LIBE can accommodate additional candidate languages by encoding their natural-language descriptions with the same checkpoint and preprocessing. Identification performance for new candidates, and their effect on closely related languages, should be evaluated separately.

Using the model

The easiest way to try LIBE is through the interactive app. Enter a sentence to view the predicted language and the highest-scoring candidates.

For local use, install Sentence Transformers:

pip install sentence-transformers

The following example loads a local copy of the model repository. Run it from the repository root, or replace MODEL_PATH with the local checkpoint directory or the published Hugging Face model ID.

from sentence_transformers import SentenceTransformer, util

MODEL_PATH = "."
model = SentenceTransformer(MODEL_PATH, device="cpu")

# These two descriptions illustrate the API only.
# For full LIBE inference, use the project's complete description inventory.
descriptions = {
    "eng": "English is a West Germanic language.",
    "ita": "Italian is a Romance language.",
}
labels = sorted(descriptions)

# Compute once and reuse while the model and descriptions remain unchanged.
description_embeddings = model.encode(
    [descriptions[label] for label in labels],
    convert_to_tensor=True,
    normalize_embeddings=True,
)

text = "Questa è una frase in italiano."
query_embedding = model.encode(
    [text],
    convert_to_tensor=True,
    normalize_embeddings=True,
)

hits = util.semantic_search(
    query_embedding,
    description_embeddings,
    top_k=min(5, len(labels)),
)[0]

for hit in hits:
    label = labels[hit["corpus_id"]]
    print(f"{label}: {hit['score']:.4f}")

This two-language example demonstrates the inference procedure; it does not reproduce the full demo or the paper's evaluation. Reproduction requires the same checkpoint, candidate descriptions, preprocessing, and evaluation settings.

Evaluation

The paper reports competitive performance against strong language-identification systems across multiple benchmarks, with substantially less task-specific supervision in several adaptation settings.

The result of up to 90% F1 with as few as five labeled examples refers to the paper's hybrid bi-encoder–cross-encoder formulation in challenging open-set scenarios. It is not a standalone benchmark score for this checkpoint or a claim of uniform performance across the entire candidate inventory.

Consult the paper for benchmark definitions, language subsets, evaluation protocols, and comparisons. No checkpoint-specific benchmark table is supplied in this card.

Intended uses

• Research on multilingual language identification and description-based classification.
• Candidate language ranking for multilingual corpus construction and analysis.
• Experiments on low-resource languages, related languages, and dialect distinctions.
• Investigation of zero-shot and few-shot adaptation through language descriptions.

For corpus filtering, validate the model on representative examples from the target languages and domain before using predictions to include or discard data.

Limitations

• Similarity is not probability. Scores are cosine similarities, not calibrated confidence estimates.
• Candidate-set dependence. The basic search procedure returns the closest available language even when the correct language is absent. It does not automatically reject unknown languages.
• Uneven performance. Accuracy can vary with language, training representation, domain, and the quality of the descriptions.
• Ambiguous inputs. Very short texts, shared vocabulary, names, closely related languages, and dialects can be difficult to distinguish.
• Mixed-language text. A single sentence-level prediction does not identify every language in a code-switched passage.
• Input length. Texts exceeding the encoder's configured maximum sequence length may be truncated. Longer documents require an appropriate segmentation strategy.
• Description sensitivity. Changing descriptions or their preprocessing can change predictions. Regenerate cached description embeddings whenever the model or descriptions change.
• Downstream impact. Incorrect labels may exclude low-resource-language material or introduce mislabeled data into downstream training corpora. Per-language evaluation and review are advisable.

License

This LIBE model checkpoint is licensed under the Creative Commons Attribution 4.0 International license (CC BY 4.0).

The application code and accompanying data may be subject to separate licenses; consult their respective repositories and license notices.

Citation

If you use LIBE in your work, please cite:

Rocco Tripodi. 2026. Representation Learning Enables Language Identification in Zero- and Few-Shot Settings. Transactions of the Association for Computational Linguistics. To appear.

@article{tripodi-2026-libe,
  author  = {Tripodi, Rocco},
  title   = {Representation Learning Enables Language Identification in Zero- and Few-Shot Settings},
  journal = {Transactions of the Association for Computational Linguistics},
  year    = {2026},
  note    = {To appear}
}

Project links

• LIBE demonstration
• Source code and data
• Ca’ Foscari NLP