| | --- |
| | language: es |
| | library_name: transformers |
| | license: apache-2.0 |
| | tags: |
| | - roberta |
| | - spanish |
| | - scientific |
| | - fill-mask |
| | --- |
| | |
| | # Sci-BETO-base |
| |
|
| | **Sci-BETO** is a domain-specific RoBERTa encoder pretrained entirely on **Spanish scientific texts**. |
| |
|
| | --- |
| |
|
| | ## Model Description |
| |
|
| | Sci-BETO-base is a transformer-based encoder following the **RoBERTa** architecture (125M parameters). |
| | It was pretrained from scratch using byte-level BPE tokenization on a large corpus of **Spanish open-access scientific publications**, including theses, dissertations, and peer-reviewed papers from Colombian universities and international repositories. |
| |
|
| | The model was designed to capture **scientific discourse**, terminology, and abstract reasoning patterns typical of research documents in economics, engineering, medicine, and the social sciences. |
| |
|
| | | Property | Value | |
| | |-----------|--------| |
| | | Architecture | RoBERTa-base | |
| | | Parameters | ~125M | |
| | | Vocabulary size | 50,262 | |
| | | Tokenizer | Byte-Level BPE (trained from scratch) | |
| | | Pretraining objective | Masked Language Modeling (MLM) | |
| | | Pretraining steps | 85K | |
| | | Max sequence length | 512 tokens | |
| | | Framework | Transformers | |
| |
|
| | --- |
| |
|
| | ## Pretraining Data |
| |
|
| | The pretraining corpus includes over **11 billion tokens** from Spanish academic and scientific sources: |
| |
|
| | - Open-access repositories of Colombian universities (Universidad de los Andes, Universidad Nacional, Universidad Javeriana, Universidad del Rosario). |
| | - CORE API and institutional repositories (theses, dissertations, working papers). |
| | - Tax Statutes in Colombia |
| |
|
| | The final dataset covers multiple disciplines (economics, medicine, engineering, humanities), ensuring representation across scientific domains. |
| |
|
| | | **Source** | **# Documents** | **# Words (deduplicated)** | **Percentage (%)** | |
| | |--------------------------------|----------------:|----------------------------:|-------------------:| |
| | | Universidad de los Andes | 33,858 | 365,752,780 | 3.23 | |
| | | Universidad Nacional | 44,686 | 537,022,975 | 4.75 | |
| | | CORE API | 2,181,689 | 9,624,189,002 | 85.10 | |
| | | Universidad del Rosario | 22,404 | 183,356,109 | 1.62 | |
| | | Universidad Javeriana | 25,624 | 323,918,445 | 2.86 | |
| | | Tax Statutes in Colombia | 392 | 13,924,060 | 0.12 | |
| | | Extra | 2 | 261,131,453 | 2.31 | |
| | | **Total** | **2,308,655** | **11,309,295,824** | **100.00** | |
| |
|
| | --- |
| |
|
| | ## Benchmarks |
| |
|
| | Sci-BETO was fine-tuned and benchmarked across multiple downstream tasks, both general-domain and scientific: |
| |
|
| | | **Dataset** | **Metric** | **Sci-BETO Large** | **Sci-BETO Base** | **BETO** | **BERTIN** | |
| | |---------------------|----------------|-------------------:|------------------:|----------:|------------:| |
| | | **WikiCAT** | F1 (macro) | **0.7738** | 0.7583 | 0.7624 | 0.7598 | |
| | | **PAWS-X (es)** | F1 (macro) | **0.9148** | 0.8794 | 0.8985 | 0.8961 | |
| | | **PharmaCoNER** | F1 (micro) | **0.8959** | 0.8733 | 0.8845 | 0.8802 | |
| | | **CANTEMIST** | F1 (micro) | 0.8809 | 0.8784 | 0.8954 | **0.8956** | |
| | | **NLI (ESNLI-R)** | F1 (micro) | — | — | — | — | |
| | | **BanRep (JEL)** | Exact Match | **0.6116** | 0.6043 | 0.5933 | 0.5807 | |
| | | **Rosario** | F1 (macro) | **0.9203** | 0.9194 | 0.9079 | 0.9121 | |
| | | **Econ-IE** | F1 (micro) | **0.5256** | 0.5158 | 0.5199 | 0.4992 | |
| |
|
| |
|
| | On average, **Sci-BETO** achieves comparable or superior results to general-domain Spanish models in specialized contexts (scientific, biomedical, economic), while maintaining strong performance in general text understanding. |
| |
|
| | --- |
| |
|
| | ## Intended Use |
| |
|
| | - Research and experimentation in **Spanish scientific NLP**. |
| | - Downstream fine-tuning for: |
| | - Text classification (scientific or academic domains), |
| | - Named Entity Recognition (NER), |
| | - Semantic similarity and paraphrase detection, |
| | - Knowledge extraction from academic documents. |
| |
|
| | --- |
| |
|
| | ## Limitations |
| |
|
| | - The model may underperform on highly informal or non-academic Spanish (e.g., social media). |
| | - It is not designed for generative tasks (e.g., text completion, chat). |
| | - Domain bias toward academic register and Latin American Spanish variants. |
| | - Pretraining corpus excludes English or bilingual data. |
| |
|
| | --- |
| |
|
| | ## Example Usage |
| |
|
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForMaskedLM |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("Flaglab/Sci-BETO-base") |
| | model = AutoModelForMaskedLM.from_pretrained("Flaglab/Sci-BETO-base") |
| | |
| | text = "El Banco de la República va a subir las [mask] de interes." |
| | inputs = tokenizer(text, return_tensors="pt") |
| | outputs = model(**inputs) |
| | logits = outputs.logits |
| | masked_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1] |
| | predicted_token = tokenizer.decode(logits[0, masked_index].argmax(dim=-1)) |
| | print("Predicted token:", predicted_token) |
