Final checks.

README.md

@@ -3,64 +3,24 @@ license: apache-2.0
 library_name: transformers
 pipeline_tag: text-generation
 language:
-- bg
 - ca
-- code
-- cs
-- cy
-- da
-- de
-- el
 - en
 - es
-- et
 - eu
-- fi
-- fr
-- ga
 - gl
-- hr
-- hu
-- it
-- lt
-- lv
-- mt
-- nl
-- nn
-- \no
-- oc
-- pl
-- pt
-- ro
-- ru
-- sh
-- sk
-- sl
-- sr
-- sv
-- uk
 datasets:
-- oscar-corpus/colossal-oscar-1.0
+- CohereLabs/aya_dataset
+- projecte-aina/CoQCat
+- databricks/databricks-dolly-15k
+- projecte-aina/dolly3k_ca
+- projecte-aina/MentorES
+- projecte-aina/MentorCA
+- HuggingFaceH4/no_robots
+- projecte-aina/RAG_Multilingual
+- Unbabel/TowerBlocks-v0.2
+- OpenAssistant/oasst2
+- open-r1/OpenR1-Math-220k
 - HuggingFaceFW/fineweb-edu
-- joelniklaus/eurlex_resources
-- joelniklaus/legal-mc4
-- projecte-aina/CATalog
-- UFRGS/brwac
-- community-datasets/hrwac
-- danish-foundation-models/danish-gigaword
-- HiTZ/euscrawl
-- PleIAs/French-PD-Newspapers
-- PleIAs/French-PD-Books
-- AI-team-UoA/greek_legal_code
-- HiTZ/latxa-corpus-v1.1
-- allenai/peS2o
-- pile-of-law/pile-of-law
-- PORTULAN/parlamento-pt
-- hoskinson-center/proof-pile
-- togethercomputer/RedPajama-Data-1T
-- bigcode/starcoderdata
-- bjoernp/tagesschau-2018-2023
-- EleutherAI/the_pile_deduplicated
 base_model:
 - BSC-LT/ALIA-40b
 ---
@@ -72,11 +32,11 @@ base_model:
 
 # ALIA-40b-instruct Model Card
 
-The ALIA-40b-instruct model is an instructed variant of a context-extended [base ALIA-40b model](https://huggingface.co/BSC-LT/ALIA-40b), which was pre-trained from scratch on 9.83 trillion tokens of carefully curated data spanning 35 European languages (including code). This instructed version is optimized to follow user prompts and engage in dialogue. It supports a broad range of languages (e.g. Spanish, Catalan, Basque, English, etc.) and is capable of text generation, translation, summarization, and question-answering in these languages. This version has also been gone through a preliminary alignment phase for helpfulness and safety with synthetically generated preference pairs. 
+The ALIA-40b-instruct model is an instructed variant of a context-extended [base ALIA-40b model](https://huggingface.co/BSC-LT/ALIA-40b), which was pre-trained from scratch on 9.83 trillion tokens of carefully curated data spanning 35 European languages (including code). This instructed version is optimized to follow user prompts and engage in dialogue. It supports a broad range of languages (e.g. Spanish, Catalan, Basque, English, etc.) and is capable of text generation, translation, summarization, and question-answering in these languages. This version has also gone through a preliminary alignment phase for helpfulness and safety with synthetically generated preference pairs. 
 
-In keeping with our commitment to open-source development, all tools and sources used to process and create the training data are open-licensed. For clarity, our definition of open-licensed excludes any source, tool, model, or dataset whose terms of use  impose restrictive conditions that impede standard open reuse.
+In keeping with our commitment to open-source development, all tools and sources used to process and create the training data are open-licensed. For clarity, our definition of open-licensed excludes any source, tool, model, or dataset whose terms of use impose restrictive conditions that impede standard open reuse.
 
-This model is released under the apermissive [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). Along with the open weights, all training scripts and configuration files are made publicly available in [this GitHub repository](https://github.com/langtech-bsc/alia).
+This model is released under the permissive [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). Along with the open weights, all training scripts and configuration files are made publicly available in [this GitHub repository](https://github.com/langtech-bsc/alia).
 
 To visit the model cards of other model versions, please refer to the [Model Index](https://www.notion.so/Alia-2025-09-29-Model-Card-27db93cf5c1b808aa1f1fc8229255f24?pvs=21).
 
@@ -87,15 +47,15 @@ To visit the model cards of other model versions, please refer to the [Model Ind
 
 ### Description
 
-The ALIA-40b is a transformer-based, decoder-only language model tha was pre-trained from scratch on 9.37 trillion tokens of meticulously curated data. It subsequently underwent continued pretraining on additional 424 billion high-quality tokens, and was further extended with a supplementary 39 biilion tokens drawn from a similarly diverse mixture, totalling 9.83 trillion tokens.
+The ALIA-40b is a transformer-based, decoder-only language model that was pre-trained from scratch on 9.37 trillion tokens of meticulously curated data. It subsequently underwent continued pretraining on additional 424 billion high-quality tokens, and was further extended with a supplementary 39 billion tokens drawn from a similarly diverse mixture, totalling 9.83 trillion tokens.
 
-ALIA-40b-Instruct is an instructed variant of this latest ALIA-40b version. Its post-training process comprises three consecutive stages, each targetting a specific capility: (1) long-context adaptation to extend the model’s context window, (2) supervised fine-tuning to improve instruction following capabilities, and (3), a preliminary alignment stage to better match human preferences and safety. 
+ALIA-40b-Instruct is an instructed variant of this latest ALIA-40b version. Its post-training process comprises three consecutive stages, each targeting a specific capability: (1) long-context adaptation to extend the model’s context window, (2) supervised fine-tuning to improve instruction following capabilities, and (3) a preliminary alignment stage to better match human preferences and safety. 
 
-After the long-context adaptation, the model enters the supervised fine-tuning (SFT) stage. This stage is implemented in two phases for efficiency reasons: a short-context SFT con 469k conversation examples to strengthen instruction following, followed by a long-context SFT con 14k long-context instances. We separate these phases because full-context fine-tuning is computationally expensive.
+After the long-context adaptation, the model enters the supervised fine-tuning (SFT) stage. This stage is implemented in two phases for efficiency reasons: a short-context SFT with 469k conversation examples to strengthen instruction following, followed by a long-context SFT with 9k long-context instances. We separate these phases because full-context fine-tuning is computationally expensive.
 
-In the third stage, the model is aligned with human preferences through Direct Policy Optimization (DPO) using a mixture of 401k preference pairs. Of this mixture, approximately 82% of the pairs target general model helpfulness, while 18% focus on response safety. This aligment stage is preliminary, and further work is ongoin to strengthen safety and reliability.
+In the third stage, the model is aligned with human preferences through Direct Policy Optimization (DPO) using a mixture of 403k preference pairs. Of this mixture, approximately 82% of the pairs target general model helpfulness, while 18% focus on response safety. This alignment stage is preliminary, and further work is ongoing to strengthen safety and reliability.
 
-Although the base model is highly multilingual, the post-training process concentrated primarily on Spanish, Catalan, Basque, Galician, and English. We also incoporated data from other related languages where inclusion empirically improved the performance on the target languages. However, performance in those additional languages is not guaranteed duet to the limited amount of available data and the scarcity of evaluation resources.
+Although the base model is highly multilingual, the post-training process concentrated primarily on Spanish, Catalan, Basque, Galician, and English. We also incorporated data from other related languages where inclusion empirically improved the performance on the target languages. However, performance in those additional languages is not guaranteed due to the limited amount of available data and the scarcity of evaluation resources.
 
 ### Hyperparameters
 
@@ -132,7 +92,7 @@ Here we list the specific hyperparameters used during the different training sta
 | Epochs | 1 |
 | LR Scheduler | Cosine |
 | Warmup Ratio | 0.03 |
-|  Number of Samples | 12,928 |
+|  Number of Samples | 9,380 |
 
 #### Alignment
 
@@ -140,7 +100,7 @@ Here we list the specific hyperparameters used during the different training sta
 | --- | --- |
 | Learning rate | 2e-6 |
 | Batch size | 1024 |
-| Epoch | 2 |
+| Epochs | 2 |
 | LR Scheduler | Linear |
 | Number of samples | 402,917 |
 
@@ -181,7 +141,7 @@ The model is not intended for malicious activities, such as harming others or vi
 
 ### Training Framework
 
-The post-training process was conducted using three complementary frameworkds, each selecto to best support its corresponding stage:
+The post-training process was conducted using three complementary frameworks, each selected to best support its corresponding stage:
 
 - Supervised Fine-Tuning (SFT): Conducted with an internal fork of the FastChat codebase, adapted to our infrastructure and optimized for stability and efficiency in our use case.
 - Long-Context SFT: Performed using NeMo-Aligner, chosen to ensure compatibility with extended-context training while maintaining consistency with the FastChat-based SFT.
@@ -198,7 +158,7 @@ The accelerated partition is composed of 1,120 nodes with the following specific
 - 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
 - 4x NDR200 (BW per node 800Gb/s)
 - 512 GB of Main memory (DDR5)
-- 460GB on NVMe storage
+- 460GB of NVMe storage
 
 The table below specifies the number of nodes and GPUs employed for each post-training stage:
 
@@ -247,8 +207,8 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
 
 ---
-Using this template, each turn in the conversation is preceded by a `<|im_start|>` delimiter indicating the beginning of a message, followwed by and the role of the entity
-(either `user`, for content supplied by the user, or `assistant` for model’s responses), and finished with the `<|im_end|>` token:
+Using this template, each turn in the conversation is preceded by a `<|im_start|>` delimiter indicating the beginning of a message, followed by the role of the entity
+(either `user`, for content supplied by the user, or `assistant` for the model's responses), and finished with the `<|im_end|>` token:
 
 ```
 <s><|im_start|>user
@@ -265,7 +225,7 @@ The dataset used in the initial supervised fine-tuning stage consists of 469k co
 
 The synthetic conversations are generated using [DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), leveraging seed data and prompts from pre-training corpora, as well as other openly available instruction datasets.
 
-The table below provides a detailed breakdown of the datasets included in this mixture, specifying their origin, type, license, and contribution to the overall corpus.:
+The table below provides a detailed breakdown of the datasets included in this mixture, specifying their origin, type, license, and contribution to the overall corpus:
 
 | **Dataset** | **ca** | **en** | **es** | **eu** | **gl** | **pt** | **Total Conversations** |
 | --- | --- | --- | --- | --- | --- | --- | --- |
@@ -286,7 +246,7 @@ The table below provides a detailed breakdown of the datasets included in this m
 | **fineweb-edu_qa** | 23374 | 20803 | 23311 | 22284 | 22307 |  | 112079 |
 | **Total** | **81633** | **199730** | **89313** | **49265** | **36605** | **21711** | **478257** |
 
-Following the short-context supervised fine-tuning, a second stage was introduced using the remaining 12k short-context samples from our mix, together with 480 long-context samples. 
+Following the short-context supervised fine-tuning, a second stage was introduced using the remaining 9k short-context samples from our mix, together with 480 long-context samples. 
 
 The long-context data was synthetically generated with Salamandra-7B using source texts from FineWebEdu, FineWeb2, and Wikipedia. The length of the examples varies between 16k and 160k tokens. The resulting outputs were subsequently filtered with the same DeepSeek-V3-0324 model to ensure quality and consistency.
 
@@ -378,7 +338,7 @@ The following table provides a detailed overview of the supervised fine-tuning d
     <td>tower-blocks</td>
     <td>Mixture</td>
     <td>Various licenses (only open licensed instances are used)</td>
-    <td><a href="https://huggingface.co/datasets/projecte-aina/RAG_Multilingual">TowerBlocks-v0.2</a> filtered by subdataset license and the languages of interest.</td>
+    <td><a href="https://huggingface.co/datasets/Unbabel/TowerBlocks-v0.2">TowerBlocks-v0.2</a> filtered by subdataset license and the languages of interest.</td>
   </tr>
   <tr>
     <td>oasst2_self-identity-rephrase</td>
@@ -419,14 +379,15 @@ The following table provides a detailed overview of the supervised fine-tuning d
 
 ### Alignment Data
 
-The alignment data was synthetically generated from a corpus of approximately 401k prompts designed to improve both helpfulness and safety.
+The alignment data was synthetically generated from a corpus of approximately 403k prompts designed to improve both helpfulness and safety.
 
 - **Helpfulness**: Prompts include instruction following, mathematics, question answering, and reasoning tasks across Catalan, Spanish, English, Euskera, and Galician. Additionally, M-Personas conversations, a resource specifically generated for this project, were incorporated and will also be released.
 - **Safety**: Prompts were synthetically generated from seed prompts written by human annotators, covering nine harm categories to ensure broad coverage of safety-related scenarios.
 
-Following approaches similar to UltraFeedback  and PKU, each instrucition underwent the following process:
+Following approaches similar to UltraFeedback  and PKU, each instruction underwent the following process:
 
 1. Multiple responses were produced using a pool of permissively licensed models (see [Model Pool](#model-pool-for-synthetic-data-generation) on helpfulness or safety, depending on the prompt.
+2. These responses were rated by a judge (Deepseek-V3-0324). Helpfulness responses were given an overall rating, while safety responses were given a score based on their level of severity over a list of harm categories.
 3. Preference pairs were constructed from these ratings. This phase should be considered preliminary, as future versions of the model will incorporate human annotators to refine and curate the generation and evaluation pipeline.
 
 The table below presents the distribution of helpfulness prompts by language, detailing the number of examples contributed from each language:
@@ -487,7 +448,7 @@ In the table below, we list the permissively licensed models that were used to g
   <tr>
     <td>Deepseek</td>
     <td>DeepSeek-V3-0324</td>
-    <td>671</td>
+    <td>685</td>
     <td>aligned</td>
     <td>MIT</td>
   </tr>
@@ -528,14 +489,14 @@ In the table below, we list the permissively licensed models that were used to g
   </tr>
   <tr>
     <td>Mistral</td>
-    <td>Mixtral-8x7B-Instruct-v0_1</td>
+    <td>Mixtral-8x7B-Instruct-v0.1</td>
     <td>56</td>
     <td>aligned</td>
     <td>Apache 2.0</td>
   </tr>
   <tr>
     <td></td>
-    <td>Mistral-7B-Instruct-v0_3</td>
+    <td>Mistral-7B-Instruct-v0.3</td>
     <td>7</td>
     <td>aligned</td>
     <td>Apache 2.0</td>
@@ -579,7 +540,7 @@ In the table below, we list the permissively licensed models that were used to g
     <td>FLOR_BSC</td>
     <td>Aitana_6_3B_BSC_Instructed</td>
     <td>6.3</td>
-    <td>base</td>
+    <td>instructed</td>
     <td>Apache 2.0</td>
   </tr>
   <tr>
@@ -640,7 +601,7 @@ In the table below, we list the permissively licensed models that were used to g
 
 ### Gold-standard benchmarks
 
-Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from [SpanishBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/spanish_bench), [CatalanBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/catalan_bench), [BasqueBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/basque_bench) and [GalicianBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench), as well as existing English tasks available in the LM Evaluation Harness. These benchmarks include both new and existing tasks and datasets. The tables below report results for a representative selection of evaluation datasets, capturing model’s performance across a variety of tasks within these benchmarks.
+Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from [SpanishBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/spanish_bench), [CatalanBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/catalan_bench), [BasqueBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/basque_bench) and [GalicianBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench), as well as existing English tasks available in the LM Evaluation Harness. These benchmarks include both new and existing tasks and datasets. The tables below report results for a representative selection of evaluation datasets, capturing model's performance across a variety of tasks within these benchmarks.
 
 Only tasks that are human-generated, human-translated, or involve strong human-in-the-loop process (i.e., machine translation followed by professional revision or machine generation followed by human revision and annotation) were used. This approach explains the variation in the number of tasks reported across languages. As additional high-quality tasks are published, we will update the evaluation results accordingly. We also plan to expand evaluation to other languages, provided that the datasets meet our quality standards.
 
@@ -740,7 +701,7 @@ To assess the long-context capabilities of our model, we performed a "needle in
 - **Needle Phrase**: *"The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day."*
 - **System Prompt:** “You are a helpful AI bot that answers questions for a user. Keep your response short and direct”
 - **Retrieval Question**: *"What is the best thing to do in San Francisco?"*
-- **Evaluator**: [prometheus-8x7b-v2.0](https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0), used as the evaluation judge to determine wheter the model correctly retrieved and utilized the long-context information.
+- **Evaluator**: [prometheus-8x7b-v2.0](https://huggingface.co/prometheus-eval/prometheus-8x7b-v2.0), used as the evaluation judge to determine whether the model correctly retrieved and utilized the long-context information.
 
 This test specifically targets the model’s ability to retain and access information across very long sequences, providing a benchmark for evaluating its extended-context reasoning and retrieval performance.
 
@@ -795,7 +756,7 @@ This project has benefited from the contributions of numerous teams and institut
 
 We are especially grateful to our ILENIA project partners: CENID, HiTZ and CiTIUS for their participation. We also extend our genuine gratitude to the Spanish Senate and Congress, Fundación Dialnet, and the ‘Instituto Universitario de Sistemas Inteligentes y Aplicaciones Numéricas en Ingeniería (SIANI)’ of the University of Las Palmas de Gran Canaria. Many other institutions have been involved in the project. Our thanks to Òmnium Cultural, Parlament de Catalunya, Institut d'Estudis Aranesos, Racó Català, Vilaweb, ACN, Nació Digital, El món and Aquí Berguedà. We thank the Welsh government, DFKI, Occiglot project, especially Malte Ostendorff, and The Common Crawl Foundation, especially Pedro Ortiz, for their collaboration. 
 
-We would also like to give special thanks to the NVIDIA team, with whom we have met regularly, specially to: Ignacio Sarasua, Adam Henryk Grzywaczewski, Oleg Sudakov, Sergio Perez, Miguel Martinez, Felipes Soares and  Meriem Bendris. Their constant support has been especially appreciated throughout the entire process.
+We would also like to give special thanks to the NVIDIA team, with whom we have met regularly, especially to: Ignacio Sarasua, Adam Henryk Grzywaczewski, Oleg Sudakov, Sergio Perez, Miguel Martinez, Felipe Soares and Meriem Bendris. Their constant support has been especially appreciated throughout the entire process.
 
 Their valuable efforts have been instrumental in the development of this work.