laurent-maille/Valuoty-industry-plc-4B-V0.6Adap

See axolotl config

axolotl version: 0.8.0.dev0

# === MODÈLE ===
base_model: Qwen/Qwen3-4B-Thinking-2507
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

# === LORA CONFIGURATION ===
adapter: lora
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
lora_modules_to_save:
  - lm_head

# === DATASET ===
datasets:
  - path: laurent-maille/pcl-test-S27
    type: chat_template
    field_messages: messages
    conversation: chat
    test_size: 0.2  #  Split automatique 80/20

dataset_prepared_path: ./prepared/plc_sharegpt
output_dir: ./outputs/valuoty-indus-plc-4B

# === DATA LOADING ===
dataloader_num_workers: 4
dataloader_pin_memory: true
group_by_length: true

# === TRAINING SETTINGS ===
train_on_inputs: false
mask_user_tokens: true

# === SEQUENCES ===
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

# === QUANTIZATION ===
load_in_4bit: true
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: true
bnb_4bit_compute_dtype: bfloat16
bf16: true

# === OPTIMIZER ===
optimizer: adamw_torch
learning_rate: 2e-5
weight_decay: 0.01
max_grad_norm: 1.0
lr_scheduler: cosine
warmup_ratio: 0.08
seed: 42

# === TRAINING PARAMETERS ===
num_epochs: 3
micro_batch_size: 2
gradient_accumulation_steps: 32
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

# === ATTENTION ===
flash_attention: true

# === SAVING & EVALUATION ===
save_safetensors: true
save_strategy: steps
save_steps: 250
save_total_limit: 3
eval_strategy: steps
eval_steps: 125
logging_steps: 25

# === EARLY STOPPING  ===
load_best_model_at_end: true
metric_for_best_model: eval_loss  
greater_is_better: false
#early_stopping_patience: 3

# === LOGGING ===
wandb_project: Valuoty-plc
wandb_run_name: V0.5_a40_run_01
wandb_watch: gradients

outputs/valuoty-indus-plc-4B

This model is a fine-tuned version of Qwen/Qwen3-4B-Thinking-2507 on the laurent-maille/pcl-test-S27 dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 7
• num_epochs: 3.0

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.51.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.4