danruban/gemma3-1b-tunedaxolotl2

See axolotl config

axolotl version: 0.8.0

base_model: google/gemma-3-1b-it
model_type: gemma
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_4bit: true
adapter: qlora

lora_r: 8                          # 💡 уменьшено для маленького датасета
lora_alpha: 16                    # 💡 пропорционально уменьшено
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj


deepspeed: /mnt/d/druban/axolotl/try1/zero1.json

datasets:
  - path: /mnt/d/druban/axolotl/try1/data/finetuning-dataset-chatml.jsonl
    type: chat_template
    field_messages: conversations

val_set_size: 0.1                 # 💡 10% для более надёжной валидации

output_dir: /mnt/d/druban/axolotl/try2

sequence_len: 2048
sample_packing: true
eval_sample_packing: false

gradient_checkpointing: true

batch_size: 2
micro_batch_size: 1

num_epochs: 10                    # 💡 увеличено с 3 до 10 для глубокого дообучения

optimizer: adamw_bnb_8bit
learning_rate: 5e-5               # 💡 чуть выше для малых данных
lr_scheduler: cosine
warmup_steps: 10

wandb_project: gemma-finetune

mnt/d/druban/axolotl/try2

This model is a fine-tuned version of google/gemma-3-1b-it on the /mnt/d/druban/axolotl/try1/data/finetuning-dataset-chatml.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 4.5024

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 2
• total_eval_batch_size: 2
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 10.0

Training results

Training Loss	Epoch	Step	Validation Loss
6.4781	1.0	7	13.0395
5.5099	2.0	14	9.8655
4.5869	3.0	21	7.0045
4.1804	4.0	28	5.8688
4.0683	5.0	35	5.1847
4.0524	6.0	42	4.7762
3.9197	7.0	49	4.6118
3.6931	8.0	56	4.5488
3.995	9.0	63	4.5053
3.7989	10.0	70	4.5024

Framework versions

• PEFT 0.15.1
• Transformers 4.51.0
• Pytorch 2.5.1+cu121
• Datasets 3.5.0
• Tokenizers 0.21.1