Dataset

Fine Tuning

• dataset_text_field="text": Specifies the field in the dataset containing the text.

• max_seq_length=512: Maximum sequence length for input texts.

• per_device_train_batch_size=2: Batch size per GPU.

• gradient_accumulation_steps=4: Number of steps to accumulate gradients before updating.

• optim="paged_adamw_32bit": Specifies the optimizer.

• save_steps=50: Save a checkpoint every 50 steps.

• logging_steps=5: Log training metrics every 5 steps.

• learning_rate=2e-4: The learning rate for training.

• fp16=True: Use mixed precision training.

• max_grad_norm=0.3: Maximum gradient norm for gradient clipping.

• max_steps=200: Total number of training steps.

• warmup_ratio=0.03: Portion of training steps used for learning rate warmup.

• lr_scheduler_type="linear": Type of learning rate scheduler.

• gradient_checkpointing=True: Use gradient checkpointing to save memory.