Skip to content

Fine-tuning Best Practices

Overview

Best Practices for fine-tuning: practical tips to improve quality, avoid pitfalls, and maximize ROI on fine-tuning investments.

  • Data quality: Beats quantity every time
  • Evaluation: Measure before, during, after
  • Iteration: Expect multiple rounds
  • Monitoring: Track degradation in production

Data Preparation Best Practices

Data Quality > Quantity

Experiment: Impact of data quality

Scenario 1: 10K low-quality examples
 - Noisy labels (50% mislabeled)
 - Inconsistent formatting
 - Many duplicates
 - Result: Model learns noise, accuracy 65%

Scenario 2: 1K high-quality examples
 - Accurate labels
 - Consistent formatting
 - No duplicates
 - Result: Model learns pattern, accuracy 88%

Lesson: 1K high-quality >> 10K low-quality

Quality checklist:
 - Are labels correct? (Verify random sample)
 - Is formatting consistent? (Standard template)
 - Are duplicates removed? (Check similarity)
 - Is data relevant? (Matches your task)
 - Is distribution realistic? (Reflects production)

Data Distribution Matters

Critical: Match training distribution to production

Bad example:
 - Training data: 80% positive examples, 20% negative
 - Production: 50% positive, 50% negative
 - Result: Model biased toward positive, poor negative classification

Good approach:
 - Analyze production distribution
 - Match training to production
 - If skewed: Use class weights during training
 - Result: Balanced model

Implementation:
```python
from sklearn.utils.class_weight import compute_class_weight

# Compute class weights
class_weights = compute_class_weight(
 'balanced',
 classes=np.unique(labels),
 y=labels
)

# Use in training
trainer = Trainer(
 model=model,
 loss_fn=CrossEntropyLoss(weight=torch.tensor(class_weights))
)

Augmentation & Synthetic Data

When you have little real data:

Strategy 1: Data augmentation
 - Paraphrase existing examples
 - Back-translation (translate to French, back to English)
 - Minor modifications (typo injection for robustness)
 - 2-5x data increase

Example:
Original: "How do I install Python?"
Paraphrase 1: "What's the process for setting up Python?"
Paraphrase 2: "How can I get Python on my computer?"
 - 3x data from 1 original!

Strategy 2: Synthetic data generation
 - Use GPT-4 to generate (Instruction, Output) pairs
 - Cost: ~$0.01-0.10 per example
 - Quality: Good, but may have hallucinations
 - Validation: Verify sample with humans

Strategy 3: Semi-supervised learning
 - Use unlabeled data
 - Pseudo-labeling: Model labels itself
 - High confidence predictions: Use as labels
 - Iteratively improve

Recommendation:
 - Combine: Real data + augmentation + synthetic
 - Validate: Check quality of synthetic data

Training Best Practices

Learning Rate & Optimization

Common mistakes and fixes:

Mistake 1: Learning rate too high
 - Symptoms: Loss oscillates, doesn't improve
 - Fix: Reduce learning rate (10x smaller)
 - Recommended: 1e-5 to 5e-5 for fine-tuning

Mistake 2: Learning rate too low
 - Symptoms: Very slow improvement
 - Fix: Increase learning rate
 - But: Don't overshoot!

Recommendation:
 - Start: Conservative (1e-5)
 - Monitor: Training curves
 - Adjust: If too slow, increase 2-3x
 - Sweet spot: Clear improvement within first epoch

Learning rate schedule:
```python
from transformers import get_linear_schedule_with_warmup

num_training_steps = len(train_dataloader) * num_epochs
warmup_steps = int(0.1 * num_training_steps) # 10% warmup

scheduler = get_linear_schedule_with_warmup(
 optimizer,
 num_warmup_steps=warmup_steps,
 num_training_steps=num_training_steps
)

# In training loop:
loss.backward()
optimizer.step()
scheduler.step()

### Batch Size & Gradient Accumulation

Batch size affects convergence:

Small batch (1-2): Higher quality gradients (noise helps exploration) Slower training, more memory swaps

  • Use if: Limited GPU memory, small dataset

Medium batch (4-16): Good balance Stable training

  • Recommended default!

Large batch (32-128): Faster training Less noise, may get stuck Needs more memory

  • Use if: Large dataset, sufficient memory

When memory limited: Gradient accumulation

# Effective batch size = per_device_batch_size × gradient_accumulation_steps

training_args = TrainingArguments(
 per_device_train_batch_size=4, # Per GPU
 gradient_accumulation_steps=4, # Accumulate 4 steps
 # Effective batch size = 4 × 4 = 16
)

### Early Stopping & Overfitting Prevention

Overfitting indicator:

  • Training loss: Decreasing
  • Validation loss: Increasing
  • Signal: Model memorizing, not learning

Prevention:

  1. Early Stopping (most important!)
early_stopping = EarlyStoppingCallback(
 early_stopping_patience=3, # Stop if val loss doesn't improve for 3 steps
 early_stopping_threshold=0.0
)

trainer = Trainer(
 callbacks=[early_stopping],
 eval_strategy="steps",
 eval_steps=100, # Evaluate every 100 steps
)
  1. Dropout & Weight Decay
training_args = TrainingArguments(
 weight_decay=0.01, # L2 regularization
 dropout_rate=0.1, # Prevent overfitting
)

  1. Reduce epochs

  2. Instead of 5 epochs: Try 2-3

  3. Less risk of overfitting

  4. Usually sufficient for fine-tuning

  5. Use larger learning rate

  6. Faster convergence

  7. Less time to overfit
  8. But: Don't overshoot!
-

## Evaluation Best Practices

### Separate Train/Val/Test

Critical: Never evaluate on training data!

WRONG approach:

  • Train on 10K examples
  • Evaluate on same 10K examples
  • Get 95% accuracy
  • Misleading! (Actually 70% on new data)

RIGHT approach:

  • Split: 80% train (8K), 10% val (1K), 10% test (1K)
  • Train on 8K
  • Monitor on 1K validation
  • Final test on separate 1K
  • Real performance estimate!

Split strategy:

  • Stratified split: Keep class distribution

Implementation:

from sklearn.model_selection import train_test_split

train, temp = train_test_split(
 data, test_size=0.2, random_state=42, stratify=labels
)
val, test = train_test_split(
 temp, test_size=0.5, random_state=42, stratify=temp_labels
)

### Multiple Evaluation Metrics

Don't rely on single metric!

Example task: Sentiment classification

Accuracy (bad alone):

  • Can be 90% by predicting majority class!

Better approach: Multiple metrics

  • Accuracy: Overall performance
  • Precision: How many positive predictions were correct
  • Recall: How many actual positives did we find
  • F1: Balance between precision & recall
  • Confusion matrix: See where model fails

Implementation:

from sklearn.metrics import classification_report, confusion_matrix

y_pred = model.predict(test_data)
y_true = test_labels

print(classification_report(y_true, y_pred))
print(confusion_matrix(y_true, y_pred))

Interpretation:

  • High accuracy, low recall: Missing positive cases
  • High precision, low recall: Too conservative
  • Balanced F1: Good overall
  • Confusion matrix: Reveals specific error patterns
### Domain-Specific Evaluation

Beyond standard metrics, measure what matters:

For medical domain:

  • Factuality: Are medical facts correct?
  • Safety: Is it safe to follow recommendations?
  • Evidence-based: Backed by research?

For legal domain:

  • Accuracy: Are legal claims correct?
  • Completeness: All relevant laws addressed?
  • Jurisdiction: Correct for user's location?

For code domain:

  • Correctness: Code runs and produces right output?
  • Efficiency: O(n) complexity as expected?
  • Security: No vulnerabilities?

Evaluation approach:

  • Human expert review: 5-10% of samples
  • Automated tests: Where applicable
  • Comparative: Compare to baseline/alternatives
  • Iterative: Improve based on feedback
-

## Common Pitfalls & Solutions

### Pitfall 1: Catastrophic Forgetting

Problem: Fine-tuning forgets base knowledge

Example:

  • Pre-trained on 1.3T tokens (general knowledge)
  • Fine-tune on 10K domain examples
  • Loses general ability, only knows domain
  • Can't handle general questions anymore!

Solution: Mix in base data during fine-tuning

  • 80% domain examples
  • 20% general examples (mixed in)
  • Result: Keeps general + gains domain

Implementation:

# Mix domain + general data
domain_dataset = load_dataset("domain_data.json")
general_dataset = load_dataset("wikipedia.json").sample(0.2)

combined = concatenate_datasets([
 domain_dataset,
 general_dataset
])

trainer = Trainer(train_dataset=combined,...)

### Pitfall 2: Overfitting to Small Dataset

Problem: Model memorizes 1K examples instead of learning

Indicators:

  • Training accuracy: 99%+
  • Validation accuracy: 60-70%
  • Large gap = overfitting!

Solutions:

  1. More data (best solution)

  2. Collect more examples

  3. Use data augmentation
  4. Use synthetic data

  5. Target: 5K-10K examples minimum

  6. Early stopping

  7. Stop before overfitting happens

  8. Monitor validation loss

  9. Regularization

  10. Dropout

  11. Weight decay

  12. Mixup (data augmentation)

  13. Smaller model

  14. Use 3B instead of 7B

  15. Less capacity = less memorization
  16. Trade-off: Slightly lower quality on sufficient data
### Pitfall 3: Distribution Mismatch

Problem: Training data doesn't match production

Example:

  • Training: Academic papers (formal, long)
  • Production: Twitter posts (casual, short)
  • Model trained on wrong distribution!

Solution: Match distribution

  • Analyze production data
  • Collect training data from similar distribution
  • Test on representative samples
  • Regular re-evaluation

Monitoring in production:

def detect_distribution_shift(new_batch, baseline_embeddings):
 """
 Detect if new data is from different distribution
 """

 new_embeddings = embed(new_batch)

 # Compare to baseline
 similarity = cosine_similarity(new_embeddings, baseline_embeddings).mean()

 if similarity < 0.8: # Threshold
 alert("Distribution shift detected!")
 return True
 return False

### Pitfall 4: Not Saving Best Model

Problem: Overfitting causes validation loss to increase later

Example:

  • Epoch 1: Val loss = 0.5, Save model
  • Epoch 2: Val loss = 0.4, Save model
  • Epoch 3: Val loss = 0.45, Don't save (overfitting)
  • Epoch 4: Val loss = 0.6, Don't save (worse)
  • If you keep training: Training at epoch 4, model is worse!

Solution: Save best model checkpoint

from transformers import EarlyStoppingCallback

# Automatically save best model
callback = EarlyStoppingCallback(
 early_stopping_patience=3,
 save_best_model=True
)

trainer = Trainer(
 callbacks=[callback],
 model=model,
)

# At end, load best model
model = AutoModelForCausalLM.from_pretrained("./best_model")

-

## Deployment Best Practices

### Version Your Models

Track everything:

  • Model architecture
  • Training data version
  • Training hyperparameters
  • Performance metrics
  • Deployment date

Example versioning:

models
 - sentiment-v1.0 (initial)
 - sentiment-v1.1 (bug fix)
 - sentiment-v2.0 (SFT on domain data)
 - sentiment-v2.1 (DPO improvement)
 - sentiment-v3.0 (new data collection)

Metadata file:

{
 "version": "v2.1",
 "model": "LLaMA 7B",
 "training_date": "2024-08-08",
 "training_data_size": 50000,
 "metrics": {
 "accuracy": 0.92,
 "f1_score": 0.91
 },
 "hyperparameters": {
 "learning_rate": 2e-4,
 "batch_size": 4,
 "epochs": 3
 },
 "status": "production"
}

### Monitor in Production

Key metrics to track:

  1. Performance metrics

  2. Accuracy on recent data

  3. Precision/Recall

  4. F1 score (biweekly evaluation)

  5. Data drift

  6. Distribution shift detection

  7. Changes in input characteristics

  8. Alert on significant changes

  9. User feedback

  10. Thumbs up/down on responses

  11. Corrections users make

  12. Use to identify problems

  13. Model staleness

  14. How old is the model?

  15. Time since last retraining
  16. Alert: Retrain if >6 months old

Implementation:

def monitor_model_performance():
 """Monitor key metrics"""

 recent_data = get_recent_predictions(days=7)

 metrics = {
 'accuracy': compute_accuracy(recent_data),
 'precision': compute_precision(recent_data),
 'recall': compute_recall(recent_data),
 }

 # Compare to baseline
 if metrics['accuracy'] < baseline_accuracy * 0.95:
 alert("Performance degradation detected!")

 # Log for tracking
 log_metrics(metrics)

```

Key Takeaways

Data quality > quantity: 1K clean beats 10K noisy Multiple evaluation metrics: Never trust accuracy alone Early stopping: Prevent overfitting automatically Monitor production: Track metrics post-deployment Mix in base data: Prevent catastrophic forgetting

-