Cost Optimization Strategies: Reducing Training and Inference Expenses¶

Overview¶

Cost optimization reduces both training costs (compute hours) and inference costs (per-query). Combining techniques achieves 10-100x cost reduction through hardware selection, parallelization, quantization, and efficiency techniques.

Training cost drivers: Compute hours, GPU hours, data transfer
Inference cost drivers: Token generation time, model size, concurrent users
Optimization: 10-100x cost reduction possible through technique combination
Trade-off: Quality vs. cost

Training Cost Optimization¶

Hardware Selection¶

GPU Cost Comparison (per hour):

GPU         Cost/hr   TFLOPS  Efficiency (TFLOPS/$)
─────────────────────────────────────────────────
V100        $2.50     125     50
A100        $3.50     312     89
H100        $6.00     990     165
L40S        $4.00     722     180

Recommendation by scale:
  - Small training (<100B FLOPs): A100 (best balance)
  - Medium training (100B-10T FLOPs): H100 (better efficiency)
  - Large training (>10T FLOPs): H100 (only practical option)

Cost example: Train LLaMA 7B

With A100 (8 GPUs):
  - Compute: 8 × $3.50/hr × 50 hours = $1,400
  - Data transfer: $100
  - Storage: $50
  - Total: $1,550

With H100 (8 GPUs):
  - Compute: 8 × $6/hr × 20 hours = $960 (4x faster)
  - Data transfer: $100
  - Storage: $50
  - Total: $1,110 (28% cheaper despite higher hourly cost!)

Key insight:
  - More expensive GPU/hour, but much faster
  - Amortizes fixed costs, saves money!

Parallelization Efficiency¶

Cost of parallelization:

Sequential training (impossible):
  - 1 GPU, many weeks → unfeasible

Data parallelism (efficient):
  - N GPUs, communication each step
  - Efficiency: 95% of ideal (only 5% waste)
  - Cost reduction: 18x for 16 GPUs

Pipeline parallelism (some overhead):
  - N GPUs, pipeline bubbles
  - Efficiency: 75% of ideal (25% waste)
  - Cost reduction: 12x for 16 GPUs
  - More complex setup

FSDP (balanced):
  - N GPUs, efficient communication
  - Efficiency: 80-90% (only 10-20% waste)
  - Cost reduction: 14x for 16 GPUs
  - Good balance of speed and complexity

Recommendation:
  - 8 GPUs: Use data parallelism
  - 16-64 GPUs: Use FSDP
  - 64+ GPUs: Consider pipeline + FSDP hybrid

Efficiency Techniques¶

Technique              Speedup  Cost Reduction
─────────────────────────────────────────────
Mixed precision (AMP) 2-3x     50-60%
Gradient checkpointing 1.3x     20% (memory, but compute cost up)
Flash Attention       1.5x     30%
Gradient accumulation 1.2x     10% (batching amortization)
All combined          ~5x      80%

Cumulative effect:
Training cost with all techniques:
  - Baseline: $10,000 (100 GPU-hours @ $100/hr)
  - +AMP: $5,000 (2.5x faster)
  - +Flash Attention: $3,300 (1.5x more)
  - +FSDP (16 GPUs): $200 (best utilization!)
  - Total: 50x cost reduction!

Reality check:
  - Baseline: 100 GPU-hours
  - With optimization: 2-5 GPU-hours
  - Cost: $100-500 (vs $10,000!)

Inference Cost Optimization¶

Per-Query Costs¶

Cost breakdown per query:

Base cost: GPU time
  - Generate 100 tokens @ 100 tokens/sec = 1 second
  - GPU hour cost: $0.003-0.01 per second
  - Query cost: $0.003-0.01

Optimization: Use smaller model
  - 70B model: 0.3x throughput (3x slower)
  - 7B model: 0.9x baseline throughput
  - 7B query: $0.001-0.003 (1/3 of 70B)
  - Cost: 3-10x reduction!

Optimization: Use quantization
  - int4 quantization: 2x faster, 4x smaller
  - Same 70B, but 2x throughput
  - Query cost: $0.0015-0.005 (1/2 of FP16)
  - Cost: 2x reduction

Combined (7B + int4 quantization):
  - Base 70B: $0.01 per 100 tokens
  - 7B: $0.003
  - 7B + int4: $0.0015
  - Total: 6-10x cost reduction!

Batching and Utilization¶

Utilization impact on cost:

Scenario 1: Low utilization (sparse requests)
  - Requests come slowly
  - GPU often idle
  - Effective cost: High (paying for idle time)

Example:
  - 10 requests/hour
  - 1 GPU, $10/hour
  - Cost per request: $1.00

Scenario 2: High utilization (many requests)
  - Requests batched efficiently
  - GPU always busy
  - Effective cost: Low

Example:
  - 1000 requests/hour (same throughput, better batching)
  - 1 GPU, $10/hour
  - Cost per request: $0.01

Improvement: 100x cost reduction through batching!

Recommendation:
  - High volume → Amortize GPU cost
  - Batch many requests together
  - Share computation
  - 10-100x per-request cost savings

Strategy: Training vs Fine-tuning¶

Cost Comparison¶

Scenario: Need domain-specific LLM

Option 1: Train from scratch
  - Training cost: $100,000-1,000,000
  - Quality: Best possible
  - Time: Months

Option 2: Fine-tune pre-trained
  - Pretrained model: Free or $100-1000 (download)
  - Fine-tune cost: $100-10,000
  - Quality: 90-95% of option 1
  - Time: Days to weeks

Option 3: Use pre-trained + LoRA
  - Pretrained model: Free
  - LoRA training: $1-100
  - Quality: 85-90% of option 1
  - Time: Hours to days

Cost-benefit:
  - Option 1: Highest quality, highest cost
  - Option 2: Good quality, moderate cost (usually best)
  - Option 3: Acceptable quality, minimal cost
  - Recommendation: Option 2 or 3 (pre-train amortizes cost!)

Multi-task model sharing:

Scenario: Need 5 domain-specific models
  - Option A: Train 5 models separately
    - Cost: 5 × $100,000 = $500,000
│
  - Option B: Train 1 base model, LoRA for 5 tasks
    - Base model: $100,000 (train once)
    - LoRA 1-5: $5 × $100 = $500
    - Total: $100,500
│
  - Option C: Use existing pre-trained, LoRA for 5 tasks
  - Pre-trained: Free
  - LoRA 1-5: $500
  - Total: $500

Cost reduction:
  - Option A: Baseline
  - Option B: 80% reduction (cost amortization)
  - Option C: 99.9% reduction (leverage pre-training)!

Key insight:
  - Pre-training cost >> fine-tuning
  - Massive ROI from reusing pre-trained models

Deployment Cost Optimization¶

Infrastructure Decisions¶

Options for deployment:

Option 1: Buy GPUs ($50K-100K upfront)
  - Initial: $100K
  - Running: $1-5K/month (power, cooling)
  - 3 years: $130-180K total
  - Good for: High traffic, long-term

Option 2: Cloud on-demand ($0.003-0.01/token)
  - Upfront: $0
  - Usage: Pay per query
  - 3 years (1B tokens): $3K-10K
  - Good for: Variable traffic, short-term, low volume

Option 3: Spot instances (AWS, GCP) ($0.001-0.003/token)
  - Upfront: $0
  - Usage: 70-80% discount vs on-demand
  - 3 years (1B tokens): $1K-3K
  - Good for: Can tolerate interruptions, cost-sensitive

Option 4: Serverless (Lambda, etc.)
  - Upfront: $0
  - Usage: $0.01-0.05 per invocation + compute
  - Good for: Occasional requests, extreme simplicity

Break-even analysis:
  - High volume (>10M tokens/month) → Buy GPUs
  - Medium volume (1-10M tokens) → Spot instances
  - Low volume (<1M tokens) → On-demand or serverless

Autoscaling for Cost¶

class CostOptimizedScaling:
    """Auto-scale based on cost, not just demand"""

    def should_add_gpu(self, queue_depth, cost_per_hour):
        """
        Add GPU only if cost-effective
        """
        # Current wait time
        wait_time = queue_depth / self.throughput

        # Cost of waiting (user churn, SLA penalties)
        cost_of_waiting = wait_time * self.penalty_per_second

        # Cost of adding GPU
        cost_of_gpu = cost_per_hour / 3600  # per second

        # Add GPU only if it pays for itself
        return cost_of_waiting > cost_of_gpu * 2  # 2x margin

# Result: Only scale when economically justified!

Cost Monitoring & Optimization¶

Key Metrics¶

Metrics to track:

1. Cost per query
  - Total cost / number of queries
  - Goal: Minimize

2. Cost per token generated
  - Total cost / total tokens
  - Goal: Minimize (use for billing)

3. GPU utilization
  - % of time GPU is computing (vs idle)
  - Goal: >80% (higher is more cost-efficient)

4. Cost per SLA compliance
  - Cost to maintain latency target
  - Goal: Optimize trade-off

5. Model efficiency
  - Quality per unit compute cost
  - Goal: Maximize (quality/$)

Cost Optimization Roadmap¶

Phase 1: Reduce training cost (one-time)
  - Switch to H100 (better efficiency)
  - Use AMP + Flash Attention
  - Use FSDP for parallelization
  - Goal: 80% training cost reduction

Phase 2: Reduce inference cost (ongoing)
  - Quantize to int4
  - Use LoRA for personalization (not full retraining)
  - Batch requests aggressively
  - Goal: 60% inference cost reduction

Phase 3: Optimize deployment
  - Use spot instances for variable load
  - Auto-scale based on cost metrics
  - Monitor and optimize constantly
  - Goal: 50% deployment cost reduction

Total impact: ~90% cost reduction possible!

Key Takeaways¶

💰 Pre-training cost >> fine-tuning cost (leverage pre-trained!)
⚡ Hardware selection crucial (H100 faster despite higher $/hr)
🔄 Parallelization efficiency matters (FSDP better than naive)
📊 Batching amortizes GPU cost (high utilization = low per-query cost)
🎯 Combined techniques: 10-100x cost reduction possible

Distributed Training - Cost-efficient training strategies
Quantization - Inference cost reduction
Lora - Cost-efficient fine-tuning
Continuous Batching - Inference utilization
Load Balancing & Request Routing - Maximize throughput per GPU