Capability Classification

Overview

Not all models are created equal. Understanding model capabilities—both their strengths and limitations—is crucial for:

• Selecting the right model for your use case
• Making cost-performance tradeoffs
• Fine-tuning efficiently
• Avoiding costly mistakes

-

Model Capability Tiers (2024)

Tier 1: Frontier Models

Examples: GPT-4, Claude 3 Opus, Llama 3 405B, Gemini Ultra

Performance Level: State-of-the-art
Cost: $$$ (Expensive)
Speed: Moderate
Context: 32K - 128K tokens
Specialization: General purpose

Characteristics:
Best performance on all benchmarks
Excellent reasoning capabilities
Strong instruction following
Few hallucinations (2-3%)
Multilingual support
Long-context understanding

Very expensive ($0.01-0.10 per 1K tokens)
Often API-only (no weights)
Slower inference
Rate-limited

Use When:
- Reasoning is critical
- Quality > cost
- Enterprise customers
- Complex problem solving

Examples:
- GPT-4: $0.03 input, $0.06 output per 1K tokens
- Claude 3 Opus: $0.015 input, $0.075 output
- Llama 3 405B: Free (self-hosted) or ~$1-2 per 1M tokens (API)

MMLU: 84-93%
HumanEval: 90-94%
GSM8K: 93-96%

Tier 2: Production Models

Examples: Llama 70B, Mixtral 8x22B, Claude 3 Sonnet, Gemini Pro

Performance Level: Strong, production-ready
Cost: $$ (Moderate)
Speed: Fast
Context: 4K - 32K tokens
Specialization: Both general + specialized variants

Characteristics:
Excellent performance (80%+ MMLU)
Fast inference
Reasonable cost
Available as open weights
Good for fine-tuning
Production-grade quality

 Slightly higher hallucination (4-6%)
 Some reasoning limitations
 Context window limitations (some)
 May need domain fine-tuning

Use When:
- Cost-performance matters
- Latency is important
- Need to fine-tune
- Production deployment
- Scale matters

Examples:
- Llama 70B: ~$0.59 per 1M tokens (self-hosted)
- Claude 3 Sonnet: ~$0.003 per 1K tokens
- Mixtral 8x22B: $0.27 per 1M tokens (self-hosted)

MMLU: 78-87%
HumanEval: 75-90%
GSM8K: 50-85%

Tier 3: Efficient Models

Examples: Mistral 7B, Llama 13B, Llama 7B, Phi, Qwen

Performance Level: Capable for constrained scenarios
Cost: $ (Cheap)
Speed: Very fast
Context: 4K - 32K tokens
Specialization: Specialized (code, math, etc.)

Characteristics:
Very fast inference (10-50ms per token)
Cheap or free
Fits on consumer GPU
Can run on edge devices
Easy to fine-tune
Large community support

Lower absolute performance
Weaker reasoning
Higher hallucination (8-15%)
Needs more hand-holding

Use When:
- Speed is critical
- Edge/mobile deployment
- Budget is constrained
- Task is specialized (with fine-tuning)
- User has limited compute

Examples:
- Mistral 7B: Free (self-hosted)
- Llama 13B: Free (self-hosted), works on 24GB GPU
- Phi 3.5: Free, fits on 8GB VRAM

MMLU: 45-72%
HumanEval: 50-73%
GSM8K: 25-60%

Tier 4: Specialized/Distilled Models

Examples: Code Llama 7B, Medical LLMs, DistilBERT

Performance Level: Excellent in narrow domain
Cost: $ (Free - Cheap)
Speed: Fastest
Context: 2K - 8K tokens
Specialization: Single domain (code, medical, legal)

Characteristics:
Best in specific domain
Extremely fast
Tiny (7B or smaller)
Minimal hallucinations in domain
Energy efficient

Poor outside domain
Limited general knowledge
Can't do multi-task

Use When:
- Single well-defined task
- Domain expertise needed
- Latency critical
- Energy limited (mobile)
- Inference at edge

Examples:
- Code Llama 7B: 84% HumanEval (vs Llama 7B: 33%)
- BloombergGPT (Finance): 97% on finance QA
- BioBERT (Biology): 95% on biomedical NER

Performance:
- Excels in domain (+30-50% vs general models)
- Struggles outside (+30-50% worse)

-

Capability Matrix

By Task Type

Task Type Tier 1 Tier 2 Tier 3 Tier 4
──────────────────────────────────────────────────
General QA 
Reasoning 
Math 
Code Generation (specialized)
Writing 
Translation 
Classification (domain)
NER/Tagging (domain)
Summarization 
Long Document 
Common Sense 

= Excellent
 = Good
 = Acceptable (may need tuning)
= Poor

By Constraint

Constraint Priority Best Tier
──────────────────────────────────────────────
Maximum Quality 1. Quality Tier 1 (Frontier)
 2. Cost ignored

Balance 1. Performance Tier 2 (Production)
 2. Cost

Cost Critical 1. Cost Tier 3 (Efficient)
 2. Acceptable quality

Speed Critical 1. Latency Tier 3
 2. Throughput (Efficient/Specialized)

Domain Specific 1. Domain perf Tier 4 (Specialized)
 2. General perf + fine-tuning Tier 3

Edge/Mobile 1. Size < 10GB Tier 3
 2. Speed

-

Performance Trade-offs

Frontier vs Production vs Efficient

# Scenario
# Requirements

# Option 1
option_1 = {
 "MMLU": 86.5,
 "Latency P99": 150, # ms (very fast, API)
 "Cost per request": 0.006,
 "Monthly cost": 6000 * 1000 * 30 = 180_000, # Assuming 100 users/day
 "Quality": "Best-in-class",
 "Hallucination": "2%"
}

# Option 2
option_2 = {
 "MMLU": 82.9,
 "Latency P99": 200, # ms (self-hosted, batch)
 "Cost per request": 0.0006,
 "Monthly cost": 5000, # GPU + infrastructure
 "Quality": "Excellent",
 "Hallucination": "5%"
}

# Option 3
option_3 = {
 "MMLU": 60, # But fine-tuned on domain data: +15%
 "Domain MMLU": 75,
 "Latency P99": 50, # Very fast
 "Cost per request": 0.000001,
 "Monthly cost": 500, # Smaller GPU
 "Quality": "Domain-specific (excellent)",
 "Hallucination": "8% overall, 2% in domain"
}

# Analysis
comparison = {
 "Quality Ranking": ["GPT-4", "Llama 70B", "Mistral 7B+FT"],
 "Cost Ranking": ["Mistral 7B+FT", "Llama 70B", "GPT-4"],
 "Speed Ranking": ["Mistral 7B+FT", "GPT-4 API", "Llama 70B"],

 "Recommendation": {
 "If budget > $100k/month": "GPT-4 (API)",
 "If budget = $5-20k/month": "Llama 70B (self-hosted)",
 "If budget < $5k/month": "Mistral 7B + fine-tuning"
 }
}

---

Specialization Dimensions

By Language

Language Focus Best Model Performance
────────────────────────────────────────────────────
English Only GPT-4, Claude, Llama SOTA
Multilingual mT5, XLM-R, Gemini Strong (40+ langs)
Chinese Optimized ChatGLM3, Qwen Near-English SOTA
Arabic Optimized Jais, AraLLaMA Better than general
Japanese Swallow, Sakura Native quality

By Domain

Domain Best Model Specialization
──────────────────────────────────────────────────────────
General Purpose GPT-4, Claude, Llama No specialization
Code Generation GPT-4, Code Llama +30-40% accuracy
Mathematics Claude 3 Opus, GPT-4 Strong CoT
Medical MedPaLM, BioLLaMA Domain knowledge
Finance BloombergGPT Domain data
Legal LegalBERT, JurisCA Legal terminology
Scientific SciBERT, ArxivGPT Paper understanding

By Context Window

Context Needed Best Model Max Context
─────────────────────────────────────────────────
Short (< 4K) Most models 4K - unlimited
Medium (4-32K) Llama 70B, Claude 32K - 128K
Long (32-128K) GPT-4, Claude 3, GPT 128K - 200K
Ultra (> 128K) GPT-4 Turbo Up to 200K

Capability Combination Matrix

 English| Chinese| Code| Math| Domain
────────────────────────────────────────────────────────
GPT-4 || | | Moderate
Claude 3 ||| | Moderate
Llama 70B |||| Low
ChatGLM3| ||| Moderate
Code Llama 7B|| || Code Only
Mistral 7B|||| Low
BloombergGPT|||| Finance

-

Selection Framework

Decision Tree

graph TD
 A["Need LLM for Task?"] --> B{What's most<br/>important?}

 B -->|Quality| C{Budget > $100k/month?}
 C -->|Yes| D["GPT-4 / Claude Opus"]
 C -->|No| E["Llama 70B<br/>self-hosted"]

 B -->|Cost| F{Need<br/>fine-tuning?}
 F -->|Yes| G["Mistral 7B<br/>+ LoRA"]
 F -->|No| H["Free Open Models<br/>Llama 7B/13B"]

 B -->|Speed| I{Latency<br/>critical?}
 I -->|< 100ms| J["Mistral 7B"]
 I -->|< 500ms| K["Llama 13B"]

 B -->|Domain| L{Specialized<br/>model exists?}
 L -->|Yes| M["Use specialized<br/>+ fine-tune if needed"]
 L -->|No| N["Fine-tune<br/>general model"]

 style D fill:#99ff99
 style E fill:#ffff99
 style G fill:#ffff99
 style H fill:#99ff99
 style J fill:#99ff99
 style K fill:#ffff99
 style M fill:#99ff99
 style N fill:#ffff99

---

Benchmarks by Tier

Raw Benchmark Scores

Metric Tier 1 Tier 2 Tier 3 Tier 4
────────────────────────────────────────────────────────────
MMLU 84-93% 78-87% 45-72% Varies
HumanEval 90-94% 75-90% 50-73% Domain-specific
GSM8K 93-96% 50-85% 25-60% Low (unless math)
HellaSwag 97% 92% 78% Varies
LongBench 82-88% 65-78% 40-55% Low
NER 92% 88% 82% 96%+ (domain)
Sentiment 93% 89% 85% 91% (domain)

What These Differences Mean

Tier 1 vs Tier 2 vs Tier 3 (MMLU: 87% vs 82% vs 60%):

Scenario: Answer 100 customer questions

Tier 1: 87 correct answers
Tier 2: 82 correct answers (5% error rate)
Tier 3: 60 correct answers (27% error rate)

Impact on Business:
- Tier 3 needs extensive fine-tuning or post-processing
- Tier 2 acceptable for most production
- Tier 1 best for complex reasoning

With Domain Fine-tuning:
Tier 3 after FT: 75%+ (competitive with Tier 2!)
Tier 3 after extensive FT: 82%+ (approaching Tier 2!)

Key Insight:
Fine-tuning can bridge tiers for specific domains

-

Real-World Examples

Example 1: Customer Support Chatbot

Requirements:
- Accurate responses (>85% quality)
- Fast response (P99 < 2s)
- Budget: $10k/month
- Volume: 1000 concurrent users

Analysis:
- Quality requirement → Tier 2 minimum
- Speed requirement → Production model
- Budget constraint → Self-hosted
- Volume → Continuous batching needed

Solution: Llama 70B
83% MMLU → Good quality
120ms latency with batching
$500/month GPU + infra
Can handle 1000 concurrent (with batching)
Fine-tune on domain data to reach 88%+

Alternative if budget cut to $3k/month:
Solution: Mistral 7B + fine-tuning
- Fine-tune on 100 customer conversations
- Reaches 82% accuracy (domain-specific)
- 50ms latency (very fast)
- $300/month GPU
- Tradeoff: Slower improvement iteration, but cost effective

Example 2: Medical Diagnosis Assistant

Requirements:
- High accuracy (>95%)
- Never hallucinate about medical facts
- Works offline (privacy)
- Can run on hospital computer

Analysis:
- Quality: Critical → Tier 1 or specialized
- Hallucination: Critical → Needs RAG or fine-tuning
- Privacy: Offline required → Self-hosted
- Hardware: Limited → Smaller model
- Domain: Medical → Specialized model preferred

Solution: MedPaLM (if available) or BioBERT + fine-tuning
Medical domain knowledge
Trained on medical literature
Lower hallucination in domain
Can run locally
Add RAG with medical knowledge base

Alternative: Llama 70B + RAG
- Fine-tune on medical dataset
- Add vector DB of medical literature
- Verify outputs against knowledge base
- Similar quality to specialized model

Example 3: Real-Time Code Completion

Requirements:
- Latency < 100ms (IDE response)
- Works offline
- 95% Pass@1 on small functions
- Resource-limited laptop

Analysis:
- Latency: Critical → Tier 3
- Offline: Required → Self-hosted
- Quality: High → Tier 2 or specialized
- Hardware: Limited → Small model
- Task: Code → Use Code Llama if possible

Solution: Code Llama 7B (optimized version)
84% Pass@1 (comparable to GPT-4)
Optimized for speed (single GPU inference)
Fits in VRAM on good laptop
Can quantize to 4-bit for smaller footprint
Native code understanding

Alternative if latency too high:
- Distill to 3B model (trade -10% accuracy for 2x speed)
- Use speculative decoding (3x faster)
- Cache common patterns

---

Capability Degradation & Limitations

When Models Fail

# Common failure modes by tier

tier_1_failures = {
 "Reasoning": 2-5, # Very rare
 "Factual errors": 2-3, # Mostly fixed
 "Long reasoning chains": 5, # Can lose thread at 15+ steps
 "Adversarial examples": 10 # But very rare
}

tier_2_failures = {
 "Reasoning": 10-15,
 "Factual errors": 5-8,
 "Long reasoning chains": 20,
 "Adversarial": 15-20
}

tier_3_failures = {
 "Reasoning": 30-40,
 "Factual errors": 10-15,
 "Long reasoning chains": 50, # Often loses track
 "Adversarial": 30-50
}

# Implications:
# - Use Tier 1 for critical reasoning
# - Use Tier 2 for production with validation
# - Use Tier 3 only for well-defined domains
# - Always implement safety checks

Mitigation Strategies

# Handle tier limitations

strategies = {
 "Hallucination": [
 "RAG (retrieve facts from DB)",
 "Fact-checking (verify outputs)",
 "Fine-tuning (learn to be conservative)",
 "Prompt engineering (request sources)"
],

 "Reasoning Failures": [
 "Chain-of-thought prompting",
 "Break into sub-tasks",
 "Add examples (few-shot learning)",
 "Use reasoning models (o1-style)"
],

 "Context Length": [
 "Summarize long docs first",
 "RoPE position interpolation (extend context)",
 "Hierarchical chunking",
 "Summarization + QA approach"
],

 "Speed": [
 "Quantization (4-bit)",
 "Distillation (smaller model)",
 "Speculative decoding",
 "Continuous batching"
]
}

---

Upgrade Path

Start: Tier 3 (cheap, learn)
 ↓
Add: Fine-tuning on domain data
 ↓
If still insufficient quality:
 ↓
Upgrade: Tier 2 (production-grade)
 ↓
Add: Specialized model variant (if exists)
 ↓
If still insufficient:
 ↓
Upgrade: Tier 1 (frontier models)
 ↓
Final: Combine Tier 1 + Tier 2 with ensemble
 (vote on results for maximum confidence)

-

References

• 01 Research Benchmarks - Detailed benchmark information
• 02 Production Metrics - Deployment considerations
• 04 Benchmark Frameworks - Custom evaluation

-

Last Updated: 2026-08-09