Capability Classification: Understanding Model Tiers & Specialization¶
Overview¶
Not all models are created equal. Understanding model capabilities—both their strengths and limitations—is crucial for:
- Selecting the right model for your use case
- Making cost-performance tradeoffs
- Fine-tuning efficiently
- Avoiding costly mistakes
Model Capability Tiers (2024)¶
Tier 1: Frontier Models¶
Examples: GPT-4, Claude 3 Opus, Llama 3 405B, Gemini Ultra
Performance Level: State-of-the-art
Cost: $$$ (Expensive)
Speed: Moderate
Context: 32K - 128K tokens
Specialization: General purpose
Characteristics:
✓ Best performance on all benchmarks
✓ Excellent reasoning capabilities
✓ Strong instruction following
✓ Few hallucinations (2-3%)
✓ Multilingual support
✓ Long-context understanding
❌ Very expensive ($0.01-0.10 per 1K tokens)
❌ Often API-only (no weights)
❌ Slower inference
❌ Rate-limited
Use When:
- Reasoning is critical
- Quality > cost
- Enterprise customers
- Complex problem solving
Examples:
- GPT-4: $0.03 input, $0.06 output per 1K tokens
- Claude 3 Opus: $0.015 input, $0.075 output
- Llama 3 405B: Free (self-hosted) or ~$1-2 per 1M tokens (API)
MMLU: 84-93%
HumanEval: 90-94%
GSM8K: 93-96%
Tier 2: Production Models¶
Examples: Llama 70B, Mixtral 8x22B, Claude 3 Sonnet, Gemini Pro
Performance Level: Strong, production-ready
Cost: $$ (Moderate)
Speed: Fast
Context: 4K - 32K tokens
Specialization: Both general + specialized variants
Characteristics:
✓ Excellent performance (80%+ MMLU)
✓ Fast inference
✓ Reasonable cost
✓ Available as open weights
✓ Good for fine-tuning
✓ Production-grade quality
⚠ Slightly higher hallucination (4-6%)
⚠ Some reasoning limitations
⚠ Context window limitations (some)
⚠ May need domain fine-tuning
Use When:
- Cost-performance matters
- Latency is important
- Need to fine-tune
- Production deployment
- Scale matters
Examples:
- Llama 70B: ~$0.59 per 1M tokens (self-hosted)
- Claude 3 Sonnet: ~$0.003 per 1K tokens
- Mixtral 8x22B: $0.27 per 1M tokens (self-hosted)
MMLU: 78-87%
HumanEval: 75-90%
GSM8K: 50-85%
Tier 3: Efficient Models¶
Examples: Mistral 7B, Llama 13B, Llama 7B, Phi, Qwen
Performance Level: Capable for constrained scenarios
Cost: $ (Cheap)
Speed: Very fast
Context: 4K - 32K tokens
Specialization: Specialized (code, math, etc.)
Characteristics:
✓ Very fast inference (10-50ms per token)
✓ Cheap or free
✓ Fits on consumer GPU
✓ Can run on edge devices
✓ Easy to fine-tune
✓ Large community support
❌ Lower absolute performance
❌ Weaker reasoning
❌ Higher hallucination (8-15%)
❌ Needs more hand-holding
Use When:
- Speed is critical
- Edge/mobile deployment
- Budget is constrained
- Task is specialized (with fine-tuning)
- User has limited compute
Examples:
- Mistral 7B: Free (self-hosted)
- Llama 13B: Free (self-hosted), works on 24GB GPU
- Phi 3.5: Free, fits on 8GB VRAM
MMLU: 45-72%
HumanEval: 50-73%
GSM8K: 25-60%
Tier 4: Specialized/Distilled Models¶
Examples: Code Llama 7B, Medical LLMs, DistilBERT
Performance Level: Excellent in narrow domain
Cost: $ (Free - Cheap)
Speed: Fastest
Context: 2K - 8K tokens
Specialization: Single domain (code, medical, legal)
Characteristics:
✓ Best in specific domain
✓ Extremely fast
✓ Tiny (7B or smaller)
✓ Minimal hallucinations in domain
✓ Energy efficient
❌ Poor outside domain
❌ Limited general knowledge
❌ Can't do multi-task
Use When:
- Single well-defined task
- Domain expertise needed
- Latency critical
- Energy limited (mobile)
- Inference at edge
Examples:
- Code Llama 7B: 84% HumanEval (vs Llama 7B: 33%)
- BloombergGPT (Finance): 97% on finance QA
- BioBERT (Biology): 95% on biomedical NER
Performance:
- Excels in domain (+30-50% vs general models)
- Struggles outside (+30-50% worse)
Capability Matrix¶
By Task Type¶
Task Type Tier 1 Tier 2 Tier 3 Tier 4
──────────────────────────────────────────────────
General QA ✓✓ ✓ ✓ ❌
Reasoning ✓✓ ✓ ⚠ ❌
Math ✓✓ ✓ ⚠ ❌
Code Generation ✓✓ ✓✓ ✓ ✓✓ (specialized)
Writing ✓✓ ✓ ✓ ❌
Translation ✓ ✓ ⚠ ❌
Classification ✓ ✓ ✓ ✓✓ (domain)
NER/Tagging ✓ ✓ ✓ ✓✓ (domain)
Summarization ✓ ✓ ⚠ ❌
Long Document ✓ ⚠ ❌ ❌
Common Sense ✓✓ ✓ ⚠ ❌
✓✓ = Excellent
✓ = Good
⚠ = Acceptable (may need tuning)
❌ = Poor
By Constraint¶
Constraint Priority Best Tier
──────────────────────────────────────────────
Maximum Quality 1. Quality Tier 1 (Frontier)
2. Cost ignored
Balance 1. Performance Tier 2 (Production)
2. Cost
Cost Critical 1. Cost Tier 3 (Efficient)
2. Acceptable quality
Speed Critical 1. Latency Tier 3/4
2. Throughput (Efficient/Specialized)
Domain Specific 1. Domain perf Tier 4 (Specialized)
2. General perf + fine-tuning Tier 3
Edge/Mobile 1. Size < 10GB Tier 3/4
2. Speed
Performance Trade-offs¶
Frontier vs Production vs Efficient¶
# Scenario: Customer support chatbot
# Requirements: 1000 concurrent users, <2s response time
# Option 1: GPT-4 (Frontier)
option_1 = {
"MMLU": 86.5,
"Latency P99": 150, # ms (very fast, API)
"Cost per request": 0.006,
"Monthly cost": 6000 * 1000 * 30 = 180_000, # Assuming 100 users/day
"Quality": "Best-in-class",
"Hallucination": "2%"
}
# Option 2: Llama 70B (Production)
option_2 = {
"MMLU": 82.9,
"Latency P99": 200, # ms (self-hosted, batch)
"Cost per request": 0.0006,
"Monthly cost": 5000, # GPU + infrastructure
"Quality": "Excellent",
"Hallucination": "5%"
}
# Option 3: Mistral 7B + Fine-tuning (Efficient)
option_3 = {
"MMLU": 60, # But fine-tuned on domain data: +15%
"Domain MMLU": 75,
"Latency P99": 50, # Very fast
"Cost per request": 0.000001,
"Monthly cost": 500, # Smaller GPU
"Quality": "Domain-specific (excellent)",
"Hallucination": "8% overall, 2% in domain"
}
# Analysis
comparison = {
"Quality Ranking": ["GPT-4", "Llama 70B", "Mistral 7B+FT"],
"Cost Ranking": ["Mistral 7B+FT", "Llama 70B", "GPT-4"],
"Speed Ranking": ["Mistral 7B+FT", "GPT-4 API", "Llama 70B"],
"Recommendation": {
"If budget > $100k/month": "GPT-4 (API)",
"If budget = $5-20k/month": "Llama 70B (self-hosted)",
"If budget < $5k/month": "Mistral 7B + fine-tuning"
}
}
Specialization Dimensions¶
By Language¶
Language Focus Best Model Performance
────────────────────────────────────────────────────
English Only GPT-4, Claude, Llama SOTA
Multilingual mT5, XLM-R, Gemini Strong (40+ langs)
Chinese Optimized ChatGLM3, Qwen Near-English SOTA
Arabic Optimized Jais, AraLLaMA Better than general
Japanese Swallow, Sakura Native quality
By Domain¶
Domain Best Model Specialization
──────────────────────────────────────────────────────────
General Purpose GPT-4, Claude, Llama No specialization
Code Generation GPT-4, Code Llama +30-40% accuracy
Mathematics Claude 3 Opus, GPT-4 Strong CoT
Medical MedPaLM, BioLLaMA Domain knowledge
Finance BloombergGPT Domain data
Legal LegalBERT, JurisCA Legal terminology
Scientific SciBERT, ArxivGPT Paper understanding
By Context Window¶
Context Needed Best Model Max Context
─────────────────────────────────────────────────
Short (< 4K) Most models 4K - unlimited
Medium (4-32K) Llama 70B, Claude 32K - 128K
Long (32-128K) GPT-4, Claude 3, GPT 128K - 200K
Ultra (> 128K) GPT-4 Turbo Up to 200K
Capability Combination Matrix¶
English | Chinese | Code | Math | Domain
────────────────────────────────────────────────────────
GPT-4 ✓✓ | ✓ | ✓✓ | ✓✓ | Moderate
Claude 3 ✓✓ | ✓ | ✓ | ✓✓ | Moderate
Llama 70B ✓✓ | ✓ | ✓ | ✓ | Low
ChatGLM3 ✓ | ✓✓ | ✓ | ⚠ | Moderate
Code Llama 7B ✓ | ✗ | ✓✓ | ⚠ | Code Only
Mistral 7B ✓ | ⚠ | ✓ | ⚠ | Low
BloombergGPT ✓ | ⚠ | ⚠ | ⚠ | Finance
Selection Framework¶
Decision Tree¶
graph TD
A["Need LLM for Task?"] --> B{What's most<br/>important?}
B -->|Quality| C{Budget > $100k/month?}
C -->|Yes| D["GPT-4 / Claude Opus"]
C -->|No| E["Llama 70B<br/>self-hosted"]
B -->|Cost| F{Need<br/>fine-tuning?}
F -->|Yes| G["Mistral 7B<br/>+ LoRA"]
F -->|No| H["Free Open Models<br/>Llama 7B/13B"]
B -->|Speed| I{Latency<br/>critical?}
I -->|< 100ms| J["Mistral 7B"]
I -->|< 500ms| K["Llama 13B"]
B -->|Domain| L{Specialized<br/>model exists?}
L -->|Yes| M["Use specialized<br/>+ fine-tune if needed"]
L -->|No| N["Fine-tune<br/>general model"]
style D fill:#99ff99
style E fill:#ffff99
style G fill:#ffff99
style H fill:#99ff99
style J fill:#99ff99
style K fill:#ffff99
style M fill:#99ff99
style N fill:#ffff99Benchmarks by Tier¶
Raw Benchmark Scores¶
Metric Tier 1 Tier 2 Tier 3 Tier 4
────────────────────────────────────────────────────────────
MMLU 84-93% 78-87% 45-72% Varies
HumanEval 90-94% 75-90% 50-73% Domain-specific
GSM8K 93-96% 50-85% 25-60% Low (unless math)
HellaSwag 97% 92% 78% Varies
LongBench 82-88% 65-78% 40-55% Low
NER 92% 88% 82% 96%+ (domain)
Sentiment 93% 89% 85% 91% (domain)
What These Differences Mean¶
Tier 1 vs Tier 2 vs Tier 3 (MMLU: 87% vs 82% vs 60%):
Scenario: Answer 100 customer questions
Tier 1: 87 correct answers
Tier 2: 82 correct answers (5% error rate)
Tier 3: 60 correct answers (27% error rate)
Impact on Business:
- Tier 3 needs extensive fine-tuning or post-processing
- Tier 2 acceptable for most production
- Tier 1 best for complex reasoning
With Domain Fine-tuning:
Tier 3 after FT: 75%+ (competitive with Tier 2!)
Tier 3 after extensive FT: 82%+ (approaching Tier 2!)
Key Insight:
Fine-tuning can bridge tiers for specific domains
Real-World Examples¶
Example 1: Customer Support Chatbot¶
Requirements:
- Accurate responses (>85% quality)
- Fast response (P99 < 2s)
- Budget: $10k/month
- Volume: 1000 concurrent users
Analysis:
- Quality requirement → Tier 2 minimum
- Speed requirement → Production model
- Budget constraint → Self-hosted
- Volume → Continuous batching needed
Solution: Llama 70B
✓ 83% MMLU → Good quality
✓ 120ms latency with batching
✓ $500/month GPU + infra
✓ Can handle 1000 concurrent (with batching)
✓ Fine-tune on domain data to reach 88%+
Alternative if budget cut to $3k/month:
Solution: Mistral 7B + fine-tuning
- Fine-tune on 100 customer conversations
- Reaches 82% accuracy (domain-specific)
- 50ms latency (very fast)
- $300/month GPU
- Tradeoff: Slower improvement iteration, but cost effective
Example 2: Medical Diagnosis Assistant¶
Requirements:
- High accuracy (>95%)
- Never hallucinate about medical facts
- Works offline (privacy)
- Can run on hospital computer
Analysis:
- Quality: Critical → Tier 1 or specialized
- Hallucination: Critical → Needs RAG or fine-tuning
- Privacy: Offline required → Self-hosted
- Hardware: Limited → Smaller model
- Domain: Medical → Specialized model preferred
Solution: MedPaLM (if available) or BioBERT + fine-tuning
✓ Medical domain knowledge
✓ Trained on medical literature
✓ Lower hallucination in domain
✓ Can run locally
✓ Add RAG with medical knowledge base
Alternative: Llama 70B + RAG
- Fine-tune on medical dataset
- Add vector DB of medical literature
- Verify outputs against knowledge base
- Similar quality to specialized model
Example 3: Real-Time Code Completion¶
Requirements:
- Latency < 100ms (IDE response)
- Works offline
- 95% Pass@1 on small functions
- Resource-limited laptop
Analysis:
- Latency: Critical → Tier 3/4
- Offline: Required → Self-hosted
- Quality: High → Tier 2 or specialized
- Hardware: Limited → Small model
- Task: Code → Use Code Llama if possible
Solution: Code Llama 7B (optimized version)
✓ 84% Pass@1 (comparable to GPT-4)
✓ Optimized for speed (single GPU inference)
✓ Fits in VRAM on good laptop
✓ Can quantize to 4-bit for smaller footprint
✓ Native code understanding
Alternative if latency too high:
- Distill to 3B model (trade -10% accuracy for 2x speed)
- Use speculative decoding (3x faster)
- Cache common patterns
Capability Degradation & Limitations¶
When Models Fail¶
# Common failure modes by tier
tier_1_failures = {
"Reasoning": 2-5, # Very rare
"Factual errors": 2-3, # Mostly fixed
"Long reasoning chains": 5, # Can lose thread at 15+ steps
"Adversarial examples": 10 # But very rare
}
tier_2_failures = {
"Reasoning": 10-15,
"Factual errors": 5-8,
"Long reasoning chains": 20,
"Adversarial": 15-20
}
tier_3_failures = {
"Reasoning": 30-40,
"Factual errors": 10-15,
"Long reasoning chains": 50, # Often loses track
"Adversarial": 30-50
}
# Implications:
# - Use Tier 1 for critical reasoning
# - Use Tier 2 for production with validation
# - Use Tier 3 only for well-defined domains
# - Always implement safety checks
Mitigation Strategies¶
# Handle tier limitations
strategies = {
"Hallucination": [
"RAG (retrieve facts from DB)",
"Fact-checking (verify outputs)",
"Fine-tuning (learn to be conservative)",
"Prompt engineering (request sources)"
],
"Reasoning Failures": [
"Chain-of-thought prompting",
"Break into sub-tasks",
"Add examples (few-shot learning)",
"Use reasoning models (o1-style)"
],
"Context Length": [
"Summarize long docs first",
"RoPE position interpolation (extend context)",
"Hierarchical chunking",
"Summarization + QA approach"
],
"Speed": [
"Quantization (4-bit)",
"Distillation (smaller model)",
"Speculative decoding",
"Continuous batching"
]
}
Upgrade Path¶
Start: Tier 3 (cheap, learn)
↓
Add: Fine-tuning on domain data
↓
If still insufficient quality:
↓
Upgrade: Tier 2 (production-grade)
↓
Add: Specialized model variant (if exists)
↓
If still insufficient:
↓
Upgrade: Tier 1 (frontier models)
↓
Final: Combine Tier 1 + Tier 2 with ensemble
(vote on results for maximum confidence)
References¶
- 01 Research Benchmarks - Detailed benchmark information
- 02 Production Metrics - Deployment considerations
- 04 Benchmark Frameworks - Custom evaluation
Last Updated: 2026-08-09