LLM Optimization & Modeling Knowledge Base¶
📚 Complete Guide to Large Language Models¶
A comprehensive 55-guide knowledge base covering the complete LLM pipeline from fundamentals through production deployment.
📂 Directory Structure¶
📖 00-Fundamentals¶
Foundational concepts every LLM engineer should know
- Attention Mechanisms - Core attention concepts
- Tokenization - Text processing and vocabulary
- Tokenization Fundamentals
- BPE (Byte Pair Encoding)
- SentencePiece
- Token Efficiency & Compression
- Best Practices
🏗️ 01-Architecture¶
LLM architectural design and optimization
[01-Core Designs](./01-Architecture/01-Core Designs/)¶
- Multi-Query Attention (MQA) & Grouped Query Attention (GQA)
- Mixture of Experts (MoE)
- Rotary Position Embeddings (RoPE)
- Sliding Window Attention
02-Attention Optimization¶
- Flash Attention (v1 & v2)
- Kernel Fusion
03-Memory Management¶
- KV Cache
- PagedAttention
04-Batching¶
- Continuous Batching
- Speculative Decoding
- Medusa (Multi-head Decoding)
🎓 02-Training¶
Training efficiency, scaling, and optimization
01-Efficiency¶
- Distributed Training
- Gradient Checkpointing
- Mixed Precision Training (AMP)
02-Scaling¶
- Scaling Laws & Optimal Allocation
03-Compression¶
- Model Distillation
- Pruning & Sparsity
- Token Merging (ToMe)
04-Adaptation¶
- LoRA & QLoRA
- Adapter Methods Beyond LoRA
05-Alignment¶
- RLHF (Reinforcement Learning from Human Feedback)
Finetuning¶
- Fine-tuning Fundamentals
- Instruction Tuning & SFT
- DPO (Direct Preference Optimization)
- Domain-Specific Fine-tuning
- Best Practices
💻 03-Inference¶
Inference optimization and deployment techniques
01-Optimization¶
- LLM Inference Optimization (complete overview)
- Beam Search vs Greedy vs Sampling
- Length Extrapolation & Long Context
02-Quantization¶
- Quantization - GPTQ
- Quantization - AWQ
Frameworks¶
- Inference Frameworks Fundamentals
- vLLM
- Framework Comparison & Selection
03-Knowledge Integration¶
- RAG - Retrieval-Augmented Generation
- RAG Fundamentals
- Vector Databases & Embeddings
- Retrieval Strategies
- Chunking & Document Preparation
- Reranking & Ranking
- RAG Integration & Prompt Engineering
- Evaluation & Metrics
🚀 04-Production¶
Production deployment, operations, and safety
01-Deployment¶
- Load Balancing & Request Routing
- Cost Optimization Strategies
02-Operations¶
- Monitoring & Observability for LLMs
- Model Merging & Ensemble Methods
- Safety & Alignment
- Safety & Alignment Fundamentals
- Prompt Injection & Adversarial Examples
📊 05-Benchmarks & Evaluation¶
Model evaluation, comparison frameworks, and performance metrics
01-Research Benchmarks¶
- Standard Benchmark Suites
- Leaderboards & Evaluation Platforms
- Benchmark Interpretation & Pitfalls
02-Production Metrics¶
- Inference Performance Metrics
- Cost & Efficiency Metrics
- User-Facing Quality Metrics
03-Capability Classification¶
- Model Capability Tiers
- Specialization Patterns
- Task-Specific Performance
04-Benchmark Frameworks¶
- Building Custom Benchmarks
- Evaluation Best Practices
- Statistical Significance Testing
🎯 Quick Navigation by Use Case¶
🎓 I'm Learning LLMs¶
Start here → 00-Fundamentals → 01-Architecture → 02-Training → 03-Inference
🏋️ I'm Training Models¶
Focus on:
- 01 Efficiency - Distributed training, AMP, checkpointing
- 02 Scaling - Compute optimization
- 04 Adaptation - LoRA, QLoRA, efficient fine-tuning
⚡ I'm Optimizing Inference¶
Focus on:
- 01 Optimization - Complete inference stack
- Frameworks - vLLM, TensorRT, deployment
- 03 Memory Management - KV cache, PagedAttention
🚀 I'm Deploying to Production¶
Focus on:
- Frameworks - Choose deployment framework
- 01 Deployment - Load balancing, cost optimization
- 02 Operations - Monitoring, safety, observability
🔒 I'm Securing a Model¶
Focus on:
- Safety & Alignment
- Frameworks - Framework security features
- 05 Alignment - RLHF alignment training
🧠 I'm Building RAG Systems¶
Focus on:
- Rag - All RAG topics
- 04 Adaptation - Fine-tuning for specific tasks
📊 Topics at a Glance¶
| Category | Topic Count | Key Files |
|---|---|---|
| Fundamentals | 6 | Attention, Tokenization (5) |
| Architecture | 13 | Core designs, Attention optimization, Memory, Batching |
| Training | 17 | Efficiency (3), Scaling, Compression, Adaptation, Alignment, Fine-tuning (5) |
| Inference | 18 | Optimization, Quantization, Frameworks, RAG (7) |
| Production | 5 | Deployment, Operations, Safety & Alignment |
| Benchmarks | 12 | Research, Production, Capability Classification, Frameworks |
| TOTAL | 71 |
💡 Navigation Tips¶
- Use Wikilinks: Click
[File](/folder/file/)to jump between related topics - Breadcrumbs: Each guide links to related topics at the bottom
- Search: Use Obsidian search to find topics by keyword
- Favorites: Star frequently-visited guides for quick access
🔄 Update Log¶
- Initial Release: 55 comprehensive guides
- Organization: Hierarchical structure for better navigation
- Coverage: Complete LLM pipeline from tokenization to production
Last Updated: 2024-08-08