Skip to content

Notes

LLM Optimization & Modeling Knowledge Base

LLM Optimization & Modeling Knowledge Base¶

Complete Guide to Large Language Models¶

A comprehensive 55-guide knowledge base covering the complete LLM pipeline from fundamentals through production deployment.

Directory Structure¶

00-Fundamentals ¶

Foundational concepts every LLM engineer should know

Attention Mechanisms - Core attention concepts
Tokenization - Text processing and vocabulary
Tokenization Fundamentals
BPE (Byte Pair Encoding)
SentencePiece
Token Efficiency & Compression
Best Practices

01-Architecture ¶

LLM architectural design and optimization

[01-Core Designs](./01-Architecture/01-Core Designs/index.md)¶

Multi-Query Attention (MQA) & Grouped Query Attention (GQA)
Mixture of Experts (MoE)
Rotary Position Embeddings (RoPE)
Sliding Window Attention

02-Attention Optimization ¶

Flash Attention (v1 & v2)
Kernel Fusion

03-Memory Management ¶

KV Cache
PagedAttention

04-Batching ¶

Continuous Batching
Speculative Decoding
Medusa (Multi-head Decoding)

02-Training ¶

Training efficiency, scaling, and optimization

01-Efficiency ¶

Distributed Training
Gradient Checkpointing
Mixed Precision Training (AMP)

02-Scaling ¶

Scaling Laws & Optimal Allocation

03-Compression ¶

Model Distillation
Pruning & Sparsity
Token Merging (ToMe)

04-Adaptation ¶

LoRA & QLoRA
Adapter Methods Beyond LoRA

05-Alignment ¶

RLHF (Reinforcement Learning from Human Feedback)

Finetuning ¶

Fine-tuning Fundamentals
Instruction Tuning & SFT
DPO (Direct Preference Optimization)
Domain-Specific Fine-tuning
Best Practices

03-Inference ¶

Inference optimization and deployment techniques

01-Optimization ¶

LLM Inference Optimization (complete overview)
Beam Search vs Greedy vs Sampling
Length Extrapolation & Long Context

02-Quantization ¶

Quantization - GPTQ
Quantization - AWQ

Frameworks ¶

Inference Frameworks Fundamentals
vLLM
Framework Comparison & Selection

03-Knowledge Integration ¶

RAG - Retrieval-Augmented Generation
RAG Fundamentals
Vector Databases & Embeddings
Retrieval Strategies
Chunking & Document Preparation
Reranking & Ranking
RAG Integration & Prompt Engineering
Evaluation & Metrics

04-Production ¶

Production deployment, operations, and safety

01-Deployment ¶

Load Balancing & Request Routing
Cost Optimization Strategies

02-Operations ¶

Monitoring & Observability for LLMs
Model Merging & Ensemble Methods
Safety & Alignment
Safety & Alignment Fundamentals
Prompt Injection & Adversarial Examples

05-Benchmarks & Evaluation ¶

Model evaluation, comparison frameworks, and performance metrics

01-Research Benchmarks ¶

Standard Benchmark Suites
Leaderboards & Evaluation Platforms
Benchmark Interpretation & Pitfalls

02-Production Metrics ¶

Inference Performance Metrics
Cost & Efficiency Metrics
User-Facing Quality Metrics

03-Capability Classification ¶

Model Capability Tiers
Specialization Patterns
Task-Specific Performance

04-Benchmark Frameworks ¶

Building Custom Benchmarks
Evaluation Best Practices
Statistical Significance Testing

I'm Learning LLMs¶

Start here → 00-Fundamentals → 01-Architecture → 02-Training → 03-Inference

I'm Training Models¶

Focus on:

01 Efficiency - Distributed training, AMP, checkpointing
02 Scaling - Compute optimization
04 Adaptation - LoRA, QLoRA, efficient fine-tuning

I'm Optimizing Inference¶

Focus on:

01 Optimization - Complete inference stack
Frameworks - vLLM, TensorRT, deployment
03 Memory Management - KV cache, PagedAttention

I'm Deploying to Production¶

Focus on:

Frameworks - Choose deployment framework
01 Deployment - Load balancing, cost optimization
02 Operations - Monitoring, safety, observability

I'm Securing a Model¶

Focus on:

Safety & Alignment
Frameworks - Framework security features
05 Alignment - RLHF alignment training

I'm Building RAG Systems¶

Focus on:

Rag - All RAG topics
04 Adaptation - Fine-tuning for specific tasks

Topics at a Glance¶

Category	Topic Count	Key Files
Fundamentals	6	Attention, Tokenization (5)
Architecture	13	Core designs, Attention optimization, Memory, Batching
Training	17	Efficiency (3), Scaling, Compression, Adaptation, Alignment, Fine-tuning (5)
Inference	18	Optimization, Quantization, Frameworks, RAG (7)
Production	5	Deployment, Operations, Safety & Alignment
Benchmarks	12	Research, Production, Capability Classification, Frameworks
TOTAL	71

Use Wikilinks: Click [File](/folder/file/) to jump between related topics
Breadcrumbs: Each guide links to related topics at the bottom
Search: Use Obsidian search to find topics by keyword
Favorites: Star frequently-visited guides for quick access

Update Log¶

Initial Release: 55 comprehensive guides
Organization: Hierarchical structure for better navigation
Coverage: Complete LLM pipeline from tokenization to production

Last Updated: 2024-08-08