Skip to content

01-Architecture

Overview

This section covers the architectural components and design patterns that enable efficient LLM inference and training.

Topics

01-Core Designs

Fundamental architectural variations of transformer attention

  • Multi-Query Attention (MQA) & Grouped Query Attention (GQA)
  • 8-32x KV cache reduction
  • Key technique in Llama 2, Mistral

  • Trade-off: Cache efficiency vs quality

  • Mixture of Experts (MoE)

  • Scale to 1T+ parameters
  • Sparse activation (2-4 experts per token)

  • Used in Switch Transformers, Llama MoE

  • Rotary Position Embeddings (RoPE)

  • Better length extrapolation
  • Industry standard (GPT, LLaMA, Mistral)

  • Supports up to 128K+ contexts

  • Sliding Window Attention

  • O(N) attention complexity
  • 4-8x speedup
  • Used in Mistral, Phi models

-

02-Attention Optimization

Efficient computation of attention operations

  • Flash Attention (v1 & v2)
  • 2-4x faster attention
  • Reduces I/O complexity

  • v2: Additional 40% speedup

  • Kernel Fusion

  • Combine multiple operations
  • Reduce GPU memory bandwidth
  • 1.5-3x speedup for element-wise ops

-

03-Memory Management

Optimizing GPU memory usage during inference

  • KV Cache
  • Problem: Quadratic intermediate storage
  • Solutions: Quantization, multi-query attention

  • 10x reduction possible

  • PagedAttention

  • Virtual memory for KV cache
  • 50% memory reduction
  • 2-4x more concurrent users

-

04-Batching

Efficient batch processing techniques

  • Continuous Batching
  • Dynamic request scheduling
  • 2-4x throughput improvement

  • Foundation for modern inference servers

  • Speculative Decoding

  • Parallel prediction verification
  • 2-3x speedup

  • Works with any model

  • Medusa (Multi-head Decoding)

  • Auxiliary prediction heads
  • 2-3x speedup with <1% quality loss
  • Model-agnostic approach

-

Reading Path

For Inference Optimization:

  1. Start with Kv Cache
  2. Understand Flash Attention
  3. Explore Continuous Batching
  4. Combine with Multi Query Attention (Mqa) & Grouped Query Attention (Gqa)

For Architecture Design:

  1. Review 01 Core Designs for modern patterns
  2. Study 02 Attention Optimization for efficiency
  3. Consider memory trade-offs in 03 Memory Management

-