Skip to content

Performance & Compilation

Overview

Eager PyTorch pays a Python interpreter tax per operation: many small kernels, no global view, no fusion. torch.compile (Dynamo + Inductor) removes most of that overhead by capturing a graph, applying graph-level optimizations, and generating fused kernels. Combined with CUDA graphs, good memory layouts, and disciplined profiling, 1.5-3x training speedups are routine.

Topics

  • torch.compile Deep Dive - How Dynamo + Inductor work; modes, backends, when it helps vs hurts.
  • CUDA Graphs - Capture a deterministic stream of kernels and replay it, removing launch overhead.
  • Memory Formats & Layouts - channels_last, blocksparse, layout-aware kernels.
  • Profiling & Benchmarking - torch.profiler, CUPTI, nsys/ncu, and the "measure first" discipline.
  • Kernel Fusion at the PyTorch Level - What Inductor fuses and how to help it.

Key Patterns

  • torch.compile(model) is the default first lever: minimal code change, large win.
  • Fusion removes launch + memory movement, not arithmetic.
  • Profile with real kernels before hand-optimizing.
  • [03 Device & Memory Management](/06-pytorch/01-foundations-and-tensor-mastery/(03-device-memory-management/)
  • 00 Readme