Skip to content

Foundations & Tensor Mastery

Overview

PyTorch's power and its quirks both come from how it models data: tensors are views over blocks of memory, and gradients are specifications (graphs), not computed values. This chapter builds that mental model.

Topics

  • Tensor Internals — Storage, Views & Strides - The data is not where you look; a view is a stride recipe over a storage.
  • Autograd — The Gradient Engine - Gradients are described by a graph and diffed with reverse-mode AD.
  • Device & Memory Management - The caching allocator, pinning, transfers, and non_blocking.
  • Precision & Numerics - dtypes, fp16/bf16/fp8 ranges, and why tiny numbers ruin gradients.

Key Patterns

  • Distinguish view (shares storage) from copy (new storage) to avoid hidden aliasing bugs.
  • Understand requires_grad as a dataflow annotation on the tensor graph.
  • The CUDA caching allocator makes memory usage look "leaky"; cache size is real.
  • Precision is a range × precision trade-off, not a binary.