Skip to content

Autograd— The Gradient Engine

Overview

PyTorch's autograd is reverse-mode automatic differentiation on a tape graph. It does not store gradients— it stores the operations that produced each tensor, then replays them backward to accumulate grad for every tensor with requires_grad=True.

  • Forward: records the graph as ops execute.
  • Backward: walks the graph forward-to-root, applying the chain rule.
  • retain_graph: keeps the graph alive after a backward (for second-order / multiple backward passes).
  • Detach / no_grad: cuts the graph— the escape hatch for inference and feature extraction.

A tensor's grad is accumulated, not assigned. Forget .zero_grad() and gradients compound.

-

How the Graph Is Built

import torch

a = torch.tensor([2.0], requires_grad=True)
b = a * 3 # grad_fn=<MulBackward0>
c = b.sin() # grad_fn=<SinBackward0>

print("c.grad_fn:", c.grad_fn)
print("b.grad_fn:", b.grad_fn)
c.backward()
print("a.grad:", a.grad) # d c / d a = cos(b)*3 = cos(6)*3 ≈ 2.88

The meta-model: fields every autograd tensor carries

graph TD
 A["Tensor"]
 A --> B["data<br/>the raw values"]
 A --> C["requires_grad<br/>participates in graph?"]
 A --> D["grad_fn<br/>the op that made me<br/>None for leaves"]
 A --> E["grad<br/>accumulated gradient<br/>after backward"]
 A --> F["is_leaf<br/>user-created or detached<br/>or params w/ requires_grad"]

-

Chain Rule in One Diagram

a ─ [ *3] ─> b ─ [ sin] ─> c ─> loss
 ↑ grad d(a)=cos(b)·3·d(loss)/dc

backward starts from loss and folds derivatives toward the leaves.

def f(x):
 return torch.sin(x * 3).sum()

x = torch.tensor([0.5, 1.0], requires_grad=True)
f(x).backward()
# grad = cos(3x) * 3
import math
expected = [math.cos(1.5) * 3, math.cos(3.0) * 3]
print([round(g, 4) for g in x.grad.tolist()])
print([round(e, 4) for e in expected])

Accumulation, Zeroing & inplace

x = torch.tensor([1.0], requires_grad=True)

(2 * x).sum().backward()
(3 * x).sum().backward() # ACCUMULATES
print("grad after two backwards:", x.grad) # 2 + 3 = 5

x.grad.zero_() # or x.grad = None
print("after zero_:", x.grad)

Inplace ops that break the graph

Inplace ops on leaf or required tensors can break correctness because the saved value is wiped.

x = torch.tensor([1.0], requires_grad=True)
y = x * 2
# x.add_(1) # RuntimeError in many versions
try:
 y.sum().backward()
except RuntimeError as e:
 print("inplace broke backprop:", str(e)[:60])

Rule: avoid _ ops on tensors that also participate in loss computation.

Detach, no_grad, enable_grad

detach()— a new leaf sharing memory, wiring removed

x = torch.randn(4, requires_grad=True)
y = x * 2
z = y.detach() # shares data, no grad path
print("z.requires_grad:", z.requires_grad) # False

Context managers

x = torch.randn(3, requires_grad=True)

with torch.no_grad(): # disable grad building
 out = (x * 3) # linear ops produce no grad

with torch.enable_grad(): # re-enable inside no_grad
 g = (x * 3)

torch.set_grad_enabled(False) # global toggle

-

create_graph & Second-Order Gradients

x = torch.tensor([2.0], requires_grad=True)
y = x ** 3

g1 = torch.autograd.grad(y, x, create_graph=True)[0] # 3x^2 = 12
g2 = torch.autograd.grad(g1, x)[0] # 6x = 12 (d^2 y / dx^2)
print("first:", g1.item(), " second:", g2.item())

# or.backward(create_graph=True) for higher-order

[Higher Order & Jacobians](/06-pytorch/03-custom-autograd-and-model-transformation/(02-higher-order-gradients-jacobians/)

grad vs backward— When to Use What

Need Use
Accumulate into .grad of every leaf loss.backward()
Grab specific grads, no .grad overwrite torch.autograd.grad(loss, params)
Keep graph for higher-order pass create_graph=True
Multiple backwards from same graph retain_graph=True 
x = torch.randn(2, requires_grad=True)
y = x.square().sum()

gx = torch.autograd.grad(y, x) # returns tuple, no.grad mutation
print("grad via autograd.grad:", gx)

y.backward() # fills x.grad
print("grad via backward:", x.grad)

Debugging the Graph

x = torch.randn(2, 2, requires_grad=True)
y = (x + 1).relu().flatten()[:, 0]
print(y.grad_fn)
fn = y.grad_fn
while fn is not None:
 print(type(fn).__name__)
 fn = fn.next_functions[0][0]

Tools: torchviz.make_dot(loss) renders the graph; torch.autograd.set_detect_anomaly(True) catches NaN-producing ops.

-

Key Takeaways

  • Autograd = tape of ops + reverse-mode chain rule; stored graph, not stored grads.
  • .grad accumulates; always zero it each step.
  • detach()/no_grad are your tools for inference, frozen branches, and metric tracking.
  • Inplace ops on required tensors silently corrupt gradients— avoid them.
  • create_graph=True unlocks Hessians and meta-learning; reserve it.

-