Autograd — The Gradient Engine¶

Overview¶

PyTorch's autograd is reverse-mode automatic differentiation on a tape graph. It does not store gradients — it stores the operations that produced each tensor, then replays them backward to accumulate grad for every tensor with requires_grad=True.

Forward: records the graph as ops execute.
Backward: walks the graph forward-to-root, applying the chain rule.
retain_graph : keeps the graph alive after a backward (for second-order / multiple backward passes).
Detach / no_grad : cuts the graph — the escape hatch for inference and feature extraction.

💡 A tensor's grad is accumulated, not assigned. Forget .zero_grad() and gradients compound.

How the Graph Is Built¶

import torch

a = torch.tensor([2.0], requires_grad=True)
b = a * 3          # grad_fn=<MulBackward0>
c = b.sin()        # grad_fn=<SinBackward0>

print("c.grad_fn:", c.grad_fn)
print("b.grad_fn:", b.grad_fn)
c.backward()
print("a.grad:", a.grad)   # d c / d a = cos(b)*3 = cos(6)*3 ≈ 2.88

The meta-model: fields every autograd tensor carries¶

Tensor
├─ data            # the raw values
├─ requires_grad   # participates in graph?
├─ grad_fn         # the op that made me (None for leaves)
├─ grad            # accumulated gradient after backward
└─ is_leaf         # user-created or detached, or params w/ requires_grad

Chain Rule in One Diagram¶

a ─ [ *3 ] ─> b ─ [ sin ] ─> c ─> loss
   ↑ grad   d(a)=cos(b)·3·d(loss)/dc

backward starts from loss and folds derivatives toward the leaves.

def f(x):
    return torch.sin(x * 3).sum()

x = torch.tensor([0.5, 1.0], requires_grad=True)
f(x).backward()
# grad = cos(3x) * 3
import math
expected = [math.cos(1.5) * 3, math.cos(3.0) * 3]
print([round(g, 4) for g in x.grad.tolist()])
print([round(e, 4) for e in expected])

Accumulation, Zeroing & `inplace`¶

x = torch.tensor([1.0], requires_grad=True)

(2 * x).sum().backward()
(3 * x).sum().backward()      # ACCUMULATES
print("grad after two backwards:", x.grad)   # 2 + 3 = 5

x.grad.zero_()                # or x.grad = None
print("after zero_:", x.grad)

Inplace ops that break the graph¶

Inplace ops on leaf or required tensors can break correctness because the saved value is wiped.

x = torch.tensor([1.0], requires_grad=True)
y = x * 2
# x.add_(1)   # RuntimeError in many versions: "a leaf that requires grad"
try:
    y.sum().backward()
except RuntimeError as e:
    print("inplace broke backprop:", str(e)[:60])

✅ Rule: avoid _ ops on tensors that also participate in loss computation.

Detach, `no_grad`, `enable_grad`¶

`detach()` — a new leaf sharing memory, wiring removed¶

x = torch.randn(4, requires_grad=True)
y = x * 2
z = y.detach()               # shares data, no grad path
print("z.requires_grad:", z.requires_grad)   # False

Context managers¶

x = torch.randn(3, requires_grad=True)

with torch.no_grad():        # disable grad building
    out = (x * 3)            # linear ops produce no grad

with torch.enable_grad():    # re-enable inside no_grad
    g = (x * 3)

torch.set_grad_enabled(False)  # global toggle

`create_graph` & Second-Order Gradients¶

x = torch.tensor([2.0], requires_grad=True)
y = x ** 3

g1 = torch.autograd.grad(y, x, create_graph=True)[0]   # 3x^2 = 12
g2 = torch.autograd.grad(g1, x)[0]                      # 6x  = 12 (d^2 y / dx^2)
print("first:", g1.item(), " second:", g2.item())

# or .backward(create_graph=True) for higher-order

🔗 Higher Order & Jacobians

`grad` vs `backward` — When to Use What¶

Need	Use
Accumulate into `.grad` of every leaf	`loss.backward()`
Grab specific grads, no `.grad` overwrite	`torch.autograd.grad(loss, params)`
Keep graph for higher-order	pass `create_graph=True`
Multiple backwards from same graph	`retain_graph=True`

x = torch.randn(2, requires_grad=True)
y = x.square().sum()

gx = torch.autograd.grad(y, x)          # returns tuple, no .grad mutation
print("grad via autograd.grad:", gx)

y.backward()                            # fills x.grad
print("grad via backward   :", x.grad)

Debugging the Graph¶

x = torch.randn(2, 2, requires_grad=True)
y = (x + 1).relu().flatten()[:, 0]
print(y.grad_fn)
fn = y.grad_fn
while fn is not None:
    print(type(fn).__name__)
    fn = fn.next_functions[0][0]

Tools: torchviz.make_dot(loss) renders the graph; torch.autograd.set_detect_anomaly(True) catches NaN-producing ops.

Key Takeaways¶

Autograd = tape of ops + reverse-mode chain rule; stored graph, not stored grads.
.grad accumulates; always zero it each step.
detach()/no_grad are your tools for inference, frozen branches, and metric tracking.
Inplace ops on required tensors silently corrupt gradients — avoid them.
create_graph=True unlocks Hessians and meta-learning; reserve it.