Tensor Internals — Storage, Views & Strides¶

Overview¶

A tensor is not a box of numbers. It is a recipe: a reference to a flat storage (the actual memory) plus a shape and a stride (how to walk that memory). Understanding this split is the difference between shipping code and debugging aliasing bugs at 2am.

Storage: the raw 1-D block of memory (.storage(), .data_ptr()).
Strides: the tuple telling you how many elements to skip per dimension.
View: a tensor sharing the same storage with a different stride recipe.
Copy: a tensor with its own storage; changes are isolated.

💡 "View" operations are free (no data movement). "Copy" operations cost memory bandwidth.

The Anatomy of a Tensor¶

What a view really is¶

import torch

x = torch.arange(12).reshape(3, 4)
print(x)
print("base storage:", x.storage())
print("data_ptr   :", x.data_ptr())
print("shape      :", tuple(x.shape))
print("strides    :", x.stride())   # (4, 1) -> move +4 elems per row, +1 per col
print("contiguous :", x.is_contiguous())

A contiguous row-major tensor always has stride (size[n-1]*...*size[1], ..., 1).

`view`, `transpose`, `squeeze` share storage¶

y = x.transpose(0, 1)   # shares storage with x
print("y strides:", y.stride())   # (1, 4) -- reversed walk
print("stroke   :", y.is_contiguous())  # False

# Aliasing: writing through one view changes the other
y[0, 0] = 999
print(x[0, 0])            # 999 -- SAME storage!

Broadcast is a zero-copy view¶

x = torch.zeros(1, 4)
bc, stride = torch.broadcast_tensors(x[None, :, :], torch.zeros(2, 1, 4))
print("new_stride:", bc[0].stride())  # still walks only 4 elems

View vs Copy: Know the Family¶

Operations that return views (share memory): - .view(), .reshape() (when possible), .transpose(), .permute(), .t(), .squeeze(), .unsqueeze(), .expand(), slicing x[1:3], .flatten()/.reshape that can reuse layout.

Operations that return copies (new memory): - .clone(), .numpy() (returns new buffer), .to() when dtype/device changes, .contiguous() when already non-contiguous, arithmetic ops, .reshape() when it can't share.

Safely detaching shared storage¶

base = torch.arange(6).reshape(2, 3)
v = base[:, :2]          # view
c = base[:, :2].clone()  # copy

base[0, 0] = -123
print(v[0, 0])  # -123 (shared)
print(c[0, 0])  # 0    (isolated)

`reshape` vs `view` vs `contiguous`¶

Call	Returns	When it copies
`.view()`	always a view	raises if incompatible layout
`.reshape()`	view if possible, else copy	when layout incompatible
`.contiguous()`	copy if needed, else self	when already non-contiguous

x = torch.arange(12).reshape(3, 4).transpose(0, 1)  # non-contiguous

try:
    x.view(2, 6)          # ValueError: cannot view this layout
except ValueError as e:
    print("view fails:", e)

print(x.reshape(2, 6).is_contiguous())   # works -> copied, contiguous
print(x.contiguous() is x)               # False (copy created)

Strides with Slicing, Step & Permute¶

x = torch.arange(20).reshape(4, 5)

sl = x[::2, ::2]          # every other row/col
print("slice strides:", sl.stride())   # walks with step 2 -> (10, 2)
print("slice shape  :", sl.shape)      # (2, 3)

p = x.permute(1, 0)
print("perm strides :", p.stride())    # (1, 5)

⚠️ Slicing with a step creates non-contiguous tensors that are slow in loops; fix with .contiguous() when it matters.

`torch.empty`, `as_strided` — Master of Storage¶

as_strided lets you define an arbitrary view over raw storage. It's how quantized/attention kernels and some libraries build exotic views. Use with extreme care.

storage = torch.arange(9)
twin = torch.as_strided(storage, size=(3, 3), stride=(3, 1))
print(twin)
print("overlaps storage:", twin.storage().data_ptr() == storage.data_ptr())

⚠️ as_strided bypasses all safety checks; aliasing & overlapping memory here is your responsibility.

Detection & Debugging Snippets¶

def describe(t: torch.Tensor):
    print(
        f"shape={tuple(t.shape)}  strides={t.stride()}  "
        f"contig={t.is_contiguous()}  ptr={t.data_ptr()}  "
        f"storage_nbytes={t.storage().nbytes() if t.storage() else 0}"
    )

a = torch.randn(4, 4)
b = a.t()
describe(a)   # ptr=0x... shape=(4,4) contig=True
describe(b)   # contig=False -- but SAME ptr family

Key Takeaways¶

Tensors = storage + strides, not boxes of numbers.
view/transpose/slicing = zero-copy; .clone()/arithmetic = new memory.
.reshape() copies when the layout forbids a view; .view() raises instead.
Aliased storage is the #1 source of silent bugs in hand-rolled layers.
Broadcast and expand are free; beware expand sharing a single row's storage.