Skip to content

Tensor Internals— Storage, Views & Strides

Overview

A tensor is not a box of numbers. It is a recipe: a reference to a flat storage (the actual memory) plus a shape and a stride (how to walk that memory). Understanding this split is the difference between shipping code and debugging aliasing bugs at 2am.

  • Storage: the raw 1-D block of memory (.storage(), .data_ptr()).
  • Strides: the tuple telling you how many elements to skip per dimension.
  • View: a tensor sharing the same storage with a different stride recipe.
  • Copy: a tensor with its own storage; changes are isolated.

"View" operations are free (no data movement). "Copy" operations cost memory bandwidth.

The Anatomy of a Tensor

What a view really is

import torch

x = torch.arange(12).reshape(3, 4)
print(x)
print("base storage:", x.storage())
print("data_ptr:", x.data_ptr())
print("shape:", tuple(x.shape))
print("strides:", x.stride()) # (4, 1) -> move +4 elems per row, +1 per col
print("contiguous:", x.is_contiguous())

A contiguous row-major tensor always has stride (size[n-1]*...*size[1],..., 1).

view, transpose, squeeze share storage

y = x.transpose(0, 1) # shares storage with x
print("y strides:", y.stride()) # (1, 4) -- reversed walk
print("stroke:", y.is_contiguous()) # False

# Aliasing
y[0, 0] = 999
print(x[0, 0]) # 999 -- SAME storage!

Broadcast is a zero-copy view

x = torch.zeros(1, 4)
bc, stride = torch.broadcast_tensors(x[None,:,:], torch.zeros(2, 1, 4))
print("new_stride:", bc[0].stride()) # still walks only 4 elems

-

View vs Copy: Know the Family

Operations that return views (share memory):

  • .view(), .reshape() (when possible), .transpose(), .permute(), .t(), .squeeze(), .unsqueeze(), .expand(), slicing x[1:3], .flatten()/.reshape that can reuse layout.

Operations that return copies (new memory):

  • .clone(), .numpy() (returns new buffer), .to() when dtype/device changes, .contiguous() when already non-contiguous, arithmetic ops, .reshape() when it can't share.

Safely detaching shared storage

base = torch.arange(6).reshape(2, 3)
v = base[:,:2] # view
c = base[:,:2].clone() # copy

base[0, 0] = -123
print(v[0, 0]) # -123 (shared)
print(c[0, 0]) # 0 (isolated)

reshape vs view vs contiguous

Call Returns When it copies
.view() always a view raises if incompatible layout
.reshape() view if possible, else copy when layout incompatible
.contiguous() copy if needed, else self when already non-contiguous 
x = torch.arange(12).reshape(3, 4).transpose(0, 1) # non-contiguous

try:
 x.view(2, 6) # ValueError: cannot view this layout
except ValueError as e:
 print("view fails:", e)

print(x.reshape(2, 6).is_contiguous()) # works -> copied, contiguous
print(x.contiguous() is x) # False (copy created)

-

Strides with Slicing, Step & Permute

x = torch.arange(20).reshape(4, 5)

sl = x[::2,::2] # every other row/col
print("slice strides:", sl.stride()) # walks with step 2 -> (10, 2)
print("slice shape:", sl.shape) # (2, 3)

p = x.permute(1, 0)
print("perm strides:", p.stride()) # (1, 5)

Slicing with a step creates non-contiguous tensors that are slow in loops; fix with .contiguous() when it matters.

-

torch.empty, as_strided— Master of Storage

as_strided lets you define an arbitrary view over raw storage. It's how quantized/attention kernels and some libraries build exotic views. Use with extreme care.

storage = torch.arange(9)
twin = torch.as_strided(storage, size=(3, 3), stride=(3, 1))
print(twin)
print("overlaps storage:", twin.storage().data_ptr() == storage.data_ptr())

as_strided bypasses all safety checks; aliasing & overlapping memory here is your responsibility.

Detection & Debugging Snippets

def describe(t: torch.Tensor):
 print(
 f"shape={tuple(t.shape)} strides={t.stride()} "
 f"contig={t.is_contiguous()} ptr={t.data_ptr()} "
 f"storage_nbytes={t.storage().nbytes() if t.storage() else 0}"
)

a = torch.randn(4, 4)
b = a.t()
describe(a) # ptr=0x... shape=(4,4) contig=True
describe(b) # contig=False -- but SAME ptr family

-

Key Takeaways

  • Tensors = storage + strides, not boxes of numbers.
  • view/transpose/slicing = zero-copy; .clone()/arithmetic = new memory.
  • .reshape() copies when the layout forbids a view; .view() raises instead.
  • Aliased storage is the #1 source of silent bugs in hand-rolled layers.
  • Broadcast and expand are free; beware expand sharing a single row's storage.

-