Skip to content

C++ & libtorch Integration

Overview

The ultimate deployment: no Python in the loop. libtorch (the C++ front-end of PyTorch) loads your exported model and runs it directly in C++ services, edge devices, or performance-critical code. With AOTInductor you can even compile to a dependency-free .so.

  • Load via TorchScript (torch::jit::load)— classic path, still widely used.
  • Load via torch.export → AOTInductor— modern path; produces a self-contained shared library.
  • C++ API mirrors Python: torch::tensor, model.forward({x}), torch::no_grad.
  • Link with Torch_DIR, CMake find_package(Torch).

Prefer AOTInductor for new products: no runtime interpreter, tiny binary, fused kernels. TorchScript remains for legacy/ROCm/older stacks.

-

The C++ Side— Minimal Inference

#include <torch/script.h>
#include <torch/torch.h> // if using torch:: API
#include <iostream>

int main() {
 // load TorchScript model (exported with torch.jit.save)
 torch::jit::script::Module model = torch::jit::load("model.pt");
 model.eval();

 torch::NoGradGuard no_grad;
 auto x = torch::randn({1, 8});
 auto out = model.forward({x}).toTensor();
 std::cout << "out " << out.sizes() << "\n";
 return 0;
}

Exporting From Python for C++

import torch, torch.nn as nn

model = nn.Linear(8, 4).eval()
with torch.no_grad():
 mod = torch.jit.trace(model, torch.randn(1, 8))
mod.save("model.pt")
print("saved TorchScript module")

# modern alternative
lib = torch._export.aot_compile(model, (torch.randn(1, 8),))
print(lib) # path to compiled.so

-

CMake Build

cmake_minimum_required(VERSION 3.18)
project(serve_app)

find_package(Torch REQUIRED) # set Torch_DIR to libtorch

add_executable(server main.cpp)
target_link_libraries(server "${TORCH_LIBRARIES}")
target_compile_features(server PRIVATE cxx_std_17)

export Torch_DIR=$(python -c "import torch; print(torch.utils.cmake_prefix_path)")
mkdir build && cd build
cmake -DTorch_DIR=$Torch_DIR..
make
./server

-

AOTInductor— Python-Free Inference

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(128, 128), nn.GELU(), nn.Linear(128, 10)).eval()
code, lib_path = torch._export.aot_compile(model, (torch.randn(1, 128),))
print(lib_path) # e.g. /tmp/.../model.so

// dlopen the.so and call __torch_... entry points — no torch script runtime needed
#include <dlfcn.h>
// entry: __top_level_model_forward(...) generated in export headers

AOTInductor requires torch >= 2.3-ish and cares about shapes/dtypes (match exactly).

-

Pitfalls Going C++

  1. No autograd: everything under NoGradGuard; C++ path is inference only.
  2. Shape/device mismatches crash loudly— validate inputs before forward.
  3. Control flow in script: exported code must be traceable; avoid data-dependent branches.
  4. Thresholds: AOTInductor hates dynamic shapes— fix batch at compile time or export ONNX RT.
  5. Linking bloat: TORCH_LIBRARIES drags CUDA deps; link only what you need for CPU-only edge.

-

When C++/libtorch Wins vs Loses

Scenario Verdict
High-frequency inference (per-ms) C++ strong win
Edge/embedded, no python runtime C++ (or AOT.so)
Rapid iteration / research protos Python + export
Heavy framework orchestration keep Python, call torch ops

-

Key Takeaways

  • torch::jit::load + NoGradGuard + forward({x}).toTensor() = the core C++ loop.
  • AOTInductor turns torch.export output into a standalone .so— the modern prod path.
  • CMake: find_package(Torch) + link ${TORCH_LIBRARIES}.
  • C++ is inference-only; validate shapes/dtypes and stick to exported graphs.
  • Prefer AOT/ONNX over TorchScript for new builds; keep JIT for legacy.

-