PyTorch Deep Learning Core Interview Preparation Guide

PyTorch uses define-by-run semantics where the computation graph is constructed on-the-fly during the forward pass based on actual execution paths. This allows dynamic control flow such as loops and conditional statements to change the network architecture per batch without recompilation. Static graph frameworks, by contrast, require defining the entire graph structure upfront before execution, which facilitates heavy upfront compiler optimizations but complicates debugging and dynamic architectures. In PyTorch 2.x, TorchDynamo bridges this gap by capturing dynamic graphs JIT-style from Python bytecode while retaining Python's imperative flexibility.

OOM errors in PyTorch are frequently caused by memory fragmentation within the c10 Caching Allocator or uncollected computation graphs. When tensors of varying sizes are allocated and freed repeatedly, free memory becomes fragmented into non-contiguous blocks that cannot satisfy large new allocation requests, even if the sum of free blocks is high. Additionally, retaining references to tensors involved in the forward pass keeps their associated autograd graph nodes and intermediate activations alive in VRAM. Developers can diagnose this using torch.cuda.memory_summary() and resolve fragmentation by tuning batch sizes or avoiding unnecessary intermediate tensor retention.

You implement a custom backward pass by creating a subclass of torch.autograd.Function and defining static forward and backward methods. In the forward method, you perform computations and save necessary tensors for the backward pass using ctx.save_for_backward(). In the backward method, you receive incoming gradients from downstream layers, compute the Jacobian-vector product analytically using the saved tensors, and return gradients corresponding to each input of the forward method. This pattern is essential for custom CUDA kernels or numerical stability adjustments.

torch.no_grad() is a context manager that disables the autograd engine, preventing the creation of computation graphs and stopping gradient tracking, which drastically reduces memory consumption and compute overhead during inference or evaluation. model.eval(), on the other hand, is a module-level method that switches specific layers—such as BatchNorm and Dropout—into evaluation mode, ensuring that running statistics are not updated and dropout masks are disabled. Both are required simultaneously for correct, efficient model inference in production.

PyTorch accumulates gradients by default (adding new gradients to existing .grad attributes rather than overwriting them) to support gradient accumulation across multiple micro-batches, which enables training models with effective batch sizes larger than what fits in physical GPU memory. Because of this behavior, developers must explicitly call optimizer.zero_grad() at the beginning of every training iteration. Forgetting this step causes gradients from previous steps to accumulate, leading to corrupted weight updates, exploding loss, and immediate training divergence.

View operations in PyTorch create a new tensor object that shares the exact same underlying physical data storage buffer (c10::Storage) as the original tensor, but modifies the view metadata—specifically its sizes, strides, and storage offsets. This enables zero-copy tensor reshaping and transposition, executing in O(1) time without allocating new memory or copying data. However, operations like transpose() can result in non-contiguous tensors, which cause memory access errors if passed directly into certain low-level C++ or CUDA kernels without calling .contiguous().

TorchDynamo hooks into CPython's frame evaluation API to inspect Python bytecode as it executes. Instead of interpreting bytecode line by line, it symbolically executes the code to trace operations and extract static computation graphs, generating Guard objects that ensure safety when inputs change. If unsupported Python constructs or dynamic control flow changes are encountered, TorchDynamo triggers a graph break, falling back to standard Python interpreter execution for that portion of code while successfully compiling the valid graph regions.

NCCL (NVIDIA Collective Communications Library) provides high-performance inter-GPU and inter-node communication primitives tailored for NVIDIA hardware. In PyTorch DDP, NCCL executes collective operations like AllReduce and Broadcast to synchronize gradients across multiple GPUs during the backward pass. By leveraging NVLink, NVSwitch, and InfiniBand fabrics, NCCL maximizes network bandwidth utilization and enables seamless overlapping of gradient communication with ongoing backward pass computations.

torch.load relies on Python's built-in pickle module by default to deserialize checkpoint files. Because pickle can execute arbitrary code during unpickling, loading an untrusted model checkpoint file poses a severe remote code execution (RCE) security vulnerability. Setting weights_only=True restricts the unpickling process strictly to safe tensor data structures and basic numeric types, neutralizing malicious payload execution while successfully restoring model weights.

A graph break occurs when TorchDynamo encounters Python code, data structures, or external library calls that it cannot symbolically trace into the compiled computation graph. Common causes include printing tensor values, calling non-tensor Python functions, or using dynamic dictionary keys. Graph breaks force the runtime to switch back and forth between compiled Triton kernels and the slow Python interpreter, neutralizing compilation speedups. Developers minimize graph breaks by refactoring training loops to use tensor-native operations and avoiding arbitrary Python control flow within compiled regions.

Activation checkpointing discards intermediate activation tensors computed during the forward pass of deep sequential blocks rather than storing them all in VRAM for backpropagation. During the backward pass, when those activations are required to compute gradients, PyTorch recomputes them on-the-fly by running a mini-forward pass over the checkpointed module. This reduces peak VRAM consumption significantly, enabling larger batch sizes or deeper architectures, but incurs a performance penalty of approximately 20% to 30% due to redundant forward computations.

Dispatch Keys are enumerated flags attached to every PyTorch tensor that indicate its runtime state, backend device, and active transformations. When a tensor operation is invoked, the PyTorch Dispatcher examines the tensor's dispatch key set (e.g., AutogradCUDA, DispatcherKey::CUDA, DispatcherKey::CPU) to determine the correct kernel implementation to route to. This mechanism allows PyTorch to cleanly separate autograd tracking, tracing, device execution, and debugging hooks without cumbersome conditional logic in the operator code.