Deep Learning Fundamentals Interview Preparation Guide

Shallow machine learning models, such as Support Vector Machines or Random Forests, rely on manual feature engineering where human domain experts extract relevant signals before feeding them into algorithms. Deep learning neural networks, by contrast, utilize multiple hidden layers to automatically learn hierarchical feature representations directly from raw input data. In deep architectures, lower layers detect primitive features (e.g., edges or basic tokens), while deeper layers compose these into complex semantic representations. This end-to-end learning capability allows deep models to scale effectively with massive datasets, though it requires significantly greater compute infrastructure and careful management of gradient flow and regularization.

If you stack multiple linear layers (matrix multiplications) without intervening non-linear activation functions, the entire composite transformation collapses mathematically into a single equivalent linear transformation. No matter how many hidden layers a network contains, linear combinations of linear functions remain linear, severely restricting the hypothesis space to linear decision boundaries. Non-linear activation functions such as ReLU, GELU, or Swish introduce non-linear transformations that empower the network to approximate arbitrary functions, as guaranteed by the Universal Approximation Theorem. This enables neural networks to capture complex, multi-dimensional patterns in data.

Backpropagation computes gradients by applying the chain rule of calculus in reverse order across the computational graph. During the forward pass, the network caches intermediate activation tensors for every operation. During the backward pass, starting from the loss scalar at the output, the algorithm computes local partial derivatives (Jacobians) and multiplies them successively backward through each layer. This vector-Jacobian product traversal efficiently computes the exact partial derivative of the loss function with respect to every weight and bias parameter in the network simultaneously, avoiding redundant calculations and enabling gradient-based optimization.

The vanishing gradient problem occurs in deep networks when gradients propagated backward through many layers approach zero due to repeated multiplication of small Jacobian values (such as derivatives of sigmoid or tanh activation functions less than one). As gradients approach zero, parameters in early layers cease updating, halting learning entirely. Modern architectures mitigate this by replacing saturating activations with ReLU or GELU (which have constant positive gradients for positive inputs), incorporating residual skip connections that allow gradients to flow unimpeded around layers, and employing normalization techniques like LayerNorm and Batch Normalization.

Mean Squared Error (MSE) calculates the average squared difference between predicted continuous values and ground truth targets, assuming normally distributed errors, making it standard for regression tasks. Cross-Entropy Loss quantifies the divergence between predicted probability distributions and true discrete class labels, heavily penalizing confident incorrect predictions via logarithmic scaling, making it the standard for classification tasks. Using MSE for multi-class classification is suboptimal because it penalizes 'over-correct' predictions where the model assigns high probability to the correct class, whereas Cross-Entropy penalizes incorrect confidence exponentially.

During training, batch normalization computes the mean and variance of activations across the current mini-batch, normalizes them, and updates running exponential moving averages of population mean and variance. It also applies learnable scaling (gamma) and shifting (beta) parameters. During evaluation (inference) mode, batch normalization disables mini-batch statistics calculation and instead utilizes the accumulated running population mean and variance to normalize input tensors deterministically. Failing to switch modes via model.eval() causes predictions to depend stochastically on batch composition, leading to erratic validation metrics.

Activation checkpointing is a memory optimization technique designed to reduce the high memory footprint caused by caching intermediate activation tensors during the forward pass of deep models. Instead of storing all activations, checkpointing discards non-critical activations and recalculates them on-the-fly during the backward pass when gradients are computed. The primary trade-off is exchanging memory capacity for compute time: it can reduce peak memory consumption by 50-70%, allowing training of larger models or batch sizes, at the expense of a 20-30% increase in training compute time due to redundant forward pass recomputations.

Mixed-precision training utilizes 16-bit floating-point formats (FP16 or BF16) for forward and backward passes to leverage hardware Tensor Cores, which double training throughput and halve memory bandwidth requirements compared to FP32. The primary risk in FP16 training is numerical underflow, where very small gradient values round to zero due to limited dynamic range, which is mitigated using dynamic loss scaling. BF16 retains the same dynamic range as FP32 by allocating more bits to the exponent, eliminating the need for loss scaling while still accelerating matrix multiplications.

Traditional L2 regularization adds a penalty proportional to squared weight magnitudes directly into the loss function, meaning the penalty gets scaled by the adaptive learning rate divisors in optimizers like Adam, distorting the regularization effect. Decoupled weight decay (implemented in AdamW) separates the weight penalty from the gradient update step, applying the decay directly to the model weights independent of the gradient scaling history. This ensures consistent, predictable regularization strength across all parameters regardless of their historical gradient variance.

Dynamic computation graphs ('define-by-run') construct the execution graph on-the-fly during every forward pass based on Python control flow, enabling intuitive debugging, variable input shapes, and flexible architectures. Static computation graphs ('define-and-run') compile the neural network into a fixed computational graph before execution, allowing compilers to perform aggressive graph optimizations, kernel fusion, and memory planning that maximize production hardware throughput and deployment portability.

The dying ReLU problem occurs when a large gradient flows through a ReLU neuron, causing its weight updates to shift the neuron such that its input is always negative for all training data. Once negative, the gradient of ReLU is zero, so the neuron never updates again and permanently outputs zero. LeakyReLU solves this by introducing a small, non-zero slope (e.g., 0.01) for negative inputs, ensuring a non-zero gradient flows backward. Exponential Linear Units (ELU) introduce an exponential curve for negative inputs that smoothly approaches a negative saturation limit, encouraging mean activations closer to zero.

The primary objective of FSDP is to overcome single-GPU memory capacity limits when training models with billions of parameters. In standard Data Parallelism, every GPU replicates the entire model, optimizer states, and gradients, leading to severe memory bloat. FSDP shards model parameters, gradients, and optimizer states across all available data-parallel worker ranks. During forward and backward passes, worker ranks dynamically fetch required parameter slices via collective communication primitives (All-Gather and Reduce-Scatter), enabling the training of massive models that would otherwise exceed GPU high bandwidth memory.