Machine Learning Fundamentals Interview Preparation Guide

Supervised learning algorithms require explicit ground-truth target labels paired with input feature vectors to optimize predictive mapping functions against a defined loss metric. In contrast, unsupervised learning models operate exclusively on unlabeled observational data to discover latent structures, clusters, or probability distributions. In system design, supervised models are deployed for direct classification and regression tasks where historical outcomes are known, whereas unsupervised models are utilized for exploratory data analysis, dimensionality reduction, anomaly detection, and customer segmentation where target labels are absent or cost-prohibitive to acquire.

The bias-variance tradeoff describes the inverse relationship between a model's simplification error (bias) and its sensitivity to training set fluctuations (variance). High bias models, such as linear regression on complex non-linear data, suffer from underfitting and fail to capture underlying patterns. High variance models, such as unpruned deep decision trees, memorize training noise and fail to generalize on unseen test data. Tuning hyperparameters involves balancing model capacity to minimize total expected error, utilizing validation curves to identify the optimal complexity sweet spot where out-of-sample generalization error reaches its minimum.

Computing summary statistics like mean, standard deviation, or min-max bounds across an entire dataset prior to partitioning introduces data leakage. Because the summary metrics incorporate information from the held-out test or validation sets, the scaling transformation leaks future knowledge into the training fold. When models evaluate on test data that was effectively touched during scaling, cross-validation metrics become artificially inflated and fail to reflect true out-of-sample generalization performance. The correct engineering pattern encapsulates scaling transformers inside cross-validation pipelines so they fit strictly on training folds.

L1 regularization adds a penalty proportional to the absolute sum of feature coefficients to the loss function, which introduces non-differentiable sharp corners at zero and drives irrelevant feature coefficients to exact zero, performing automatic feature selection. L2 regularization adds a penalty proportional to the squared sum of coefficients, shrinking them smoothly toward zero without eliminating them entirely. L2 handles multi-collinearity by distributing weight across correlated predictors, whereas L1 tends to select one predictor arbitrarily and zero out the rest.

On highly imbalanced datasets—such as fraud detection where positive events constitute 0.1% of transactions—a naive model that always predicts the majority class achieves 99.9% accuracy while completely failing to detect any fraudulent events. Accuracy treats all errors equally and fails to distinguish between true performance on minority classes. Engineers must use domain-appropriate evaluation metrics such as Precision, Recall, F1-Score, Precision-Recall AUC, or ROC-AUC, alongside cost-sensitive learning or resampling techniques to ensure models actually perform their intended function.

Gradient descent optimizers update model parameters iteratively in the opposite direction of the gradient of the loss function. In deep learning and complex ML models, loss surfaces are non-convex, featuring numerous local minima, saddle points, and ravines. Training stagnation occurs when gradients approach zero at saddle points or flat plateaus, or when learning rates are improperly tuned, causing loss values to oscillate wildly. Advanced optimizers like Adam incorporate momentum and adaptive learning rate scaling to accelerate movement through ravines and escape saddle points.

Look-ahead bias occurs when a model is trained using information from time periods subsequent to the prediction target, effectively allowing the model to cheat by seeing the future. In time-series forecasting, standard random K-Fold cross-validation causes look-ahead bias because training folds randomly interleave future and past observations. Engineers prevent this by implementing rolling origin time-series cross-validation splits, ensuring that training data strictly precedes validation and test data chronologically, perfectly mimicking real-world production inference conditions.

Using mean squared error with logistic regression's sigmoid activation function results in a highly non-convex error surface riddled with numerous local minima, causing gradient descent optimization to fail or get trapped easily. Cross-entropy loss, derived from maximum likelihood estimation principles, pairs with the sigmoid function to yield a strictly convex optimization loss surface. This convexity guarantees that any local minimum found by gradient descent is also the global minimum, ensuring reliable and stable model convergence.

Early stopping monitors out-of-sample validation loss during iterative training epochs and halts optimization when validation performance ceases to improve for a specified patience threshold. By terminating training before the model has fully converged on the training set, early stopping prevents the optimizer from over-optimizing on training set noise and capturing spurious patterns. This restricts the effective parameter search space and acts as an implicit regularization constraint, preserving strong generalization performance.

Tree-based models partition feature spaces based on relative feature value ordering and split thresholds rather than absolute numerical magnitudes or linear distances. Consequently, monotonic transformations, feature scaling, and normalization have zero impact on tree split decisions. Additionally, decision trees handle non-linear feature interactions and outliers natively. In contrast, linear models require strict feature scaling, multi-collinearity checks, and explicit interaction terms because their coefficients assume linear additivity and sensitivity to scale.

The curse of dimensionality refers to various phenomena that arise when analyzing data in high-dimensional spaces (hundreds or thousands of features) that do not occur in low-dimensional settings. As dimensionality increases, the volume of the feature space grows exponentially, causing data points to become extremely sparse. For distance-based algorithms like K-Nearest Neighbors, the distance between the nearest and farthest points converges, making distance metrics lose their discriminatory power and degrading model predictive performance.

Exhaustive Grid Search evaluates every discrete combination of hyperparameters across a predefined grid, resulting in exponential computational time complexity as the number of parameters increases. Bayesian Optimization treats hyperparameter tuning as a global optimization problem by constructing a probabilistic surrogate model (such as a Gaussian Process) of the objective function. It uses an acquisition function to intelligently balance exploration of uncertain regions and exploitation of promising parameter spaces, finding optimal configurations significantly faster.