Apache Spark Engine & Scale Interview Preparation Guide

In traditional eager evaluation languages like Python or Java, statements execute immediately when encountered by the interpreter or compiler. In contrast, Apache Spark utilizes lazy evaluation for all transformations. When a developer applies transformations such as map, filter, or join, Spark does not compute the results immediately. Instead, it builds up an immutable Directed Acyclic Graph (DAG) of logical operations called lineage. Execution is deferred entirely until an action such as collect(), count(), or write() is invoked. This allows the Catalyst optimizer to analyze the complete execution plan holistically, enabling powerful optimizations like predicate pushdown, column pruning, and pipeline fusion before any data is processed.

The distinction lies in data dependency across partitions. A narrow transformation (e.g., map, filter, sample) allows each output partition to be computed from a single input partition, requiring zero network shuffle across cluster nodes. This enables pipelining where multiple narrow operations execute within a single stage. A wide transformation (e.g., groupByKey, reduceByKey, join, repartition) requires data from multiple input partitions to be combined, triggering a shuffle boundary where records are redistributed across different executors over the network via Netty RPC. Wide transformations divide the DAG into distinct stages and represent the primary performance bottleneck in distributed Spark applications.

A driver out-of-memory error occurs when the Spark driver JVM heap is overwhelmed by data or metadata. Common causes include calling collect() on massive DataFrames, broadcasting excessively large datasets via broadcast(), collecting large driver-side logs, or registering millions of partition metadata entries in the SparkContext. To prevent driver OOMs, engineers should avoid collect() entirely in production, use take(n) or limit(n) for inspection, ensure broadcast table size estimates are accurate before calling broadcast(), and offload large aggregations to distributed executor tasks rather than gathering results on the driver.

Standard JVM memory management allocates objects on the heap with object header overhead and pointer indirection, making it highly susceptible to lengthy Stop-The-World garbage collection pauses when millions of small objects are created. Project Tungsten bypasses the JVM heap entirely by managing memory explicitly off-heap using the sun.misc.Unsafe API. Tungsten encodes data into raw binary byte arrays tailored for CPU cache locality, performing sorts, joins, and hashes directly on binary payloads without deserialization. This eliminates garbage collection overhead and drastically reduces memory footprint and CPU instruction cycles.

Adaptive Query Execution (AQE) is a framework introduced in Spark 3 that re-optimizes physical query execution plans mid-flight using accurate runtime statistics collected from completed shuffle stages. AQE solves three primary runtime inefficiency problems: first, it dynamically coalesces shuffle partitions to prevent small-file performance degradation; second, it dynamically converts Sort-Merge Joins into Broadcast Hash Joins at runtime if join side sizes fall below thresholds; and third, it automatically detects and splits skewed shuffle partitions into smaller sub-partitions to eliminate executor stragglers.

Standard Python UDFs force Spark to serialize JVM-based row objects into Python-interpretable formats, send them across a local socket pipe to a separate Python worker process, execute the Python function row-by-row, and serialize the results back to the JVM. This per-row inter-process communication and serialization overhead destroys Catalyst optimization opportunities. Pandas UDFs (vectorized UDFs) solve this by leveraging Apache Arrow to perform zero-copy columnar data transfer between JVM and Python processes, allowing vector operations using NumPy and Pandas across batches of rows.

Exit code 137 indicates that the executor container was forcibly killed by the operating system, YARN container manager, or Kubernetes out-of-memory killer because it exceeded its allocated memory limits. This typically happens when executor memory overhead (spark.executor.memoryOverhead) is set too low, or when tasks consume excessive off-heap memory buffers, native library allocations, or unmanaged Python worker memory. To resolve this, engineers should increase executor memory overhead, optimize memory-intensive data structures, or scale up container memory limits in the cluster resource configuration.

cache() is a shorthand alias for persist(StorageLevel.MEMORY_AND_DISK), which stores RDD or DataFrame partitions in JVM heap memory and spills to local disk if memory is exhausted. persist() allows developers to explicitly specify custom StorageLevels, such as MEMORY_AND_DISK_SER (serialized objects in memory) or DISK_ONLY. Custom storage levels should be used when object serialization reduces memory footprint significantly or when datasets are too large to fit comfortably in un-serialized heap memory, balancing memory consumption against CPU serialization overhead.

Spark achieves fault tolerance through RDD lineage tracking and shuffle file management. When shuffle map tasks complete successfully on an executor, they write shuffle block files to local scratch space. If a downstream executor fails while fetching shuffle blocks, the TaskScheduler notifies the DAGScheduler. The DAGScheduler identifies which shuffle map stage generated the lost blocks and re-runs only the specific upstream map tasks necessary to regenerate those missing blocks, without requiring the entire application to restart from the beginning.

The optimal shuffle partition count depends on total input data volume, cluster core capacity, and target partition size, with best practices recommending 100MB to 200MB per partition. The default value of 200 is optimized for small-scale development workloads; for petabyte-scale production datasets, 200 partitions result in massive gigabyte-sized partitions that trigger executor memory spills. Conversely, setting partitions too high creates millions of tiny tasks and output files, overwhelming the driver with scheduling overhead and causing small-file storage issues.