Each test is 5 questions with varying difficulty.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.
In the era of large-scale transformer models, the performance of GPU-accelerated workloads is increasingly dictated by memory hierarchy rather than raw compute cycles. Understanding the distinction between on-chip SRAM and off-chip High Bandwidth Memory (HBM) is critical for AI engineers and system architects in 2026. SRAM serves as the ultra-low latency, high-bandwidth scratchpad directly integrated into the streaming multiprocessors (SMs), while HBM provides the massive capacity required to store model weights and KV caches. Interviewers focus on this topic to assess a candidate's ability to reason about the memory wall—the fundamental bottleneck where compute cores sit idle waiting for data from HBM. Junior candidates are expected to understand the latency and capacity trade-offs, while senior engineers must demonstrate knowledge of how kernel fusion, tiling strategies, and FlashAttention leverage SRAM to minimize HBM round-trips. Mastery of this topic is essential for roles involving LLM inference optimization, CUDA kernel development, and distributed training infrastructure.
The 'Memory Wall' is the primary constraint in 2026 AI system design. With HBM3e bandwidth reaching several terabytes per second, it still pales in comparison to the aggregate bandwidth of on-chip SRAM, which can exceed 100 TB/s for a single GPU. Engineers who understand this disparity can design kernels that maximize arithmetic intensity. For example, in transformer inference, the KV cache resides in HBM; if the attention mechanism is not tiled to fit into SRAM, the system becomes memory-bound, leading to low GPU utilization and increased Time to First Token (TTFT). This topic is a high-signal interview area because it separates candidates who treat GPUs as black boxes from those who understand the physical constraints of data movement. A strong answer demonstrates awareness of how modern algorithms like FlashAttention-3 explicitly manage the SRAM-HBM boundary to avoid redundant HBM reads/writes, directly impacting the cost-per-token of large-scale inference deployments.
The GPU memory hierarchy operates as a multi-stage funnel. Data flows from the massive but slow HBM through the L2 cache, then into the L1/Shared Memory (SRAM) located within each Streaming Multiprocessor (SM). The execution model relies on threads requesting data from HBM; if the data is not in SRAM, a high-latency transaction occurs. Efficient kernels use tiling to load data into SRAM once, perform all necessary computations, and then write back to HBM, minimizing redundant traffic.
[HBM (Global Memory)]
↓
[L2 Cache]
↓
[Streaming Multiprocessor]
↓ ↓
[L1 Cache] [Shared Memory (SRAM)]
↓ ↓
[Register File]
↓
[ALU]Partitioning large tensors into blocks that fit within the SRAM capacity of an SM.
Trade-offs: Reduces HBM traffic but increases complexity of index calculation and synchronization.
Loading the next tile into SRAM while the current tile is being processed by the ALU.
Trade-offs: Hides memory latency but doubles the SRAM footprint required for data storage.
| Reliability | ECC memory in HBM prevents silent data corruption, but SRAM errors require hardware-level parity checks. |
| Scalability | Scaling is limited by HBM bandwidth; adding more GPUs requires NCCL to manage inter-GPU memory movement. |
| Performance | Bottlenecks occur when arithmetic intensity is below the roofline limit of the GPU. |
| Cost | HBM is the primary driver of GPU cost; optimizing for SRAM usage extends the lifecycle of existing hardware. |
| Security | Side-channel attacks can exploit SRAM access patterns to infer data content. |
| Monitoring | Track 'dram_throughput' and 'shared_memory_utilization' metrics in Nsight. |
SRAM is on-chip, extremely fast, and low-capacity, used as a scratchpad. HBM is off-chip, stacked DRAM, high-capacity, and slower, used for main storage.
Even with multi-terabyte speeds, the compute power of modern GPUs grows faster than HBM bandwidth, leading to situations where cores are idle waiting for data.
It occurs when multiple threads in a warp attempt to access different memory addresses that map to the same memory bank, forcing the hardware to serialize the accesses.
It tiles the attention matrix into blocks that fit into SRAM, allowing the GPU to compute attention scores without writing intermediate results back to HBM.
Yes, L2 cache is implemented using SRAM technology, but it is distinct from the user-managed 'Shared Memory' which is explicitly controlled by the developer.
It is the ratio of floating-point operations (FLOPs) to memory bytes accessed. High intensity means the kernel is compute-bound; low intensity means it is memory-bound.
No, HBM capacity is a hardware constraint. Software can only optimize how that capacity is used (e.g., via quantization or KV cache eviction).
A performance model that plots peak performance against arithmetic intensity to show whether a kernel is limited by compute or memory bandwidth.
Coalesced access allows the GPU to combine multiple thread requests into a single memory transaction, maximizing the utilization of the HBM bus.
Occupancy is the ratio of active warps to the maximum supported by the SM. High shared memory usage per thread block reduces the number of blocks that can run, lowering occupancy.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.