Each test is 5 questions with varying difficulty.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.
GPU memory bandwidth bottlenecks occur when the rate at which data is transferred between high-bandwidth memory (HBM) and the GPU compute units (SMs) becomes the limiting factor for performance. In 2026, as LLMs grow in parameter count and context length, understanding these bottlenecks is critical for AI Engineers and ML Infrastructure specialists. While compute-bound operations are limited by FLOPs, memory-bound operations—common in transformer inference—are limited by the throughput of the memory bus. Interviewers test this to see if candidates can distinguish between compute-intensive tasks like matrix multiplication and memory-intensive tasks like KV cache lookups. Junior candidates are expected to understand the basic distinction between HBM and SRAM, while senior candidates must be able to perform roofline analysis, optimize kernel fusion, and explain how quantization or KV cache compression mitigates bandwidth saturation.
In modern LLM serving, the 'Memory Wall' is the primary obstacle to scaling inference throughput. Because transformer models require loading massive weight matrices and KV cache states for every token generated, the time spent moving data from HBM to the SMs often dwarfs the time spent on actual floating-point calculations. For example, a model with 70B parameters requires 140GB of data transfer per forward pass (at FP16). If the GPU HBM bandwidth is 2TB/s, the theoretical minimum latency per token is constrained by this transfer speed, regardless of how many TFLOPS the GPU provides. This topic is high-signal because it separates candidates who treat GPUs as 'black boxes' from those who understand the physical constraints of hardware. A strong candidate can explain why quantization (e.g., INT8/FP8) improves performance not just by reducing compute, but by halving the bandwidth requirement, effectively doubling throughput in memory-bound scenarios. In 2026, with the rise of long-context models, bandwidth efficiency is the difference between a viable product and one that suffers from unacceptable Time to First Token (TTFT).
The GPU execution model relies on a hierarchical memory structure. Data flows from the system RAM to the GPU's High Bandwidth Memory (HBM), then to the L2 cache, and finally to the register file and shared memory (SRAM) within the Streaming Multiprocessors (SMs).
Data is fetched from HBM via the Memory Controller, cached in L2, and loaded into SM registers for computation.
[System RAM]
↓
[Memory Controller]
↓
[HBM]
↓
[L2 Cache]
↓
[SM Register File]
↓
[Compute Units (ALU)]Merging multiple operations into one kernel to keep data in registers.
Trade-offs: Increases kernel complexity and register pressure.
Loading sub-blocks into shared memory to reuse data.
Trade-offs: Requires complex indexing and synchronization.
Loading INT8/FP8 weights and dequantizing in registers.
Trade-offs: Requires hardware support for fast dequantization.
| Reliability | Use ECC memory and monitor HBM temperature to prevent bit flips and throttling. |
| Scalability | Scale by partitioning models across multiple GPUs to distribute memory bandwidth load. |
| Performance | Target arithmetic intensity above the 'ridge point' of the roofline model. |
| Cost | Reduce cost by using smaller precision formats to increase throughput per GPU. |
| Security | Prevent side-channel attacks by clearing shared memory between user contexts. |
| Monitoring | Track 'Memory Throughput' and 'L2 Cache Hit Rate' in Nsight. |
Memory-bound means the execution speed is limited by the rate of data transfer from HBM to the SMs. Compute-bound means the execution speed is limited by the number of floating-point operations the SMs can perform per second. Most LLM inference is memory-bound.
Quantization reduces the bit-width of weights and activations. Since memory-bound tasks are limited by the volume of data moved, reducing the size of the data moved directly increases the number of operations that can be completed in the same amount of time, effectively increasing throughput.
Kernel fusion is the process of combining multiple sequential operations (like matrix multiply followed by activation) into a single GPU kernel. This keeps intermediate results in the fast on-chip SRAM instead of writing them back to the slower HBM, saving significant bandwidth.
The roofline model plots performance (GFLOPS) against arithmetic intensity (FLOPs/byte). It provides a theoretical 'roof' for performance based on hardware limits. If a kernel is below the ridge point, it is memory-bound; if it is above, it is compute-bound.
The Memory Wall refers to the growing gap between processor performance and memory bandwidth. As GPUs become faster at computing, the HBM bandwidth has not kept pace, making data movement the primary bottleneck for many modern AI workloads.
Registers are the fastest memory on the GPU. If a kernel uses too many registers, the compiler must 'spill' data to local memory (which is actually in HBM). This drastically increases latency and consumes precious memory bandwidth.
Coalesced access occurs when threads in a warp access contiguous memory addresses. The memory controller can combine these into a single transaction, maximizing the utilization of the memory bus. Non-coalesced access requires multiple transactions, wasting bandwidth.
PagedAttention manages KV cache memory in non-contiguous blocks, similar to virtual memory in an OS. This prevents memory fragmentation, allowing for more efficient use of HBM and reducing the need for excessive data movement.
Increasing batch size can amortize the cost of loading model weights (since they are shared across the batch), which helps. However, it also increases the KV cache size, which can eventually lead to memory exhaustion or bandwidth saturation if not managed properly.
HBM is the primary high-capacity memory on the GPU, while L2 cache is a smaller, faster on-chip buffer. L2 cache acts as a staging area to reduce the number of expensive fetches from HBM to the SMs.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.