Each test is 5 questions with varying difficulty.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.
Disaggregated prefill-decode serving is a critical architectural pattern in modern LLM inference, designed to solve the fundamental performance mismatch between the compute-bound prefill phase and the memory-bandwidth-bound decode phase. In 2026, as models grow larger and context windows expand, monolithic serving engines often struggle with head-of-line blocking, where long prefill tasks stall token generation for other requests. This topic is essential for AI Engineers and System Architects building high-scale inference platforms. Interviewers ask about this to evaluate your understanding of GPU utilization, scheduling complexity, and the trade-offs between Time to First Token (TTFT) and total system throughput. Junior candidates are expected to understand the basic bottleneck distinction, while senior candidates must demonstrate mastery of cross-node scheduling, KV cache migration, and the impact of disaggregation on system reliability and complexity.
The core problem in LLM serving is that the prefill phase (processing the input prompt) is compute-intensive, while the decode phase (generating tokens) is memory-bandwidth-bound. In a standard monolithic setup, a long prompt can saturate the GPU's compute units, delaying the decode phase of other active requests, which significantly increases latency. Disaggregation decouples these phases, allowing operators to assign different GPU pools to each task. This increases resource utilization by matching the hardware profile (e.g., high-compute GPUs for prefill, high-bandwidth GPUs for decode) to the specific phase requirements. For companies serving millions of tokens daily, this can improve throughput by 2x-3x and significantly reduce tail latency. This is a high-signal interview topic because it tests whether a candidate understands the hardware-software interface. A strong answer shows an awareness of the overhead introduced by cross-node communication and the complexity of state synchronization (KV cache transfer) between pools, which are the primary failure modes in real-world production systems.
The architecture splits the inference engine into two distinct pools. The Prefill Pool processes incoming requests, computes the KV cache, and then offloads the state to the Decode Pool. The Decode Pool maintains the KV cache in its local memory and performs iterative token generation. Communication between pools is handled via a high-speed interconnect, often using specialized protocols to minimize latency during the handoff.
[Incoming Request]
↓
[Request Dispatcher]
↙ ↘
[Prefill Pool] [Decode Pool]
(Compute) (Generate)
↓ ↑
[KV Cache Transfer]
↓ ↑
[Interconnect Layer]Asynchronous transfer of KV cache tensors from prefill to decode nodes using RDMA.
Trade-offs: Reduces latency but introduces significant network complexity.
Dynamically adjusting the ratio of prefill-to-decode nodes based on incoming request distribution.
Trade-offs: Optimizes utilization but requires sophisticated monitoring.
Running different batch sizes in prefill and decode pools to match throughput capacity.
Trade-offs: Maximizes throughput but increases memory pressure.
| Reliability | Requires robust state synchronization; use checkpointing for KV cache transfers to handle node failures. |
| Scalability | Scale pools independently based on prefill-to-decode ratio; use service discovery for dynamic pool membership. |
| Performance | Bottlenecks are often network bandwidth or interconnect latency; monitor TTFT and tokens-per-second (TPS). |
| Cost | Reduces cost by allowing cheaper, compute-optimized GPUs for prefill and memory-optimized GPUs for decode. |
| Security | Secure inter-node communication with mTLS; ensure KV cache data is encrypted in transit. |
| Monitoring | Track queue depths, transfer latency, and GPU utilization per pool. |
Monolithic serving executes both prefill and decode on the same GPU, leading to resource contention. Disaggregated serving separates these into distinct pools, allowing for specialized hardware allocation and independent scaling of compute and memory-bandwidth resources, which significantly improves throughput and reduces tail latency.
The prefill phase involves processing the entire input sequence using matrix multiplications, which are compute-intensive. The GPU must perform massive parallel operations to compute the attention scores for all tokens in the prompt, saturating the compute units rather than the memory bandwidth.
During decoding, the system generates tokens one by one. Each step requires loading the model weights and the KV cache from memory to the GPU compute cores. Because the compute required for a single token is relatively small compared to the amount of data loaded, the system is limited by how fast it can move data from HBM to the cores.
The primary challenge is the overhead of transferring the KV cache between the prefill and decode pools. If the interconnect is slow or the transfer logic is inefficient, the latency added by the migration can negate the performance gains achieved by decoupling the phases.
Standard load balancers are insufficient because they lack awareness of the request's state (prefill vs. decode). You need a specialized dispatcher that understands the current load on both pools and the specific requirements of the request, such as prompt length and expected generation length.
Disaggregation improves TTFT by ensuring that incoming prefill requests are not blocked by long-running decode tasks. By offloading prefill to a dedicated pool, the system can guarantee that prompt processing starts immediately, leading to lower and more consistent Time to First Token.
No. Disaggregation introduces significant architectural complexity and network overhead. For small-scale deployments or models with very short context windows, the overhead of managing multiple pools and transferring state may outweigh the performance benefits. It is best suited for large-scale production environments.
NCCL (NVIDIA Collective Communications Library) is the standard for high-performance communication between GPUs. In disaggregated serving, it is used to efficiently transfer the large KV cache tensors between the prefill and decode nodes, minimizing the time spent in the handoff phase.
Fault tolerance requires stateful recovery mechanisms. If a node in the decode pool fails, the dispatcher must be able to redirect the request to another node, potentially re-fetching or re-computing the KV cache. This often involves checkpointing the state or maintaining redundant copies of the cache.
As context windows grow, the size of the KV cache increases linearly. This makes the transfer of the cache between pools more expensive, increasing the importance of efficient interconnects and potentially requiring techniques like KV cache compression or smarter caching strategies to keep latency low.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.