Each test is 5 questions with varying difficulty.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.
Designing a rate limiter is a classic system design interview question that tests a candidate's ability to balance performance, accuracy, and distributed state management. In 2026, as AI-driven applications and high-frequency API services proliferate, the need for robust traffic shaping has moved from a basic security feature to a core architectural requirement for protecting downstream LLM inference endpoints and microservices. This topic is essential for Backend, Platform, and SRE roles. Interviewers expect junior candidates to understand basic algorithms like fixed window counters, while senior candidates must demonstrate proficiency in distributed consistency, race condition mitigation in Redis, and the trade-offs between local vs. global rate limiting. A strong answer must address how to handle high-concurrency environments without introducing significant latency or becoming a single point of failure.
Rate limiting is the primary defense mechanism against DoS attacks, brute-force attempts, and resource exhaustion in multi-tenant environments. In modern AI infrastructure, rate limiting is critical to manage GPU quota allocation and prevent cost spikes from runaway prompt generation. A poorly designed rate limiter can introduce a latency bottleneck at the edge, while an overly permissive one fails to protect the backend. Interviewers use this topic to assess how a candidate handles distributed state. If you suggest a simple in-memory counter, you demonstrate a lack of awareness of distributed environments. If you suggest a database-backed counter, you reveal a lack of performance awareness. A strong candidate navigates these trade-offs by proposing a tiered approach: local rate limiting for immediate protection and Redis-backed global limiting for cross-node consistency. This topic is increasingly relevant in 2026 due to the rise of 'token-based' billing, where rate limiting must now account for variable request costs, such as input/output token counts in LLM APIs, rather than just simple request counts.
The architecture typically involves an edge layer (Gateway) that intercepts requests and queries a distributed state store (Redis) to determine if the request should proceed or be rejected with a 429 status code.
[Client Request]
↓
[API Gateway Layer]
↓ ↓
[Local Cache] [Redis Cluster]
↓ ↓
[Decision Logic]
↓ ↓
[Allow Request] [Reject 429]
↓
[Backend Service]Encapsulate read-check-update logic in a single Redis Lua script to prevent race conditions.
Trade-offs: Reduces network round trips but blocks the Redis main thread if the script is long.
Use a local in-memory cache for high-frequency checks and sync periodically with Redis.
Trade-offs: Improved latency but introduces slight inaccuracy in quota enforcement.
Use Redis cluster hash tags (e.g., {user_id}:rate) to ensure all keys for a user map to the same shard.
Trade-offs: Simplifies cross-key operations but can create hotspots on specific shards.
| Reliability | Use Redis Sentinel or Cluster for high availability. Whether to fail-open or fail-closed when the store is unreachable is a risk decision, not a default: fail-open keeps the API responsive but removes rate-limit protection exactly when an outage-triggered abuse spike is most likely, while fail-closed blocks legitimate traffic but preserves backend protection. |
| Scalability | Partition Redis by user ID; use local caches to reduce load on the global store. |
| Performance | Target sub-millisecond latency for the decision engine; use Lua scripts to minimize RTT. |
| Cost | Minimize memory usage in Redis by using short TTLs and efficient data structures like bitmaps. |
| Security | Validate API keys before rate limiting; protect the rate limiter itself from being flooded. |
| Monitoring | Track 429 error rates, Redis latency, and cache hit/miss ratios. |
Rate limiting restricts the number of requests a user can make in a time window. Throttling is a broader term that can include rate limiting but also encompasses shaping traffic, such as delaying responses to maintain a specific throughput.
Redis provides sub-millisecond latency and atomic in-memory operations, which are critical for high-frequency traffic. PostgreSQL is disk-backed and introduces significant overhead that would bottleneck API throughput.
Only if your service is a single instance. In a distributed system, an in-memory counter will not be shared across nodes, leading to inaccurate limits and potential abuse.
It occurs when many rate limit keys expire simultaneously, causing a spike in database load as nodes attempt to re-initialize or refresh their state at the exact same moment.
Avoid relying on client-side or local node timestamps. Use a centralized time source or relative time offsets provided by the distributed store (e.g., Redis TIME command).
API Key is more accurate for identifying users, especially behind NATs or proxies. IP-based limiting is a fallback for unauthenticated traffic but is prone to false positives.
Fixed window resets at specific clock intervals, leading to 'boundary spikes'. Sliding window uses a rolling time frame, providing a smoother and more accurate enforcement of limits.
Use a distributed Redis cluster with replication to minimize outages in the first place. For the residual case where the store is still unreachable, decide deliberately between fail-open (traffic is allowed, keeping the API responsive but leaving it unprotected during the outage) and fail-closed (traffic is blocked, protecting the backend but reducing availability). Fail-open is reasonable for low-risk internal services; fail-closed is often safer for public endpoints where an outage could be exploited as a DoS or cost-abuse window.
A bucket is filled with tokens at a fixed rate. Each request consumes a token. If the bucket is empty, the request is rejected. This allows for short bursts of traffic while enforcing a long-term average rate.
Lua scripts run atomically on the Redis server. This prevents race conditions where two requests might read the same counter value before either has incremented it.
AI Prep covers AI Agents, Generative AI, ML Fundamentals, NLP & LLMs and a lot more, with adaptive tests and daily challenges. Fully offline on Android. Free to try, one-time unlock for lifetime access.