Research Engineer path · operating a service
Optimize an LLM inference stack.
Symptom → hypothesis → mechanism → measurement → decision. Need the depth; don’t just consume it.
Explore the mission and system map freely. Sign in to investigate and save your work.
Flagship mission · Qwen3 fleet
Ship this model.
You have a Transformer checkpoint. Get it from a notebook to a production-grade serving stack.
Start with sizing and a real, instrumented baseline →
Your assignment
You inherited a Qwen serving endpoint.
It works for one user.
At 100 concurrent requests, p99 latency explodes.
At 32.8B parameters, theoretical BF16 parameter memory is ~65.6 decimal GB before KV cache, allocator/runtime state, and temporary workspace—making single-80-GB-GPU serving heavily memory constrained.
Your job: make it production-ready.
Four checkpoints: 0.6B · 1.7B · 4B · 32B. Diagnose the incident, predict a change, inspect the replay, then decide what earns a rollout.
Training simulation, not a hosted inference service. FlashAttention is the first incident-led chapter; the system map and existing deep material remain available below.
System map
Select a technique to investigate it. Solid arrows organize the reasoning; dashed branches are optional levers, not mandatory optimizations. This is a conceptual dependency map, not a claim that batching requires speculation.
On narrow screens, scroll the map horizontally or use the accessible chapter index below.
Chapter index · outcomes & dependencies
01 · Start with the workload · ~20 min
Build the inference mental model →
Separate the latency budget from the throughput budget; identify the bytes that dominate a decode step.
Builds on: Attention shapes and basic FLOP/byte accounting
02 · Understand cost · ~25 min
KV cache: derive the persistent state →
Calculate Llama, Qwen, and DeepSeek cache footprints without applying the GQA formula to MLA.
Builds on: Cost & latency
03 · Understand latency · ~25 min
Long prompts are killing TTFT →
Explain tiled exact attention, online softmax, and why fewer bytes do not promise a fixed speedup.
Builds on: Cost & latency
04 · Understand cost · ~20 min
PagedAttention: allocate what is actually alive →
Distinguish allocation waste from attention arithmetic and derive block-level capacity.
Builds on: KV cache
05 · Combine the budgets · ~25 min
Continuous batching: schedule iterations, not requests →
Reason about request arrivals/departures and optimize SLO-compliant throughput instead of batch occupancy alone.
Builds on: Cost & latency · PagedAttention
06 · Understand cost · ~25 min
Quantization: choose the representation and the kernel →
Separate weight, activation, and KV precision; budget payload, scales, quality, and actual kernel support.
Builds on: Cost & latency · KV cache
07 · Understand latency · ~30 min
Speculative decoding: buy accepted tokens, not drafts →
Derive distribution-preserving acceptance and the break-even condition including draft and verification cost.
Builds on: Cost & latency
08 · Combine the budgets · ~35 min
Distributed serving: place compute, state, and queues →
Choose parallelism and phase placement using per-rank memory, network cost, prefix reuse, and SLO evidence.
Builds on: Continuous batching · Quantization · FlashAttention
09 · Optional model lever · ~20 min
Model-side efficiency: change the contract deliberately →
Distinguish checkpoint architecture from runtime choices, and evaluate routing or distillation without hiding quality loss.
Builds on: KV cache · Quantization
10 · Defend the whole system · ~60 min
Capstone: design an inference stack →
Design a 70B service for 10,000 connected users, p99 TTFT < 1.5 s and p99 TPOT < 50 ms—with explicit traffic assumptions, quality gates, and GPU cost.
Builds on: Distributed serving · Speculative decoding · Model-side efficiency
Separate 70B interview case · existing quantitative capstone
70B. 10,000 users. A defensible GPU bill.
Constrain p99 TTFT below 1.5 seconds and p99 TPOT below 50 ms at an 8k average cached context. First distinguish connected users from active generations. Then choose precision, parallelism, KV policy, batching, scheduling, speculation, hardware, and autoscaling—with synthetic evidence and explicit assumptions.
Design the inference stack →Use the existing depth where it matters
Architecture stays focused on model mechanisms. Here, those mechanisms are inputs to a serving decision—not a replacement for workload evidence.