Tensorium flagship path · Understand → Implement → Experiment → Defend
Ship this model.
You have a Transformer checkpoint. Get it from a notebook to a production-grade serving stack.
Mission brief
“We need to serve it to 5,000 concurrent users. p95 TTFT < 1s. Keep GPU cost under control. Go.”
The checkpoint works in your notebook. Production does not have a notebook-shaped workload. Before buying GPUs, clarify arrival rate, active generations, prompt/output distributions, quality requirements, and what the latency clock includes. 5,000 connected users is not automatically 5,000 simultaneous generations.
model = AutoModelForCausalLM.from_pretrained(...)
tokens = model.generate(...)
# What breaks first?One dense family, four scales: 0.6B → 1.7B → 4B → 32B.
Opening release: two complete investigations with saved work. Run the downloadable baseline on your own GPU; Tensorium does not execute GPU inference or fabricate performance measurements.
Your Qwen3 fleet
Selection persists across missions
One dense model family, not identical shapes: depth, width, and attention dimensions still vary. No universal bottleneck or speedup is assigned by model size.
| Model / select | ||||
|---|---|---|---|---|
| DERIVED · BF16 payload | ~1.2 GB | ~3.4 GB | ~8 GB | ~65.6 GB |
| Baseline output tok/s | Not recorded | Not recorded | Not recorded | Not recorded |
| Baseline p95 TTFT | Not recorded | Not recorded | Not recorded | Not recorded |
| Run / clients | — | — | — | — |
Measurements appear only after you save an external run report; they are user-supplied, not verified by Tensorium. Match GPU, precision, revision, workload, and concurrency before comparing columns. The older inference path’s simulated metrics are not used here.
Problems first. Mechanisms when you need them.
The opening two investigations are available now. Later mission stages are the build plan below—not completed lessons or simulated benchmark claims.
00 · Investigation available
It works on my GPU. Will it fit on yours? →
Estimate all four checkpoints before revealing the memory budget. Make a defensible 48 GB admission decision.
01 · Investigation available
It answers one request. Now measure it. →
Run a deliberately serialized Transformers server on your GPU, collect client-visible latency, and retain real baseline reports.
02 · Planned mission
Why is one token so expensive?
Investigate prefill versus decode with a token-level trace.
03 · Planned mission
We’re recomputing the whole conversation.
Implement cache reuse and price the live state across the Qwen fleet.
04 · Planned mission
One user is easy. Now 100 arrive.
Build a tiny iteration scheduler; handle completion, cancellation, fairness, and backpressure.
05 · Planned mission
We’re out of memory, but half of it is empty.
Diagnose reserved versus live KV state before introducing block allocation.
06 · Planned mission
Stop maintaining a homemade serving engine.
Compare the identical checkpoint and workload in Transformers, vLLM, and SGLang.
07 · Planned mission
The 32B still won’t fit comfortably.
Compare precision, quality, kernels, and tensor parallelism—not just ideal weight bytes.
08 · Planned mission
We keep paying for the same prompt.
Investigate prefix reuse, retention cost, and isolation boundaries.
09 · Planned mission
One long prompt stalls everyone.
Trade prompt progress against decode latency using chunked prefill.
10 · Planned mission
Can the small model save the big one time?
Use a 1.7B draft and 32B target; account for accepted tokens, verification, and rollback.
11 · Planned mission
One GPU becomes two.
Price tensor versus pipeline parallelism on the actual interconnect.
12 · Planned mission
Prefill and decode are fighting.
Compare interference savings against KV transfer and operational costs.
13 · Planned mission
Defend the cheapest architecture that meets the SLO.
Qwen3-32B · 250 requests/s peak · 50K DAU · p95 TTFT < 800 ms · p95 TPOT < 40 ms · H100 80 GB.
Need mechanism depth now? The existing inference reference path remains available, including FlashAttention investigations and the separate 70B worksheet.
Technical ground truth
Qwen recommends vLLM/SGLang for deployment. Mechanisms and runtime support are grounded in the sources below; support depends on the pinned engine release, device, checkpoint, and configuration. Later mission stages will use these sources, not pretend that every toggle has a universal speedup.