YOUR INTERACTIVE RESEARCH STUDIO
The research room 3D
EXPLORE
TRY A CHANGE
EXPLORE · EXPERIMENT · UNDERSTAND / THE INTERACTIVE RESEARCH LAB
Step inside inference.
Predict an outcome. Run an experiment. Understand what changed.
YOUR INFERENCE ESTIMATESAnalytical model · not a benchmark
Will it fit?
What’s the bottleneck?
per phaseWork per byte
Arithmetic intensity: how much math you get from moving data.
Log scales · simplified aggregate ceiling. Latency adds separate matmul and attention stage estimates; the ceiling is not a measured rate.
More requests. More throughput?
Click a batch size to try it. Everything else stays the same.
OPTIONAL EXPERIMENTSplit the model across GPUs
Tensor parallelism shards weights and attention heads across identical GPUs. Synchronization adds time.
Illustrative effective, one-way link rates—not advertised bidirectional totals. Even head sharding only; unsupported GPU counts are disabled. Two ring all-reduces per layer; no communication overlap or topology modeling.
EXPERIMENT 02 / THE SCHEDULER
Don’t let empty slots wait.
See how real arrival timing, prompt work and memory limits affect a stream of requests.
Start simpler: compare four slots, one iteration at a time
Eight requests. Equal-duration decode iterations. This small illustration isolates slot reuse.
Static batching waits for the entire batch to finish before admitting more requests. Continuous batching reuses a freed slot on the next iteration. Here every iteration takes the same time and prompt admission is free; real prefill, KV limits, and scheduling overhead can reduce or reverse the benefit. Hardware controls above do not change this illustration.
UNDER THE HOOD
A useful approximation.
An honest set of limits.
Use this lab to build intuition, then benchmark your actual model, GPU, and serving engine.
How to read the 3D room +
Inference uses learned weights; training updates them. Click the room labels to inspect weights, GPU compute, cache and interconnect. Drag the room to orbit, or use its camera buttons.
Click actual objects in the 3D scene or use the accessible object buttons. The researcher walks to the selected station. Object controls run requests, generate tokens, release KV memory, quantize weights and add GPU shards. The blocks above the workbench are a magnified memory illustration, not separate physical servers. During inference they show KV cache; during training they represent saved activations. Animation time is deliberately slowed and is not measured latency. Up to eight GPU towers represent the selected tensor-parallel devices; glowing paths show conceptual data flow, not a physical network topology.
Training follows examples → forward pass → loss → backward pass → optimizer update. A one-weight toy classifier calculates the probability of “blue” versus “red”: probability = sigmoid(weight), loss = −log(probability), gradient = probability − 1. The optimizer subtracts 0.5 × gradient from the weight. This is real arithmetic for one teaching example; no LLM runs and it predicts no real-world model quality. The toy has no relation to the inference model’s parameter count. The example assumes 18 bytes per parameter: 2-byte model copy + 4-byte master weights + 4-byte gradients + 8-byte Adam states, before activations and runtime overhead. Different optimizers and training methods use different amounts.
The calculator remains an inference model in both scene modes. Training does not use the inference precision selector. The traffic experiment below uses the same hardware assumptions and adds arrival times, memory admission and three scheduling policies. The simpler slot-only illustration remains available beneath it.
How the numbers are calculated +
Weights = parameters × bytes per weight. For example, 8 billion weights at 2 bytes each need 16 GB. INT8/INT4 include an assumed 5% metadata overhead.
KV cache = 2 × layers × KV heads × head width × context × batch × bytes per KV value. The 2 means keys and values. Each request keeps its own cache. Allocation reserves the full prompt + output budget.
Stage time ≈ max(math time, data-movement time). Like a kitchen: cooking faster helps only when ingredients arrive fast enough. We add the matmul and attention stages, communication, and 0.1 ms overhead per pass.
Math time = FLOPs ÷ effective FLOP/s. Data-movement time = bytes ÷ effective bandwidth. Effective rates apply your utilization percentages.
Arithmetic intensity = FLOPs ÷ bytes moved. The roofline knee is effective FLOP/s ÷ effective bytes/s. More reuse moves an operation toward the compute ceiling.
Show operation-count assumptions
Dense linear work: 2 × parameters × processed tokens. Causal prefill attention: 2 × layers × batch × prompt × (prompt + 1) × model width. Decode attention: 4 × layers × batch × current context × model width. Width = query heads × head width.
Linear activation traffic: 4 × layers × processed tokens × width bytes, plus weights. Attention traffic includes KV reads/writes and 4 × layers × processed tokens × width bytes. This simplified fused-attention estimate omits re-reads and other kernel traffic.
Workspace reserve: 8 BF16 activation buffers (8 × batch × prompt × width × 2 bytes), evenly sharded for TP. Runtime reserve is per GPU. These are adjustable planning assumptions, not allocator predictions.
What each latency metric means +
TTFT: time to first token, approximated by prefill; excludes network and queue wait.
TPOT: average time between output tokens after the first. We calculate each of the output − 1 decode steps as context grows. A one-token response has no TPOT.
Output throughput: batch × output tokens ÷ complete batch time, including prefill. Decode-only throughput is batch ÷ TPOT. Per-request generation speed is 1 ÷ TPOT.
End-to-end: TTFT + (output − 1) × TPOT. All requests in the hardware simulation have equal prompt and output lengths and arrive together.
What this version leaves out +
Dense causal transformers with conventional MHA/GQA only. No MoE, MLA, speculative decoding, prefix caching, sliding-window attention, CPU offload, distributed prefill, or real GPU execution.
Quantization changes storage and data traffic, not the configured compute rate or model quality. Activation precision is fixed to BF16 in the traffic and workspace estimates.
Memory fit is estimated, with no allocator fragmentation, KV paging, temporary quantization buffers, or quadratic attention workspace. Long prompts assume fused attention. An out-of-memory result suppresses performance metrics.
Tensor parallelism assumes balanced sharding, supported head counts, identical GPUs, and no overlap of communication with computation. NVLink speeds GPU-to-GPU exchange; it does not increase a GPU’s HBM bandwidth.
Configuration and pinned baseline are saved in this browser. Reset clears both. No model calls, tokens, or GPU rentals are used.