77,000
GPUs at the evening peak, before headroom
Assume 100M daily users × 20 messages, 58,000 a second at peak and 23M output tokens a second. New GPUs take months to buy.
How do tokens reach the phone?
Bad: Poll every 500 ms
1.6M mostly empty requests a second at peak, and jerky text.
Good: SSE over HTTP/2
One plain HTTP response with event IDs and resume built in.
Your phone drops from Wi-Fi to 4G mid-answer.
The orchestrator keeps writing tokens to a per-run Redis log. Reconnect with Last-Event-ID 31, and any gateway replays from 32.
The model remembers nothing. Every turn re-sends the whole conversation, so cost grows with length squared.
The trick
Keep the prompt’s start identical. Cached tokens bill at 0.1× (OpenAI, October 2026), cutting input cost 78%.
Edit an old message. What happens to the answer?
Nothing is overwritten. Edits and regenerations add siblings in a tree; one active leaf decides what you see and what the model reads.
30%
over capacity, 9:15 pm on launch day
Demand is 30M tokens/s against 23M. Assume the small model needs a fifth of the GPU time; move free users there and it’s 12M.
Safety without killing latency
- Classify the input before spending any GPU time
- Check output every 15 tokens, ≈ 0.65 s behind
- On a flag, stop the GPU and retract
- Watch accounts and behaviour, not just text
Remember
Generation lives in a per-run token log, not the connection. Prompts keep a stable prefix, chats are trees, and fair queues guard scarce GPUs.