A pre-launch checklist for production LLM inference

Before you point production traffic at a self-hosted inference stack, walk through this list.

The checklist

1. P99 latency under sustained load — not just cold p50.
2. Cold start behavior when autoscaling from zero.
3. KV cache eviction policy under memory pressure.
4. Request cancellation — does an aborted client actually free the GPU?
5. Token accounting for billing/quota accuracy.
6. Structured output failure modes (schema validation, retries).
7. Prompt injection surface on any tool-using agent.
8. Rollback plan — can you revert a model version in < 60s?

The one people forget

Request cancellation. Many inference servers happily keep generating tokens after the client disconnects. On a bursty workload that's 20–40% wasted GPU time.

What to monitor

• Tokens per second, split by prompt length bucket.
• Queue depth per replica.
• GPU memory headroom at p95.
• Time to first token vs total.