I thought keeping one GPU warm would solve cold starts. It doesn't. Memory snapshots, queue depth, and the 2-minute wall are a different problem entirely.
When you're dealing with massive AI rendering tasks, cold starting a container on a cloud provider like Modal or RunPod takes tens of seconds, sometimes minutes. If you're building a real-time system, that's unacceptable. So, the obvious solution is to keep a pool of GPUs warm.
However, what the documentation doesn't tell you is that "warm" doesn't necessarily mean "ready to compute immediately". Even if the container is running, the model weights might need to be transferred from CPU RAM to VRAM, or the CUDA context might need to be initialized on the first request. I ran into this wall hard when building Prometheus. We had paying users staring at a loading spinner because the keep_warm instance was warm in name only.
The Memory Snapshot Pivot
The real solution involved a hybrid approach. We had to implement persistent memory snapshots and aggressively manage our queue depth to ensure requests were routed only to instances that had already warmed their CUDA contexts. We also had to implement a Temporal-backed durable execution layer just to handle the inevitable timeouts when the 2-minute wall hit.
It was a brutal week of debugging, but it fundamentally changed how I view GPU orchestration in the cloud. You can't just trust the orchestrator's definition of "ready". You have to measure it yourself at the hardware level.