12 hours of training. 4GB VRAM. The model crashed on frame 3. Here's the cost analysis and the pivot.
When I first started working on the RECAM Masters model for Prometheus, I was stubbornly convinced that I could optimize the architecture enough to run on commodity hardware. I spun up an instance with an NVIDIA T4, thinking I was being incredibly cost-efficient.
The reality hit me about 12 hours into the first test run. The VRAM peaked, the CUDA out-of-memory error killed the process, and I was left staring at a completely useless checkpoint file. 6 Degrees of Freedom (6DOF) video matting requires depth-aware subject extraction across multiple temporal frames. You are essentially asking the model to hold the context of 3D space in memory while predicting alpha channels. A T4 simply doesn't have the memory bandwidth or capacity for it.
The Financial Reality of Scale
Switching to an H100 was financially painful but technically revelatory. The training run that took 12 hours (before crashing) on the T4 completed in under 45 minutes on the H100. More importantly, the massive VRAM allowed me to increase the batch size, leading to significantly better gradient stability.
The lesson here wasn't just "buy bigger GPUs." It was about understanding the fundamental constraints of the architecture you're building. Sometimes, being cheap with infrastructure is the most expensive mistake you can make in terms of engineering time. Optimize the model, absolutely, but know when hardware is the only answer.