The Quadratic Memory Wall: A Precise Analysis of GPU Memory Requirements per Context Length
Total GPU memory = model weights + KV cache + activations + workspace. Here's the exact formula to compute maximum context length for any GPU configuration.
// tag
Posts tagged "tensor parallelism"
1 post found