How much memory do I need?
At 4-bit, budget a bit over half a GB per billion parameters. At 8-bit, budget a bit over one GB per billion.
| Model size | FP8 (8-bit) | FP4 (4-bit) |
|---|---|---|
| 3B | ~3.6 GB | ~1.8 GB |
| 8B | ~9.6 GB | ~4.8 GB |
| 14B | ~16.8 GB | ~8.4 GB |
| 32B | ~38.4 GB | ~19.2 GB |
| 70B | ~84.0 GB | ~42.0 GB |
Estimates assume a 4096-token context. Longer contexts need more memory for the KV cache.
Then compare that against usable memory: a 16 GB Mac has about 12 GB, and a 24 GB GPU has about 21.6 GB. Mac or GPU? · Look up a model