Quantization, in plain words
A model is billions of numbers. Quantization stores each number with fewer bits, so the model takes less memory at a small cost in quality.
Short version: download a 4-bit build such as Q4_K_M first. It is roughly a quarter of the full-precision size and is what most people run. Move up to 8-bit if you have spare memory.
Every format we track
| Format | Bits | 8B model needs (est.) | Recommended for |
|---|---|---|---|
| FP32 | 32 | ~38.4 GB | Training, research, or when maximum accuracy is critical. Requires high-end hardware like A100 GPUs or professional workstations. |
| FP16 | 16 | ~19.2 GB | Standard choice for GPU users. Works well on RTX 4090, H100, or other modern GPUs with good speed and reasonable memory requirements. |
| BFLOAT16 | 16 | ~19.2 GB | Excellent for TPUs and newer GPUs (H100, RTX 6000 Ada). Better training stability than FP16 while maintaining small memory footprint. |
| FP8 | 8 | ~9.6 GB | For GPU users with limited VRAM (RTX 3090, RTX 4060 Ti). Good quality maintained, but watch for slight accuracy reduction on specialized tasks. |
| INT8 | 8 | ~9.6 GB | CPU inference on servers, edge devices, or GPUs with space constraints. Slightly faster than FP8 for integer operations. |
| FP4 | 4 | ~4.8 GB | MacBook Pro users or laptops with 8-16GB unified memory. Budget gaming GPUs like RTX 2060. Popular for experimentation at extreme compression. |
| INT4 | 4 | ~4.8 GB | Cost-effective inference at scale. Works on mobile devices, edge hardware, and consumer laptops. Most popular for local LLM use. |
| NF4 | 4 | ~4.8 GB | Best 4-bit choice for quality. Good for fine-tuning on consumer hardware. Popular with tools like llama.cpp and Ollama on Mac/Windows. |
| INT2 | 2 | ~2.4 GB | Research and experimentation only. May produce lower quality outputs. Only use if memory is extremely limited or for non-critical tasks. |
Estimates include headroom for the KV cache. Methodology