Meta Llama (Large Models) · open weights

Llama 3 70B

70.6B params8,192 ctx

Family

Meta Llama

Lab

Meta Llama (Large Models)

Parameters

70.6B

Max context

8,192 tokens

Quick answer: what should I run it on?

Estimated at a 4096-token context. New to quantization? Read the plain-words guide.

How much memory it needs

A quantization is the same model stored with fewer bits per number. Fewer bits use less memory but lose some output quality. Each card shows the estimated memory for Llama 3 70B, including KV-cache headroom at a 4096-token context.

• FP3232-bit

  ~338.9 GB est.

  full FP32 size · full quality, largest size

  Full precision with 32 bits per number. Uses maximum memory but provides highest accuracy. A 7B model needs ~28GB RAM.

  Good for Training, research, or when maximum accuracy is critical. Requires high-end hardware like A100 GPUs or professional workstations.

  Runs on 0 of 18 popular devices

• FP1616-bit

  ~169.4 GB est.

  50% of the FP32 size · full quality, largest size

  Half precision with 16 bits per number. Uses 50% less memory than FP32 with minimal quality loss. A 7B model needs ~14GB RAM.

  Good for Standard choice for GPU users. Works well on RTX 4090, H100, or other modern GPUs with good speed and reasonable memory requirements.

  Runs on 0 of 18 popular devices

• BFLOAT1616-bit

  ~169.4 GB est.

  50% of the FP32 size · full quality, largest size

  Brain floating point with 16 bits per number. Uses 50% less memory than FP32, optimized for modern AI hardware. A 7B model needs ~14GB RAM.

  Good for Excellent for TPUs and newer GPUs (H100, RTX 6000 Ada). Better training stability than FP16 while maintaining small memory footprint.

  Runs on 0 of 18 popular devices

• FP88-bit

  ~84.7 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  8-bit precision using ~25% of FP32 memory. Aggressive compression with good quality trade-off. A 7B model needs ~7GB RAM.

  Good for For GPU users with limited VRAM (RTX 3090, RTX 4060 Ti). Good quality maintained, but watch for slight accuracy reduction on specialized tasks.

  Runs on 0 of 18 popular devices

• INT88-bit

  ~84.7 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  Integer-based 8-bit quantization using ~25% of FP32 memory. Stable and efficient. A 7B model needs ~7GB RAM.

  Good for CPU inference on servers, edge devices, or GPUs with space constraints. Slightly faster than FP8 for integer operations.

  Runs on 0 of 18 popular devices

• FP44-bit

  ~42.4 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Ultra-aggressive 4-bit compression using only ~12.5% of FP32 memory. May see minor quality loss. A 7B model needs ~3.5GB RAM.

  Good for MacBook Pro users or laptops with 8-16GB unified memory. Budget gaming GPUs like RTX 2060. Popular for experimentation at extreme compression.

  Runs on 2 of 18 popular devices

• INT44-bit

  ~42.4 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Integer 4-bit quantization using ~12.5% of FP32 memory. Very efficient for inference. A 7B model needs ~3.5GB RAM.

  Good for Cost-effective inference at scale. Works on mobile devices, edge hardware, and consumer laptops. Most popular for local LLM use.

  Runs on 2 of 18 popular devices

• NF44-bit

  ~42.4 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Normalized 4-bit optimized for neural networks using ~12.5% of FP32 memory. Better quality than INT4 at same size. A 7B model needs ~3.5GB RAM.

  Good for Best 4-bit choice for quality. Good for fine-tuning on consumer hardware. Popular with tools like llama.cpp and Ollama on Mac/Windows.

  Runs on 2 of 18 popular devices

• INT22-bit

  ~21.2 GB est.

  6% of the FP32 size · very compressed, noticeable quality loss

  Extreme 2-bit compression using only ~6.25% of FP32 memory. Experimental and lossy. A 7B model needs ~1.75GB RAM.

  Good for Research and experimentation only. May produce lower quality outputs. Only use if memory is extremely limited or for non-critical tasks.

  Runs on 8 of 18 popular devices

Hatched bars are estimates. For comparison, a 24 GB GPU has about 21.6 GB usable.

Fit on popular hardware at FP4

Each bar shows how much of a device's usable memory this model takes at FP4, the format most people download first. The tick marks the usable limit, and a bar that reaches past it means the model does not fit.

• Intel Arc A770 8GB✕ Does not fit42.4 GB / 7.2 GB usable
• Qualcomm Snapdragon 8 Gen 3 Leading Version✕ Does not fit42.4 GB / 9.0 GB usable
• Qualcomm Snapdragon 8 Elite✕ Does not fit42.4 GB / 9.0 GB usable
• Mac Mini M4✕ Does not fit42.4 GB / 12.0 GB usable
• MacBook Pro M4 14-inch✕ Does not fit42.4 GB / 12.0 GB usable
• AMD Radeon RX 7800 XT✕ Does not fit42.4 GB / 14.4 GB usable
• Intel Arc A770 16GB✕ Does not fit42.4 GB / 14.4 GB usable
• Qualcomm Snapdragon X Elite✕ Does not fit42.4 GB / 12.0 GB usable
• AMD Radeon RX 7900 XT✕ Does not fit42.4 GB / 18.0 GB usable
• Mac Mini M4 Pro✕ Does not fit42.4 GB / 18.0 GB usable
• NVIDIA RTX 4090◐ Partial offload42.4 GB / 21.6 GB usable
• AMD Radeon RX 7900 XTX◐ Partial offload42.4 GB / 21.6 GB usable
• Mac Mini M4 32GB◐ Partial offload42.4 GB / 24.0 GB usable
• NVIDIA RTX 5090◐ Partial offload42.4 GB / 28.8 GB usable
• Mac Studio M5 Max◐ Partial offload42.4 GB / 27.0 GB usable
• MacBook Pro M4 Max 16-inch◐ Partial offload42.4 GB / 36.0 GB usable
• NVIDIA RTX 6000 Ada✓ Runs42.4 GB / 43.2 GB usable
• Mac Studio M5 Ultra✓ Runs42.4 GB / 72.0 GB usable

Every quantization × popular device

These verdicts are estimated at a 4096-token context. How fit is calculated

Not seeing your device? Check every device we track

Measured throughput

Other Meta Llama models

• Llama 3.2 1B
• Llama 3.2 3B
• Llama 2 7B
• Llama 3 8B
• Llama 3.1 8B
• Llama 2 13B
• Llama 3.1 70B
• Llama 2 70B
• Llama 3.1 405B
• Llama 3.3 8B
• Llama 3.3 70B

Similar-sized models from other families

• Qwen 2.5 72B (72B)
• Falcon 40B (40B)
• Qwen 2.5 32B (32B)
• Qwen 3 32B (32B)
• OLMo 3 32B (32B)
• Gemma 4 31B (31B)