Mistral AI · open weights

Mistral Nemo 12B

12B params128,000 ctx

Family

Mistral AI

Lab

Mistral AI

Parameters

12B

Max context

128,000 tokens

Quick answer: what should I run it on?

Estimated at a 4096-token context. New to quantization? Read the plain-words guide.

How much memory it needs

A quantization is the same model stored with fewer bits per number. Fewer bits use less memory but lose some output quality. Each card shows the estimated memory for Mistral Nemo 12B, including KV-cache headroom at a 4096-token context.

• FP3232-bit

  ~57.6 GB est.

  full FP32 size · full quality, largest size

  Full precision with 32 bits per number. Uses maximum memory but provides highest accuracy. A 7B model needs ~28GB RAM.

  Good for Training, research, or when maximum accuracy is critical. Requires high-end hardware like A100 GPUs or professional workstations.

  Runs on 1 of 18 popular devices

• FP1616-bit

  ~28.8 GB est.

  50% of the FP32 size · full quality, largest size

  Half precision with 16 bits per number. Uses 50% less memory than FP32 with minimal quality loss. A 7B model needs ~14GB RAM.

  Good for Standard choice for GPU users. Works well on RTX 4090, H100, or other modern GPUs with good speed and reasonable memory requirements.

  Runs on 4 of 18 popular devices

• BFLOAT1616-bit

  ~28.8 GB est.

  50% of the FP32 size · full quality, largest size

  Brain floating point with 16 bits per number. Uses 50% less memory than FP32, optimized for modern AI hardware. A 7B model needs ~14GB RAM.

  Good for Excellent for TPUs and newer GPUs (H100, RTX 6000 Ada). Better training stability than FP16 while maintaining small memory footprint.

  Runs on 4 of 18 popular devices

• FP88-bit

  ~14.4 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  8-bit precision using ~25% of FP32 memory. Aggressive compression with good quality trade-off. A 7B model needs ~7GB RAM.

  Good for For GPU users with limited VRAM (RTX 3090, RTX 4060 Ti). Good quality maintained, but watch for slight accuracy reduction on specialized tasks.

  Runs on 12 of 18 popular devices

• INT88-bit

  ~14.4 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  Integer-based 8-bit quantization using ~25% of FP32 memory. Stable and efficient. A 7B model needs ~7GB RAM.

  Good for CPU inference on servers, edge devices, or GPUs with space constraints. Slightly faster than FP8 for integer operations.

  Runs on 12 of 18 popular devices

• FP44-bit

  ~7.2 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Ultra-aggressive 4-bit compression using only ~12.5% of FP32 memory. May see minor quality loss. A 7B model needs ~3.5GB RAM.

  Good for MacBook Pro users or laptops with 8-16GB unified memory. Budget gaming GPUs like RTX 2060. Popular for experimentation at extreme compression.

  Runs on 18 of 18 popular devices

• INT44-bit

  ~7.2 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Integer 4-bit quantization using ~12.5% of FP32 memory. Very efficient for inference. A 7B model needs ~3.5GB RAM.

  Good for Cost-effective inference at scale. Works on mobile devices, edge hardware, and consumer laptops. Most popular for local LLM use.

  Runs on 18 of 18 popular devices

• NF44-bit

  ~7.2 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Normalized 4-bit optimized for neural networks using ~12.5% of FP32 memory. Better quality than INT4 at same size. A 7B model needs ~3.5GB RAM.

  Good for Best 4-bit choice for quality. Good for fine-tuning on consumer hardware. Popular with tools like llama.cpp and Ollama on Mac/Windows.

  Runs on 18 of 18 popular devices

• INT22-bit

  ~3.6 GB est.

  6% of the FP32 size · very compressed, noticeable quality loss

  Extreme 2-bit compression using only ~6.25% of FP32 memory. Experimental and lossy. A 7B model needs ~1.75GB RAM.

  Good for Research and experimentation only. May produce lower quality outputs. Only use if memory is extremely limited or for non-critical tasks.

  Runs on 18 of 18 popular devices

Hatched bars are estimates. For comparison, a 24 GB GPU has about 21.6 GB usable.

Fit on popular hardware at FP4

Each bar shows how much of a device's usable memory this model takes at FP4, the format most people download first. The tick marks the usable limit, and a bar that reaches past it means the model does not fit.

• Intel Arc A770 8GB✓ Runs7.2 GB / 7.2 GB usable
• Qualcomm Snapdragon 8 Gen 3 Leading Version✓ Runs7.2 GB / 9.0 GB usable
• Qualcomm Snapdragon 8 Elite✓ Runs7.2 GB / 9.0 GB usable
• Mac Mini M4✓ Runs7.2 GB / 12.0 GB usable
• MacBook Pro M4 14-inch✓ Runs7.2 GB / 12.0 GB usable
• AMD Radeon RX 7800 XT✓ Runs7.2 GB / 14.4 GB usable
• Intel Arc A770 16GB✓ Runs7.2 GB / 14.4 GB usable
• Qualcomm Snapdragon X Elite✓ Runs7.2 GB / 12.0 GB usable
• AMD Radeon RX 7900 XT✓ Runs7.2 GB / 18.0 GB usable
• Mac Mini M4 Pro✓ Runs7.2 GB / 18.0 GB usable
• NVIDIA RTX 4090✓ Runs7.2 GB / 21.6 GB usable
• AMD Radeon RX 7900 XTX✓ Runs7.2 GB / 21.6 GB usable
• Mac Mini M4 32GB✓ Runs7.2 GB / 24.0 GB usable
• NVIDIA RTX 5090✓ Runs7.2 GB / 28.8 GB usable
• Mac Studio M5 Max✓ Runs7.2 GB / 27.0 GB usable
• MacBook Pro M4 Max 16-inch✓ Runs7.2 GB / 36.0 GB usable
• NVIDIA RTX 6000 Ada✓ Runs7.2 GB / 43.2 GB usable
• Mac Studio M5 Ultra✓ Runs7.2 GB / 72.0 GB usable

Every quantization × popular device

These verdicts are estimated at a 4096-token context. How fit is calculated

Not seeing your device? Check every device we track

Measured throughput

No one has submitted a measured benchmark for this model yet. The fit table above is an estimate.

Other Mistral AI models

• Mistral 7B
• Mistral Large 2 (MoE)
• Mistral Small 3 7B

Similar-sized models from other families

• Gemma 3 12B (12B)
• Llama 2 13B (13B)
• OLMo 2 13B (13B)
• Phi-4 (14B)
• Gemma 2 9B (9B)
• Llama 3 8B (8B)