Mistral AI · open weights

Mistral 7B

7B params8,192 ctx

Family: Mistral AI
Lab: Mistral AI
Parameters: 7B
Max context: 8,192 tokens

Quick answer: what should I run it on?

Cheapest popular Mac

Qualcomm Snapdragon 8 Gen 3 Leading Version

$0 · runs it at FP8, near-full quality, half the size of FP16.

Cheapest popular GPU

Intel Arc A770 8GB

$299 · runs it at FP4, the usual sweet spot of size and quality.

Estimated at a 4096-token context. New to quantization? Read the plain-words guide.

How much memory it needs

A quantization is the same model stored with fewer bits per number. Fewer bits use less memory but lose some output quality. Each card shows the estimated memory for Mistral 7B, including KV-cache headroom at a 4096-token context.

FP3232-bit
~33.6 GB est.
full FP32 size · full quality, largest size
Full precision with 32 bits per number. Uses maximum memory but provides highest accuracy. A 7B model needs ~28GB RAM.
Good for Training, research, or when maximum accuracy is critical. Requires high-end hardware like A100 GPUs or professional workstations.
Runs on 3 of 18 popular devices
FP1616-bit
~16.8 GB est.
50% of the FP32 size · full quality, largest size
Half precision with 16 bits per number. Uses 50% less memory than FP32 with minimal quality loss. A 7B model needs ~14GB RAM.
Good for Standard choice for GPU users. Works well on RTX 4090, H100, or other modern GPUs with good speed and reasonable memory requirements.
Runs on 10 of 18 popular devices
BFLOAT1616-bit
~16.8 GB est.
50% of the FP32 size · full quality, largest size
Brain floating point with 16 bits per number. Uses 50% less memory than FP32, optimized for modern AI hardware. A 7B model needs ~14GB RAM.
Good for Excellent for TPUs and newer GPUs (H100, RTX 6000 Ada). Better training stability than FP16 while maintaining small memory footprint.
Runs on 10 of 18 popular devices
FP88-bit
~8.4 GB est.
25% of the FP32 size · near-full quality, half the size of FP16
8-bit precision using ~25% of FP32 memory. Aggressive compression with good quality trade-off. A 7B model needs ~7GB RAM.
Good for For GPU users with limited VRAM (RTX 3090, RTX 4060 Ti). Good quality maintained, but watch for slight accuracy reduction on specialized tasks.
Runs on 17 of 18 popular devices
INT88-bit
~8.4 GB est.
25% of the FP32 size · near-full quality, half the size of FP16
Integer-based 8-bit quantization using ~25% of FP32 memory. Stable and efficient. A 7B model needs ~7GB RAM.
Good for CPU inference on servers, edge devices, or GPUs with space constraints. Slightly faster than FP8 for integer operations.
Runs on 17 of 18 popular devices
FP44-bit
~4.2 GB est.
13% of the FP32 size · the usual sweet spot of size and quality
Ultra-aggressive 4-bit compression using only ~12.5% of FP32 memory. May see minor quality loss. A 7B model needs ~3.5GB RAM.
Good for MacBook Pro users or laptops with 8-16GB unified memory. Budget gaming GPUs like RTX 2060. Popular for experimentation at extreme compression.
Runs on 18 of 18 popular devices
INT44-bit
~4.2 GB est.
13% of the FP32 size · the usual sweet spot of size and quality
Integer 4-bit quantization using ~12.5% of FP32 memory. Very efficient for inference. A 7B model needs ~3.5GB RAM.
Good for Cost-effective inference at scale. Works on mobile devices, edge hardware, and consumer laptops. Most popular for local LLM use.
Runs on 18 of 18 popular devices
NF44-bit
~4.2 GB est.
13% of the FP32 size · the usual sweet spot of size and quality
Normalized 4-bit optimized for neural networks using ~12.5% of FP32 memory. Better quality than INT4 at same size. A 7B model needs ~3.5GB RAM.
Good for Best 4-bit choice for quality. Good for fine-tuning on consumer hardware. Popular with tools like llama.cpp and Ollama on Mac/Windows.
Runs on 18 of 18 popular devices
INT22-bit
~2.1 GB est.
6% of the FP32 size · very compressed, noticeable quality loss
Extreme 2-bit compression using only ~6.25% of FP32 memory. Experimental and lossy. A 7B model needs ~1.75GB RAM.
Good for Research and experimentation only. May produce lower quality outputs. Only use if memory is extremely limited or for non-critical tasks.
Runs on 18 of 18 popular devices

Hatched bars are estimates. For comparison, a 24 GB GPU has about 21.6 GB usable.

Fit on popular hardware at FP4

Each bar shows how much of a device's usable memory this model takes at FP4, the format most people download first. The tick marks the usable limit, and a bar that reaches past it means the model does not fit.

Intel Arc A770 8GB✓ Runs4.2 GB / 7.2 GB usable
Qualcomm Snapdragon 8 Gen 3 Leading Version✓ Runs4.2 GB / 9.0 GB usable
Qualcomm Snapdragon 8 Elite✓ Runs4.2 GB / 9.0 GB usable
Mac Mini M4✓ Runs4.2 GB / 12.0 GB usable
MacBook Pro M4 14-inch✓ Runs4.2 GB / 12.0 GB usable
AMD Radeon RX 7800 XT✓ Runs4.2 GB / 14.4 GB usable
Intel Arc A770 16GB✓ Runs4.2 GB / 14.4 GB usable
Qualcomm Snapdragon X Elite✓ Runs4.2 GB / 12.0 GB usable
AMD Radeon RX 7900 XT✓ Runs4.2 GB / 18.0 GB usable
Mac Mini M4 Pro✓ Runs4.2 GB / 18.0 GB usable
NVIDIA RTX 4090✓ Runs4.2 GB / 21.6 GB usable
AMD Radeon RX 7900 XTX✓ Runs4.2 GB / 21.6 GB usable
Mac Mini M4 32GB✓ Runs4.2 GB / 24.0 GB usable
NVIDIA RTX 5090✓ Runs4.2 GB / 28.8 GB usable
Mac Studio M5 Max✓ Runs4.2 GB / 27.0 GB usable
MacBook Pro M4 Max 16-inch✓ Runs4.2 GB / 36.0 GB usable
NVIDIA RTX 6000 Ada✓ Runs4.2 GB / 43.2 GB usable
Mac Studio M5 Ultra✓ Runs4.2 GB / 72.0 GB usable

Every quantization × popular device

These verdicts are estimated at a 4096-token context. How fit is calculated

All hardware →

Quantization	Mac Mini M4	Mac Mini M4 32GB	Mac Mini M4 Pro	Mac Studio M5 Max	Mac Studio M5 Ultra	MacBook Pro M4 14-inch	MacBook Pro M4 Max 16-inch
FP32	✕ Does not fit	◐ Partial offload	◐ Partial offload	◐ Partial offload	✓ Runs	✕ Does not fit	✓ Runs
FP16	◐ Partial offload	✓ Runs	✓ Runs	✓ Runs	✓ Runs	◐ Partial offload	✓ Runs
BFLOAT16	◐ Partial offload	✓ Runs	✓ Runs	✓ Runs	✓ Runs	◐ Partial offload	✓ Runs
FP8	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs
INT8	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs
FP4	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs
INT4	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs
NF4	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs
INT2	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs	✓ Runs

Quantization	AMD Radeon RX 7900 XTX	AMD Radeon RX 7900 XT	AMD Radeon RX 7800 XT
FP32	◐ Partial offload	◐ Partial offload	✕ Does not fit
FP16	✓ Runs	✓ Runs	◐ Partial offload
BFLOAT16	✓ Runs	✓ Runs	◐ Partial offload
FP8	✓ Runs	✓ Runs	✓ Runs
INT8	✓ Runs	✓ Runs	✓ Runs
FP4	✓ Runs	✓ Runs	✓ Runs
INT4	✓ Runs	✓ Runs	✓ Runs
NF4	✓ Runs	✓ Runs	✓ Runs
INT2	✓ Runs	✓ Runs	✓ Runs

Quantization	Intel Arc A770 16GB	Intel Arc A770 8GB
FP32	✕ Does not fit	✕ Does not fit
FP16	◐ Partial offload	✕ Does not fit
BFLOAT16	◐ Partial offload	✕ Does not fit
FP8	✓ Runs	◐ Partial offload
INT8	✓ Runs	◐ Partial offload
FP4	✓ Runs	✓ Runs
INT4	✓ Runs	✓ Runs
NF4	✓ Runs	✓ Runs
INT2	✓ Runs	✓ Runs

Quantization	NVIDIA RTX 4090	NVIDIA RTX 6000 Ada	NVIDIA RTX 5090
FP32	◐ Partial offload	✓ Runs	◐ Partial offload
FP16	✓ Runs	✓ Runs	✓ Runs
BFLOAT16	✓ Runs	✓ Runs	✓ Runs
FP8	✓ Runs	✓ Runs	✓ Runs
INT8	✓ Runs	✓ Runs	✓ Runs
FP4	✓ Runs	✓ Runs	✓ Runs
INT4	✓ Runs	✓ Runs	✓ Runs
NF4	✓ Runs	✓ Runs	✓ Runs
INT2	✓ Runs	✓ Runs	✓ Runs

Quantization	Qualcomm Snapdragon 8 Gen 3 Leading Version	Qualcomm Snapdragon 8 Elite	Qualcomm Snapdragon X Elite
FP32	✕ Does not fit	✕ Does not fit	✕ Does not fit
FP16	◐ Partial offload	◐ Partial offload	◐ Partial offload
BFLOAT16	◐ Partial offload	◐ Partial offload	◐ Partial offload
FP8	✓ Runs	✓ Runs	✓ Runs
INT8	✓ Runs	✓ Runs	✓ Runs
FP4	✓ Runs	✓ Runs	✓ Runs
INT4	✓ Runs	✓ Runs	✓ Runs
NF4	✓ Runs	✓ Runs	✓ Runs
INT2	✓ Runs	✓ Runs	✓ Runs

Not seeing your device? Check every device we track

Measured throughput

Hardware	Quantization	Tokens/sec (measured)
RTX 4090	Q4_K_M	110