TII Falcon · open weights

Falcon 40B

40B params2,048 ctx

Family

TII Falcon

Lab

TII Falcon

Parameters

40B

Max context

2,048 tokens

Quick answer: what should I run it on?

Estimated at a 4096-token context. New to quantization? Read the plain-words guide.

How much memory it needs

A quantization is the same model stored with fewer bits per number. Fewer bits use less memory but lose some output quality. Each card shows the estimated memory for Falcon 40B, including KV-cache headroom at a 4096-token context.

• FP3232-bit

  ~192.0 GB est.

  full FP32 size · full quality, largest size

  Full precision with 32 bits per number. Uses maximum memory but provides highest accuracy. A 7B model needs ~28GB RAM.

  Good for Training, research, or when maximum accuracy is critical. Requires high-end hardware like A100 GPUs or professional workstations.

  Runs on 0 of 18 popular devices

• FP1616-bit

  ~96.0 GB est.

  50% of the FP32 size · full quality, largest size

  Half precision with 16 bits per number. Uses 50% less memory than FP32 with minimal quality loss. A 7B model needs ~14GB RAM.

  Good for Standard choice for GPU users. Works well on RTX 4090, H100, or other modern GPUs with good speed and reasonable memory requirements.

  Runs on 0 of 18 popular devices

• BFLOAT1616-bit

  ~96.0 GB est.

  50% of the FP32 size · full quality, largest size

  Brain floating point with 16 bits per number. Uses 50% less memory than FP32, optimized for modern AI hardware. A 7B model needs ~14GB RAM.

  Good for Excellent for TPUs and newer GPUs (H100, RTX 6000 Ada). Better training stability than FP16 while maintaining small memory footprint.

  Runs on 0 of 18 popular devices

• FP88-bit

  ~48.0 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  8-bit precision using ~25% of FP32 memory. Aggressive compression with good quality trade-off. A 7B model needs ~7GB RAM.

  Good for For GPU users with limited VRAM (RTX 3090, RTX 4060 Ti). Good quality maintained, but watch for slight accuracy reduction on specialized tasks.

  Runs on 1 of 18 popular devices

• INT88-bit

  ~48.0 GB est.

  25% of the FP32 size · near-full quality, half the size of FP16

  Integer-based 8-bit quantization using ~25% of FP32 memory. Stable and efficient. A 7B model needs ~7GB RAM.

  Good for CPU inference on servers, edge devices, or GPUs with space constraints. Slightly faster than FP8 for integer operations.

  Runs on 1 of 18 popular devices

• FP44-bit

  ~24.0 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Ultra-aggressive 4-bit compression using only ~12.5% of FP32 memory. May see minor quality loss. A 7B model needs ~3.5GB RAM.

  Good for MacBook Pro users or laptops with 8-16GB unified memory. Budget gaming GPUs like RTX 2060. Popular for experimentation at extreme compression.

  Runs on 6 of 18 popular devices

• INT44-bit

  ~24.0 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Integer 4-bit quantization using ~12.5% of FP32 memory. Very efficient for inference. A 7B model needs ~3.5GB RAM.

  Good for Cost-effective inference at scale. Works on mobile devices, edge hardware, and consumer laptops. Most popular for local LLM use.

  Runs on 6 of 18 popular devices

• NF44-bit

  ~24.0 GB est.

  13% of the FP32 size · the usual sweet spot of size and quality

  Normalized 4-bit optimized for neural networks using ~12.5% of FP32 memory. Better quality than INT4 at same size. A 7B model needs ~3.5GB RAM.

  Good for Best 4-bit choice for quality. Good for fine-tuning on consumer hardware. Popular with tools like llama.cpp and Ollama on Mac/Windows.

  Runs on 6 of 18 popular devices

• INT22-bit

  ~12.0 GB est.

  6% of the FP32 size · very compressed, noticeable quality loss

  Extreme 2-bit compression using only ~6.25% of FP32 memory. Experimental and lossy. A 7B model needs ~1.75GB RAM.

  Good for Research and experimentation only. May produce lower quality outputs. Only use if memory is extremely limited or for non-critical tasks.

  Runs on 15 of 18 popular devices

Hatched bars are estimates. For comparison, a 24 GB GPU has about 21.6 GB usable.

Fit on popular hardware at FP4

Each bar shows how much of a device's usable memory this model takes at FP4, the format most people download first. The tick marks the usable limit, and a bar that reaches past it means the model does not fit.

• Intel Arc A770 8GB✕ Does not fit24.0 GB / 7.2 GB usable
• Qualcomm Snapdragon 8 Gen 3 Leading Version✕ Does not fit24.0 GB / 9.0 GB usable
• Qualcomm Snapdragon 8 Elite✕ Does not fit24.0 GB / 9.0 GB usable
• Mac Mini M4◐ Partial offload24.0 GB / 12.0 GB usable
• MacBook Pro M4 14-inch◐ Partial offload24.0 GB / 12.0 GB usable
• AMD Radeon RX 7800 XT◐ Partial offload24.0 GB / 14.4 GB usable
• Intel Arc A770 16GB◐ Partial offload24.0 GB / 14.4 GB usable
• Qualcomm Snapdragon X Elite◐ Partial offload24.0 GB / 12.0 GB usable
• AMD Radeon RX 7900 XT◐ Partial offload24.0 GB / 18.0 GB usable
• Mac Mini M4 Pro◐ Partial offload24.0 GB / 18.0 GB usable
• NVIDIA RTX 4090◐ Partial offload24.0 GB / 21.6 GB usable
• AMD Radeon RX 7900 XTX◐ Partial offload24.0 GB / 21.6 GB usable
• Mac Mini M4 32GB✓ Runs24.0 GB / 24.0 GB usable
• NVIDIA RTX 5090✓ Runs24.0 GB / 28.8 GB usable
• Mac Studio M5 Max✓ Runs24.0 GB / 27.0 GB usable
• MacBook Pro M4 Max 16-inch✓ Runs24.0 GB / 36.0 GB usable
• NVIDIA RTX 6000 Ada✓ Runs24.0 GB / 43.2 GB usable
• Mac Studio M5 Ultra✓ Runs24.0 GB / 72.0 GB usable

Every quantization × popular device

These verdicts are estimated at a 4096-token context. How fit is calculated

Not seeing your device? Check every device we track

Measured throughput

Other TII Falcon models

• Falcon 7B

Similar-sized models from other families

• Qwen 2.5 32B (32B)
• Qwen 3 32B (32B)
• OLMo 3 32B (32B)
• Gemma 4 31B (31B)
• Qwen 3 30B-A3B (MoE) (30B)
• Gemma 3 27B (27B)