Mistral Large 4 (Le Chonk): Run Locally

Guide to running Mistral Large 4 GGUFs locally with Unsloth and llama.cpp.

Mistral Large 4 (Le Chonk) is Mistral’s 1 trillion parameter MoE model with 52B active parameters (including embeddings and output layers). Mistral Large 4 is the strongest Western open model to date and supports text and image inputs, a 1M-context window, 160+ languages, and reasoning. You can run Le Chonk locally with Unsloth Dynamic GGUFs via Unsloth or llama.cpp. GGUF: Mistral-Large-4-1T-A52B-GGUF

Dynamic 1bit retains 91% of top-1 accuracy across coding, agentic tasks, math and other areas whilst being 80% smaller (223GB vs 1.1TB FP8). Dynamic 2bit retains 95% top-1 accuracy whilst 66% smaller (378GB). Thank you Mistral for day zero access.

Run Mistral Large 4 TutorialsQuant Analysis

⚙️ Usage Guide

The smallest Dynamic 1-bit quant, UD-IQ1_S, uses approximately 233GB of disk space which should fit on 256GB RAM including a Mac Studio or 2x NVIDIA DGX Sparks.

Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory)

1-bit	2-bit	3-bit	4-bit	8-bit
256–270GB	310–390GB	435–500GB	630GB	1.2TB

For best performance, your total available RAM + VRAM, or unified memory, should exceed the quant’s file size with room for the operating system, KV cache and runtime buffers. Longer contexts need more memory.

Recommended Settings

Mistral recommends the same sampling settings for all reasoning efforts:

Temperature	Top-p
1.0	0.95

• Reasoning efforts = low, medium, and high

• Maximum context window is 1,048,576 tokens.

We also ran KLD benchmarks for all quants - more details in Quantization Analysis

Run Mistral Large 4 Tutorials:

The examples below use UD-Q2_K_XL for text inference. You can change the quantization type to any available quant in the GGUF repository.

🦥 Run Mistral Large 4 in Unsloth

Unsloth Desktop runs GGUFs locally on macOS, Windows and Linux, automatically offloads to RAM and detects multi-GPU setups.

1

Install Unsloth

The easiest way to get started is by downloading the Unsloth Desktop app. Works on macOS, Windows, and Linux.

Download Unsloth

• Download for macOS

• Download for Windows

• Download for Linux

Or, if you prefer to install manually:

MacOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows PowerShell:

irm https://unsloth.ai/install.ps1 | iex

2

Search and download Mistral Large 4

Go to Unsloth Chat or Model hub, search for Mistral-Large-4-1T-A52B and select unsloth/Mistral-Large-4-1T-A52B-GGUF. Download your desired quant using the sizes and memory requirements above.

3

Run Mistral Large 4

Mistral Large 4 uses high reasoning effort by default. Use low for fast chat, medium for everyday tasks, or high for complex prompts and agentic coding. Unsloth automatically selects the best configurations for you but you can still adjust manually.

For more information, you can view our Unsloth inference guide.

4

Serve Mistral Large 4 with Unsloth API

You can use the unsloth run command to serve Mistral Large 4 via an API using llama-server runtime flags, including context sizing, GPU layers, threading and sampling. For more info see our API docs or unsloth start.

unsloth run --model unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-Q2_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 0 \
    --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":"high"}'

Use a runtime with Mistral Large 4 support.

5

Unsloth is now ready

You can connect Mistral Large 4 to your tools through Unsloth Desktop. Unsloth also supports the following workflows:

• Connect tools: Claude Code, Codex, web search, MCP and more

• Train models: Fine-tune other supported text, diffusion, embedding, and more

• Generate media: Create and train images, video, TTS locally

Unsloth Desktop’s Agents settings (interface example). Select Mistral Large 4 and your downloaded quant when connecting a tool.

Quantization Analysis

We ran KLD and top-1% benchmarks across a variety of areas like tool calling, coding and agentic long traces, and show that Unsloth dynamic quantization (UD3) can retain a lot of accuracy whilst still making the model smaller.

We also ran KLD and provide it per corpus as well:

Quant	Size	Mean KLD (7 categories)	Top-1 agreement
UD-IQ1_S	233 GB	0.1263	90.6%
UD-IQ1_M	260 GB	0.0919	92.3%
UD-IQ2_XXS	302 GB	0.0566	93.9%
UD-Q2_K_XL	378 GB	0.0298	95.6%
UD-IQ3_XXS	424 GB	0.0217	96.2%
UD-Q3_K_XL	493 GB	0.0162	96.6%
UD-Q4_K_XL	605 GB	0.0071	97.8%
UD-Q5_K_XL	784 GB	0.0038	98.5%
UD-Q6_K_XL	890 GB	0.0032	98.6%
UD-Q8_K_XL	1115 GB

🦙 Run Mistral Large 4 in llama.cpp

Use a llama.cpp build with Mistral Large 4 support. The commands below use the UD-Q2_K_XL quant (approximately 378GB).

1

Install llama.cpp

Obtain llama.cpp on GitHub, or build it below. On Linux with an NVIDIA GPU, install the CUDA toolkit and build dependencies first:

sudo apt-get update
sudo apt-get install -y build-essential cmake curl libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server

For CPU or Apple Metal, use -DGGML_CUDA=OFF. Metal support is enabled by default on macOS. For Windows and other GPU backends, follow the repository’s build instructions.

2

Download and run Mistral Large 4

This command downloads the quant and starts a local chat server:

./llama.cpp/build/bin/llama-server \
    -hf unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-Q2_K_XL \
    --ctx-size 32768 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 0 \
    --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":"high"}' \
    --host 127.0.0.1 --port 8080

Open http://127.0.0.1:8080 in your browser to chat. The API is available at http://127.0.0.1:8080/v1.

The example uses a 32K context to limit KV-cache memory. --top-k 0 and --min-p 0.0 disable additional sampling filters so the recommended temperature and top-p settings apply. If you run out of memory, lower --ctx-size or choose a smaller quant.

3

Optional: download manually

Install the Hugging Face CLI and download only the quant you want:

pip install -U huggingface_hub
hf download unsloth/Mistral-Large-4-1T-A52B-GGUF \
    --local-dir unsloth/Mistral-Large-4-1T-A52B-GGUF \
    --include "UD-Q2_K_XL/*"

Keep all 9 shards in the same directory. In the server command above, replace -hf unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-Q2_K_XL with:

--model unsloth/Mistral-Large-4-1T-A52B-GGUF/UD-Q2_K_XL/Mistral-Large-4-1T-A52B-UD-Q2_K_XL-00001-of-00009.gguf

llama.cpp loads the remaining shards automatically. To use Dynamic 1-bit instead, download UD-IQ1_S/* and point --model at Mistral-Large-4-1T-A52B-UD-IQ1_S-00001-of-00006.gguf in that folder.

Benchmarks

More benchmarks are listed in Mistral's official blog.