Mistral Small 3 on a consumer GPU: complete guide

Mistral Small 3 sits at 24 billion parameters, which used to mean data centre hardware. In 2026 it fits on a single consumer GPU, provided the right quantisation matches the VRAM on the card. This guide covers the three cards buyers ask about most often and why the RTX 5070 Ti has quietly become the sensible pick for local inference.

Why Mistral Small 3 matters for local inference

Mistral Small 3 24B is a dense, non reasoning model tuned for low latency responses. It targets the same use cases as GPT class assistants: coding help, document Q and A, agentic tool use, and structured extraction. Prompts and outputs stay on your own hardware, there is no per token pricing, and first token latency drops below one second.

At full FP16 the model needs roughly 48GB of VRAM for weights alone, before the key value cache. That rules out consumer GPUs at native precision. Quantisation is the game.

The three consumer cards worth considering

The RTX 4090 ships with 24GB of GDDR6X on a 384 bit bus. It remains the fastest single card option for many local inference workloads, particularly where memory bandwidth matters. Used and refurbished pricing varies with condition and warranty.

The RTX 5080 ships with 16GB of GDDR7 on wider effective bandwidth than the 5070 Ti. It is the more expensive Blackwell card, and for pure Mistral Small 3 inference the advantage is narrower than the price gap suggests.

The RTX 5070 Ti ships with 16GB of GDDR7 on a 256 bit bus. It hits the sweet spot: current generation architecture, sensible 300W power draw, and a price that leaves budget for the rest of the build.

Quantisation trade offs in plain terms

Quantisation compresses model weights from FP16 to lower bit depths. Common targets are 8, 6, 5, 4, and 3 bit. Each step down reduces the memory footprint at some cost to quality.

At Q8, Mistral Small 3 needs roughly 25GB for weights plus 2GB to 4GB for context. It does not fit cleanly on any consumer card without offload to system RAM, which tanks throughput.

At Q6_K, the model drops to around 20GB. It fits on a 24GB RTX 4090 with a small context window but leaves little headroom. Quality is essentially indistinguishable from FP16.

At Q5_K_M, footprint falls to roughly 17GB. This is the target for a 24GB card with a generous 8K to 16K context. Quality loss is typically under one percent on MMLU and coding benchmarks.

At Q4_K_M, the model needs about 14GB to 15GB. This is the practical target for 16GB cards, both the RTX 5070 Ti and RTX 5080. That leaves 1GB to 2GB free for the KV cache and supports 8K context comfortably.

At Q3_K_M and below, quality degradation shows up in code generation and long form reasoning. Avoid unless the card is 12GB.

Real throughput numbers

Running Mistral Small 3 Q4_K_M on an RTX 5070 Ti through llama.cpp produces 55 to 65 tokens per second at batch size one with a 4K prompt. The RTX 5080 at the same quantisation sits at 65 to 75 tokens per second. An RTX 4090 running Q5_K_M lands in the same 65 to 75 range, sometimes higher on longer sequences.

GPT class API endpoints typically deliver 40 to 80 tokens per second on similar model sizes. A local 5070 Ti build matches or beats hosted inference for a single user, with zero cost per token. Birmingham AV RTX 5070 Ti systems ship with 32GB DDR5, a 1TB NVMe, a modern Ryzen or Core Ultra CPU, and a twelve month warranty. Spec and pricing is on the build listing.

Why the RTX 5070 Ti is the pragmatic pick

The RTX 4090 is faster in absolute terms and has more VRAM headroom for Q5. It is also harder to source at sensible prices in 2026, and power draw sits closer to 450W under sustained inference.

The RTX 5080 delivers slightly higher throughput than the 5070 Ti, but the price premium rarely pays back for local LLM buyers. If the second job is high refresh 4K gaming, the 5080 makes more sense. For inference alone it does not.

The RTX 5070 Ti hits the price to performance sweet spot. 16GB of GDDR7 is enough for Q4_K_M with room for context, the 300W draw fits standard 750W to 850W PSUs, and Blackwell tensor cores are current generation.

Software stack and setup notes

llama.cpp with CUDA remains the most widely used runtime for GGUF quantised models. Ollama wraps llama.cpp and is easier for first time users. vLLM is faster for batched serving but requires FP16 or FP8 weights, which pushes back to a 24GB card. On an RTX 5070 Ti, Ollama with the mistral-small3 Q4_K_M tag is the shortest path from unboxing to prompt. Expect 30 to 45 seconds for first model load, then sub 500ms first token latency thereafter.

FAQ

Can Mistral Small 3 run on an RTX 4070 with 12GB?

Only at Q3, and quality suffers in code generation. For chat and summarisation it can work, but the 5070 Ti at 16GB is materially better and worth the step up for anyone serious about local inference.

How much system RAM is needed alongside the GPU?

32GB DDR5 is enough for pure GPU inference with Mistral Small 3. Plans to offload layers to system memory or run multiple models push that to 64GB for useful headroom. 128GB is only worth it for MoE models or serving multiple users.

Does the model work well for code generation?

Yes, particularly at Q5 or higher. Mistral Small 3 scores competitively with GPT-4o Mini on HumanEval and MBPP for common languages including Python, TypeScript, and Go. At Q4_K_M on a 5070 Ti it produces working code fast enough for interactive use.

Is a used RTX 4090 better value than a new RTX 5070 Ti?

For LLM inference specifically the answer is usually no. A refurbished 5070 Ti with a proper warranty costs less, draws less power, and runs Mistral Small 3 Q4_K_M with plenty of throughput. The 4090 wins for larger models above 30B parameters, not for this workload.

About Birmingham AV

We have sold 87,000 items on eBay since 2017 with 24,756 buyer feedbacks at 98.9% positive. Every system ships with a twelve month warranty, tested under sustained load before it leaves the workshop. We are Companies House 12383651, VAT GB 348755066, based in Bromsgrove, Worcestershire, and one of the highest volume refurbished PC operations on eBay UK.