64GB vs 128GB RAM for local LLM inference

Local inference has changed shape over the past twelve months. The question is no longer whether a desktop can run a serious model, but which desktop, and how much system memory sits behind the GPU. Mixture of experts weights and long context windows have pushed system RAM back into the conversation. This piece walks through when 64GB is enough, when 128GB pays for itself, and why the choice of platform matters as much as the capacity number.

The workload has changed

Two years ago a 24GB GPU and 32GB of system RAM was a comfortable local rig. A dense 13B model in 4 bit fitted on the card, the OS sat in system memory, and nobody thought about swap. That envelope no longer covers the interesting workloads.

Mixture of experts checkpoints such as Mixtral 8x22B, Qwen3 235B A22B, and the DeepSeek V3 family only activate a fraction of parameters per token, but the full weight set still has to be resident somewhere. Quantised to 4 bit, a 235B parameter MoE lands around 130GB. An 8x22B in Q4 sits near 80GB. Very little of that fits on consumer GPUs, so the tail of the weights spills into system RAM.

Long context is the second pressure point. A 128k context KV cache for a 70B class model in FP16 can consume 20GB to 40GB depending on head dimensions and grouped query attention settings. Context does not compress the way weights do.

Where 64GB is genuinely enough

64GB covers the majority of practical local setups on a single desktop.

Dense 13B, 34B and 70B models in 4 bit or 5 bit quantisation fit comfortably. A 70B in Q4_K_M is around 40GB on disk, and llama.cpp will map that into 64GB with the OS, an IDE, and a browser stack, leaving room for a modest KV cache. Up to 32k context on a 70B is fine. Coding agents, retrieval augmented pipelines, and Whisper transcription all sit inside this envelope.

For a builder targeting Llama 3.3 70B, Qwen 2.5 72B, or the Mistral Small and Medium tiers, 64GB paired with a 16GB or 24GB GPU is the sensible floor. Spending on 128GB when the working set is 48GB is money that would do more work as a second NVMe or a step up in GPU tier.

Where 128GB starts to earn its keep

Three workloads flip the maths.

First, MoE offload. Running Mixtral 8x22B or a quantised DeepSeek V3 with partial GPU offload only works if the CPU side has room for weights plus KV cache plus the OS. 96GB is tight, 128GB is comfortable, and the throughput difference between "fits in RAM" and "touches swap" is orders of magnitude.

Second, long context on large models. A 70B at 128k or 200k context needs KV cache headroom that a 64GB build does not have once weights, OS, and application memory are accounted for. Prompt caching across sessions compounds the demand.

Third, concurrent workloads. Fine tuning a smaller model with QLoRA while a larger model serves inference, or an agent stack running multiple model instances, chews through RAM in a way single session benchmarks never show.

If any of those three patterns fit the intended use, 128GB is the correct answer.

See the current BAV 128GB desktop builds on eBay

Server class DDR4 vs consumer DDR5

Capacity is only half the story. Bandwidth is what feeds tokens.

Consumer DDR5 on a modern AM5 or LGA1700 board runs dual channel at 5600 to 6400 MT/s, delivering roughly 90 to 100 GB/s of theoretical bandwidth. Fast, but two channels only.

Server class DDR4 on Xeon Scalable, EPYC Rome, or EPYC Milan platforms runs at 2933 or 3200 MT/s across six or eight channels. An eight channel EPYC config at 3200 MT/s pushes past 200 GB/s theoretical. For CPU offload of MoE weights, that channel count is worth more than the higher per stick speed of a consumer board. Token generation on CPU is memory bandwidth bound, not compute bound.

The trade off is platform cost and heat. A used EPYC 7003 or Xeon Gold 6338 workstation is serious kit, and if the plan is heavy CPU offload it is the right kit. For hybrid workloads where the GPU does most of the work, consumer DDR5 with 128GB in four sticks is the pragmatic choice.

The BAV 64GB desktop recommendation

For a first serious local inference build, the BAV 64GB desktop matches how most people actually use these tools. Dense models up to 70B in Q4, coding agents, RAG pipelines, and 32k context on a 70B all live inside 64GB. The build pairs that memory with a modern platform and NVMe primary storage so weights load in seconds.

For workloads specifically targeting MoE at scale or long context on large models, the same chassis line steps up to 128GB without changing the underlying platform. The upgrade is a swap, not a rebuild.

FAQ

Can I run a 70B model on 64GB of RAM?

Yes, in 4 bit or 5 bit quantisation, with room for a 32k context KV cache and normal OS overhead. A Q4_K_M 70B is roughly 40GB, leaving around 20GB for context, applications, and the OS. Push beyond 32k context or run alongside other heavy applications and you will feel the ceiling.

Does 128GB make token generation faster?

Not directly. Bandwidth and channel count drive token throughput, not capacity. Where 128GB helps is by keeping the full working set resident so the machine never has to page from NVMe, which is where real slowdowns hit. On a bandwidth constrained consumer platform, 64GB well fed beats 128GB half used.

Is server DDR4 slower than consumer DDR5 for LLMs?

Per stick, yes. Aggregate, no. Six or eight channels of DDR4 3200 delivers more sustained bandwidth than dual channel DDR5 6400, and CPU inference is bandwidth bound. For pure GPU inference the system memory speed matters much less, so consumer DDR5 is fine.

Do I need ECC memory for local inference?

Not required, but sensible for long running or unattended workloads. Silent bit flips in a KV cache produce weird tokens rather than crashes, which is harder to debug than an outright fault. Workstation and server platforms give ECC as standard, most consumer boards do not.

About Birmingham AV

We are a Bromsgrove, Worcestershire refurbisher with 87,000 items sold on eBay since 2017 and 24,756 buyer feedbacks at 98.9% positive. Every desktop and workstation ships with a twelve month warranty. Companies House 12383651. VAT GB 348755066. One of the highest volume refurbished PC operations on eBay UK.