For local LLM inference, GPU memory capacity and bandwidth usually constrain the useful model size and throughput more than headline CPU specifications. A defensible workstation plan therefore starts with workload, quantization, context length, and VRAM requirements before choosing the rest of the platform. This guide explains those trade-offs and shows how to evaluate single- and multi-GPU configurations without treating volatile street prices or isolated benchmarks as permanent facts.
Every generated token forces the GPU to stream the model's entire weight set from VRAM to the compute units — and then do it all over again for the next token. During that decode loop, the shader cores spend most of their time waiting on memory, not crunching math. That's the mechanical reason inference is memory-bound: compute sets the ceiling for speed, but VRAM decides whether the model runs at all. The whole model has to live in VRAM, so when I evaluate a card for LLM work, capacity and bandwidth are the first two numbers I check — raw TFLOPS lands somewhere near the bottom of the list.
At FP16/BF16 precision, I budget roughly 2 bytes per parameter. That one rule gives me instant reality checks:
Full precision at 70B scale is datacenter territory. What makes local builds viable is quantization. At Q4 (4-bit), I plan for ~0.5–0.6GB per billion parameters — roughly a 75% memory reduction with quality loss I struggle to notice in everyday use. Measured footprints:
Between quantization levels the trade is consistent: Q5_K_M costs 15–20% more memory than Q4 for a modest quality bump, while Q8 preserves nearly all quality but only halves memory versus FP16. When VRAM is the scarce resource — and it always is — Q4 is where the value sits.
Weights are a fixed cost. The KV cache scales linearly with context length. For Llama-3-8B (32 layers, 8 KV heads, 128 head-dim, FP16), the math is 2 (K and V) × 32 layers × 8 heads × 128 dim × 2 bytes = 128KiB per token:
At long context, the cache rivals the weights themselves. Larger models pay more per token: Llama 3.3 70B burns ~160KB per token in FP16, or ~327MB for a single 2,048-token request. This is why I never spec a card to the exact model size — I add 2–8GB of headroom for context, OS overhead, and whatever else shares the card.
Running out of VRAM doesn't degrade performance politely. A model generating at 40 tok/s fully inside VRAM collapses to 2–3 tok/s the moment layers spill into system RAM, and paging out to storage is another 5–30x slower. There's no warning dialog — throughput just dies while the GPU sits starved.
That cliff is what locks in my component priority order for local LLM hardware:
Or, as I keep reminding myself while browsing spec sheets: buy the counter space, not the hand speed. A faster chef doesn't help if the counter is too small for the ingredients.
The GPU is the line item I defend first now, and the logic is simple: when inference is memory-bound, two numbers on the spec sheet decide almost everything. VRAM capacity determines whether a model can live on the card at all, and memory bandwidth determines how fast it produces tokens once it's there. Clock speeds, core counts, even the CPU you pair with it — all of that sits a tier below. So let me walk the actual 2026 ladder with that filter applied.
Between the 5090 and the pro tier, the L40S (48GB GDDR6, Ada Lovelace) is a genuinely cost-effective middle step, while the A100 (80GB) and H100 remain the enterprise reference points everything else gets measured against.
Numbers from the RTX 5090 tell the real story:
Two conclusions stand out to me. First, architecture choice — dense versus MoE — can swing performance more than the GPU itself. Second, bandwidth isn't a vanity spec: decode speed follows it almost linearly, which is why the 3090-to-5090 progression is worth paying for and the core-count progression mostly isn't.
Precision is the second axis, and spec sheets can mislead here. The ladder from irrelevant to essential:
The current generation pushes much further. Fifth-generation Blackwell Tensor Cores reach up to 5x Ada Lovelace throughput for FP8 training and INT4 inference, and the RTX PRO line delivers up to 4,000 TOPS with support spanning FP4/INT8 up to FP16/FP32. My caveat: those TOPS mostly get burned during compute-heavy phases like training and prompt processing. Single-token decode is bandwidth-bound, so a card with huge TOPS but modest bandwidth will feel fast at chewing through your prompt and slow at writing the answer.
AMD's pitch is straightforward: the RX 7900 XTX offers 24GB at a clearly lower price than its NVIDIA equivalent. The software story has genuinely improved — ROCm now supports PyTorch training, llama.cpp and vLLM both provide functional AMD acceleration, and Unsloth supports AMD through its Core library. The gap is maturity: community reports consistently land at 10–20% more debugging overhead versus CUDA, mostly in driver and kernel territory. And the detail I find most telling — on Strix Halo, Vulkan through llama.cpp actually outperforms ROCm for inference. When the unofficial backend beats the official one, you know exactly where the stack still stands. Intel Arc remains a footnote: SYCL-based llama.cpp support exists, but the drivers lag too far behind to bet a build on it.
The 2026 value picture:
Hardware price isn't the final word, though — ecosystem is. CUDA is the deepest stack, and vLLM, TensorRT-LLM, and NeMo all run best on it. Model formats make this worse: GGUF, MLX, and NVFP4 are not interchangeable, and each one locks you into particular backends. The rule I apply: pick the platform that runs your exact model and backend first, then buy the card that fits it — never the other way around.
With the ladder mapped and priced, the next decision follows directly: how much VRAM your target models — and their context windows — actually need.
CPU selection still matters, but for different reasons than model execution. The processor does not perform the GPU's heavy matrix math; it handles data preparation and preprocessing, orchestrates the system, and supplies the PCIe lanes the GPUs use. Too few lanes can leave an expensive multi-GPU configuration constrained by its host platform.
When I evaluate platforms now, lane count comes first:
One boundary worth marking: CPU-only inference is technically possible for 7B models and under, but token speeds fall so far on anything larger that it's impractical. The CPU feeds the GPU; it doesn't replace it.
System RAM matters more than most spec sheets admit. When a model outgrows VRAM, it spills into system memory, and 64GB+ of DDR5 becomes the difference between a degraded run and a dead one. The tiers I work from:
Two DIMM rules, non-negotiable:
Storage affects model loading time, not inference throughput — once the weights are in VRAM, drive speed stops mattering. 1TB NVMe is the entry-level minimum, 2–4TB on PCIe Gen4/Gen5 covers professional datasets, and multi-NVMe RAID belongs to enterprise pipelines.
This is also the part of the build where cutting costs is genuinely safe. A clean single-GPU platform — CPU, motherboard, RAM, PSU, case, cooling, and a 2TB NVMe — wraps into ~$800–$1,200 with a Ryzen 7 or i5-class CPU, 64GB DDR5, and a 1000W PSU. When the budget tightens, I apply one priority order and let it settle the arguments: VRAM > system RAM > NVMe SSD > CPU. The CPU is the one component where "good enough" is a strategy, not a compromise.
Making the budget defensible starts with lining the tiers up side by side, using 2026 street prices and measured throughput instead of vendor slides. The pattern I see across the whole spectrum is consistent: below roughly $5,000 you're trading speed for capacity, and above it you're mostly trading cash for both.
The cheapest build I'd defend today is a refurbished Dell Precision 5820 — Xeon W-2155, 64GB RAM, 1TB NVMe, 950W PSU — at $1,121, paired with a used RTX 3090 at a $1,369 median. That's $2,490–$2,520 all-in, and it's not a theoretical config: measured output on Qwen3.8-27B Q4_K_M GGUF at 65,536-token context lands at 37.39 tok/s, rising to 60.33 tok/s with Q4_0 MTP speculative decoding. That's a ~61% speedup from quantization and decoding choices alone, with no extra silicon involved. Numbers like these are why I consider the used and refurb market genuinely safe at this tier — provided the card carries its full 24GB of VRAM.
The DIY equivalent runs $2,000–$2,400 for a single-GPU home-lab: a used RTX 4090 at about $1,200 (eBay range $1,100–$1,600) plus $800–$1,200 in system parts. The dual-GPU variant lands at $3,800–$4,200, and the one component I wouldn't cut there is the 1600W PSU — two 4090s under sustained inference load leave zero margin for a mediocre power supply. Apple's refurb shelf competes here too: an M2 Max Mac Studio with 64GB at roughly $2,400 runs a 70B Q4_K_M, which no $2,400 PC can match on raw capacity.
At this price the trade is no longer GPU versus GPU — it's capacity versus bandwidth, exactly the tension you'd expect from a memory-bound workload. The Framework Desktop with Ryzen AI Max+ 395 costs $3,978 complete and ships with 128GB LPDDR5X-8000, up to 96GB of it assignable as dedicated VRAM (Linux can be configured higher). The catch is bandwidth: 256GB/s shared, against roughly a terabyte per second on a discrete 4090. The vendor-reported 38 tok/s on gpt-oss-120b MXFP4 tells me where the value sits — you're not buying speed, you're buying the ability to hold a 120B-class model at all, inside a 120W sustained / 140W boost envelope.
The NVIDIA DGX Spark pushes the same idea further at $4,699, raised from $3,999 by the memory-shortage surcharge. Inside: 128GB coherent LPDDR5x at 273GB/s, the GB10 Grace Blackwell package, 1 PFLOP FP4 compute, support for models up to 200B parameters at FP4, a 20-core Arm CPU, 4TB NVMe, and a 240W PSU feeding a 140W TDP. OEM unified-memory systems follow the same logic — the CORSAIR AI WORKSTATION 300 (Ryzen AI MAX+ 395, Radeon 8060S iGPU, up to 96GB GPU-accessible out of 128GB LPDDR5X-8000, plus an XDNA 2 NPU at 50 TOPS) runs GPT-OSS 120B at ~63GB MXFP4, a footprint no single consumer card can hold.
At the top sits the RTX PRO 6000 Blackwell Workstation Edition: MSRP ~$8,565, retail $8,000–$9,200, anchoring a ~$22,000 single-GPU professional workstation or a $30,000–$33,000 dual-GPU build. What justifies the spend isn't compute — it's the 96GB frame buffer. A single card fits a 70B Q4 model with 50GB+ of headroom, and that headroom is precisely what team serving eats: each concurrent user at 8K context adds 2–4GB of KV cache. Five colleagues hitting the same model and a consumer card's margin is gone. This is the tier where capacity planning stops being a personal decision.
Every band on this ladder buys a specific, defensible capability. The mistake to avoid isn't overspending within a tier — it's buying a tier whose ceiling you'll hit before the year is out.
The first rule of multi-GPU planning is the one most build lists get wrong: VRAM does not stack. Two 32GB cards do not give you a 64GB pool that one model can fill. They give you two separate 32GB pools that software has to bridge through model parallelism — a specialized path that adds real complexity and synchronization overhead. This is also why splitting a model that already fits on a single GPU can actually make it slower: the coordination cost eats more than the parallelism gives back. When I evaluate a multi-GPU parts list, this is the first assumption I test against the actual workload.
Multi-GPU becomes genuinely necessary in only three situations:
Outside those boundaries, a single 24–48GB GPU handles inference, LoRA/QLoRA adapter fine-tuning, and prototyping just fine.
This is where the physics gets unforgiving. NVLink provides 900GB/s bidirectional on the H100; PCIe 5.0 provides only 64GB/s. Tensor parallelism splits model layers across GPUs and forces a synchronization point on every forward pass, so that gap is not a footnote — it's the difference between sharding that works and sharding that stalls. A 70B FP16 model (~140GB) shards to roughly 35GB per device across 4x A100-80GB, leaving headroom for KV cache and activations, but only because NVLink keeps those per-layer syncs cheap.
Here's the catch for consumer builders: NVLink was last available on the consumer RTX 3090. Everything since talks over PCIe. So a dual-RTX 5090 rig — 64GB total, $9,000–$12,000 at street prices, or $7,602 for the ArsenalPC MES2X prebuilt base — runs its cards over PCIe x8/x8. Looking at the measured results, a 70B Q4 model that fits in those 64GB runs at a pace comparable to or slower than a single Mac Studio M4 Max, because per-layer PCIe synchronization eats the raw bandwidth advantage of two cards.
The flip side: PCIe-connected multi-GPU is cheaper and perfectly adequate for inference parallelism, where each GPU serves separate requests independently and never has to sync mid-layer.
Run this comparison before committing anywhere. A dual RTX PRO 6000 Blackwell setup delivers ~3.5TB/s aggregate bandwidth and 20–30 tok/s decode on large models — but costs $17,000+ for the GPUs alone and draws 600–800W system-wide. A unified-memory Mac Studio M3 Ultra delivers 12–14 tok/s running silently at 250–300W. For batch-1 inference — one or two users, which describes almost every home setup — the multi-GPU throughput advantage rarely justifies 2–3x the total system cost. Multi-GPU is only the clear choice for teams serving multiple concurrent users.
NVIDIA's own hardware proves the point. The dual DGX Spark bundle ($9,449) links two 128GB units via ConnectX-7 at 200Gb/s using RoCE — not NVLink — and therefore requires TP2 sharding recipes. The reported experience: ~40 tok/s single-stream, with time-to-first-token stretching from 4 seconds to ~6 minutes at long context as the sharding overhead compounds. If a purpose-built cluster pays that tax, a DIY PCIe rig pays it with interest.
My verdict after working through these numbers: multi-GPU is expensive overkill for solo inference, and the defensible spec for most people is the largest single-GPU VRAM pool the budget allows — with the second card reserved for the day your workload crosses one of those three genuine boundaries.
This is where cheap builds actually sink — not on the GPU price itself, but on everything the GPU drags behind it. Watts, heat, and resale value are where the real money moves, so I treat power and thermals as first-class spec decisions rather than afterthoughts. A parts list that looks cheap on paper can easily cost more to own than a pricier build done right.
The RTX 5090's 575W TDP is only the headline figure — transient spikes push past it, and a PSU running at its ceiling gets loud, runs hot, and trips over-current protection at the worst possible moment. The sizing rules I'd defend to any reviewer:
A gaming rig bursts to full power for minutes at a time; an AI workstation sustains 1–3kW draws for hours, and sustained load is what the meter actually sees. Over three years, energy adds $5,000–$20,000+ to total cost of ownership — and across five years of high-utilization use, electricity can become the largest single cost category, ahead of the GPUs themselves. Any honest comparison between a local box and cloud pricing has to put that number on the table.
At this power level, cooling is functional, not cosmetic. Ultra-high-end multi-GPU configurations generally need liquid cooling or enterprise-grade air cooling with dedicated airflow paths, because cards that thermally throttle during extended training runs silently delete exactly the throughput you paid for. The chassis question answers itself once you count GPUs:
Physical space, power delivery, and cooling infrastructure have to be planned together — a rackmount stuffed into an unventilated closet is just an expensive heater. And while we're auditing power claims: appliance-class systems like the dual DGX Spark advertise 2x 240W of PSU capacity, which is not the same as 480W of measured wall draw. Trust the wall numbers, not the spec-sheet arithmetic.
The cheapest hedge against regret is planning the second GPU before you've bought the first:
The used market cuts both ways, and that's exactly why it's the budget's best friend. A three-year-old professional GPU remains highly effective for fine-tuning, inference, and development — meaning today's purchase has a real buyer three years out. Resale typically recoups 30–50% of the original price: a $15,000 workstation sold for $6,000 after three years nets to a $9,000 investment for three years of full-time compute. Unlike cloud OpEx, hardware depreciation also carries tax benefits that renting compute can't match. I price the exit before I enter.
The DRAM shortage has produced one exploit worth using: prebuilt systems are often cheaper than the standalone GPU. Right now a $3,599 MSI Aegis 5090 prebuilt undercuts the $3,695 Newegg Founders Edition card sold by itself — buying the whole machine to get the GPU is simply the cheaper route to identical silicon.
The last constraint appears on no spec sheet: if your room cannot dissipate the heat of a 600W GPU, no parts list saves you. Every watt that runs inference has to exit through a wall, a window, or your patience — and the room is the one component you can't RMA.
Inference and training pull a parts list in two different directions, and the split comes down to what has to live in memory during the run. Inference holds the weights plus the KV cache and nothing else, which keeps the math friendly: a 7B model fits comfortably in 16GB, and even 70B runs on a single 80GB GPU once quantized. Training must also hold optimizer states and gradient buffers, and the requirements climb fast — LoRA fine-tuning of 70B needs at least 40GB, and full-parameter training at that scale demands 80–160GB across multiple GPUs. When I line these numbers up side by side, the budget conclusion writes itself: inference workloads can run on smaller, cheaper GPUs than training ever could, and the most expensive mistake at this stage is buying a training-class machine for an inference-only job.
The RTX 5090 gives us the cleanest measurements available for how far one consumer card stretches:
What I take from this is that quantization at the fine-tuning tier isn't a compromise — it's the difference between fine-tuning on the card you already own and not fine-tuning at all.
The tiers aren't about prestige — they map directly onto the memory math above. The entry tier covers 7B-class inference and small QLoRA runs; the professional tier rides the 5090's 40B QLoRA ceiling; the enterprise tier exists because 70B-scale training simply cannot happen on one consumer card.
For solo developers, one box breaks the tier logic: the DGX Spark at ~$4,000–$4,699. Its memory-to-price ratio is unmatched right now — 128GB of unified memory, 1 PFLOP of compute, 200B-parameter models at FP4, and zero multi-GPU complexity to configure or debug. When your workload outgrows a single 32GB card but doesn't justify a multi-GPU build, this is where the value sits.
Exact model and backend support beats brand allegiance, because GGUF, MLX, and NVFP4 files are not interchangeable — a model that decodes quickly on one platform may not even run on another. Confirm your target models exist in the format your stack consumes before you spend anything. The smart pairing for shared-memory systems is MoE models: all weights must fit, but only a small fraction participates per token — gpt-oss-120b activates just 5.1B of its 117B total parameters, which is exactly the profile that lets a 128GB unified box punch above its price.
Above roughly $12,000, resist buying by default. When frontier weights are unreleased and three DGX Sparks cost $14,148 before networking, renting or waiting is the defensible call — a purchase you can't tie to a real workload is a benchmark chase, not a tool.
That leaves the final discipline, the one that makes the whole spec hold up: buy for the private, repetitive, context-heavy work you actually run daily, document why each part earned its slot, and treat every upgrade decision as a workload question rather than a benchmark question. Do that, and you can defend every line of your parts list to the most skeptical reviewer you'll ever face: yourself.