The right graphics card for running large language models locally comes down to one number more than any other: how much VRAM it carries. That single spec decides which models you can load, how much context you get to work with, and whether you run smoothly or fight out-of-memory errors all afternoon. For the exact VRAM each model size needs before you pick a card, see how much VRAM to run an LLM. Raw speed, price, and power draw all matter, but they come second to fitting the model in memory in the first place.
This guide ranks the best GPUs for LLM work in 2026, for local inference and fine-tuning, from a used card around a thousand dollars to a 96GB workstation card that swallows a 70B model whole. For each card you get the specs that actually move the needle for LLMs, the model sizes it can run, the going street price, and a plain verdict on who should buy it. The speed numbers are tested, not copied off a spec sheet. We re-verified every price, listing, ASIN, and availability claim in August 2026. The speed figures were captured in June 2026 on real rented hardware (an RTX 3090, RTX 4090, RTX 5090, and an L40S) running Ollama.
ComputingForGeeks is reader-supported. When you buy through links on this page we may earn an affiliate commission at no extra cost to you. As an Amazon Associate we earn from qualifying purchases.
VRAM is the whole game: how much do you actually need?
Before any brand or benchmark, work out the memory budget. A model’s weights have to fit in VRAM, and the size depends on the parameter count and the quantization level. Quantization shrinks the weights by storing them at lower precision: Q4_K_M (roughly 4.5 bits per weight) is the level most people run, Q8_0 doubles the size for a small quality bump, and FP16 is the full-precision original.
Here is the memory the weights alone need, by model size and quantization:
| Model size | Q4_K_M | Q8_0 | FP16 | Comfortable on |
|---|---|---|---|---|
| 7B / 8B | ~5 GB | ~8 GB | ~16 GB | Any 8GB+ card |
| 13B / 14B | ~8 GB | ~13 GB | ~26 GB | 12GB card |
| 32B / 34B | ~20 GB | ~32 GB | ~64 GB | 24GB card |
| 70B | ~40 GB | ~70 GB | ~140 GB | 48GB card (or 2x 24GB) |
| 405B | ~230 GB | ~405 GB | ~810 GB | Multiple GPUs |
One catch the size tables usually skip: those figures are weights only. Real usage adds the KV cache (the model’s running memory of the conversation) plus activation buffers, which together add roughly 15 to 20 percent at short context. The KV cache grows linearly with context length, so a long 128K-token prompt on a 70B model can add tens of gigabytes on top of the weights. Size for the weights, then leave headroom. If you plan to run long contexts, leave a lot.
The practical takeaway: 24GB runs anything up to a 32B model and a quantized 70B in a pinch with offload. 48GB runs a 70B comfortably. Going past 70B on a single card means 96GB, and 405B-class models mean a stack of cards. Keep that ladder in mind as you read the picks.
Our top picks at a glance
If you want the short version, here is where each tier wins. The detailed breakdowns follow.
- Best overall (consumer): NVIDIA RTX 5090 (32GB). The fastest single card most people can buy, with enough VRAM for 32B models and long context.
- Best value: a used RTX 3090 (24GB). The cheapest way into 24GB of VRAM, and still plenty fast for everyday local inference.
- Best budget / lowest power: RTX 4060 Ti 16GB. 16GB at 165W gets you into 13B models without a new power supply.
- Best for 70B on one card: RTX 6000 Ada (48GB). 48GB of ECC memory runs a 70B model without quantizing into the floor.
- No-compromise workstation: RTX PRO 6000 Blackwell (96GB). The only tower-friendly GPU that holds a 70B at high precision with room for huge context.
- Best quiet, low-power big-memory box: Mac Studio with M3 Ultra. 96GB of unified memory runs a 70B in near silence, drawing a fraction of a workstation card’s power.
1. NVIDIA RTX 3090 (24GB): the value pick
The standout budget play in 2026 is still a card from 2020. The RTX 3090 packs 24GB of GDDR6X, and on the used market it is the cheapest path to that much VRAM by a wide margin. For local LLM work, where memory capacity matters more than the last few percent of speed, that makes it the value king.
| Spec | RTX 3090 |
|---|---|
| VRAM | 24GB GDDR6X |
| Memory bandwidth | ~936 GB/s |
| CUDA cores | 10,496 (Ampere) |
| Power (TGP) | 350W (750W PSU) |
| Street price (used) | ~$800 to $1,300 |
It runs 32B models at Q4 at a usable speed (37 tokens per second in our test), though a wide context will spill past 24GB onto the CPU, and two of them on one board (check your PCIe lane budget first) give you the 48GB needed for a 70B. Where it falls short: it is a power-hungry, end-of-life card with no warranty on the used market, and its prefill speed trails the Ada and Blackwell generations. But if you want maximum VRAM per dollar to learn on, nothing else is close. Check the current 3090 listings on Amazon, and expect asks to keep drifting up: the memory crunch spilled into the used market during 2026, so a card that went for $700 last year now typically clears $1,000.
2. NVIDIA RTX 4060 Ti 16GB: budget, low power
If a used card makes you nervous and you want something new, cheap, and easy on the wall socket, the 16GB version of the RTX 4060 Ti is the entry ticket. At 165W it runs off a modest 550W power supply, so it drops into almost any existing desktop without an upgrade.
| Spec | RTX 4060 Ti 16GB |
|---|---|
| VRAM | 16GB GDDR6 |
| Memory bandwidth | ~288 GB/s |
| CUDA cores | 4,352 (Ada Lovelace) |
| Power (TGP) | 165W (550W PSU) |
| Street price (new) | ~$300 to $500, stock thinning |
The 16GB buffer holds a 13B model at Q4 with full context, or a tight 20B. Where it falls short is bandwidth: at 288 GB/s it is the slowest card here per token, so generation feels sluggish next to a 3090 even though both fit the same model. For a first local-LLM box, a quiet home server, or a low-power always-on assistant, it earns its place. Grab the 16GB model on Amazon and double-check it says 16GB, since an 8GB version shares the name and is useless for this. NVIDIA ended production back in early 2025, so what remains is draining channel stock. Its Blackwell successor is the RTX 5060 Ti 16GB, which pushed 75 tokens per second on an 8B model in our RTX 5070 Ti vs 5060 Ti testing at the same low-power, small-PSU profile.
3. NVIDIA RTX 4090 (24GB): the fast 24GB workhorse
The RTX 4090 holds the same 24GB as a 3090 but moves tokens roughly three times faster thanks to the Ada architecture and 1 TB/s of bandwidth. For most people it is the obvious local-LLM card. The complication in 2026 is price: NVIDIA stopped making it, AI demand never let up, and street prices sit well above the original $1,599.
| Spec | RTX 4090 |
|---|---|
| VRAM | 24GB GDDR6X |
| Memory bandwidth | ~1,008 GB/s |
| CUDA cores | 16,384 (Ada Lovelace) |
| Power (TGP) | 450W (850W PSU) |
| Street price | ~$2,700 to $3,800 new, ~$2,200 to $2,500 used |
It runs the same 32B-class models a 3090 does, but the speed gap is real and you feel it on every generation. Where it falls short is value: with the 5090 available and used 3090s a fraction of the price, a new 4090 occupies an awkward middle. It makes most sense bought used by someone who wants Ada speed without 5090 money. Compare RTX 4090 prices on Amazon against the 5090 before committing.
4. NVIDIA RTX 5090 (32GB): the best single consumer card
If you only pick one card off this list and budget is not the deciding factor, the RTX 5090 is it. Blackwell brings 32GB of GDDR7 and nearly 1.8 TB/s of bandwidth, and in our testing it generated tokens faster than anything else short of a data-center part. The extra 8GB over the 24GB crowd also buys real breathing room for context.
| Spec | RTX 5090 |
|---|---|
| VRAM | 32GB GDDR7 |
| Memory bandwidth | ~1,792 GB/s |
| CUDA cores | 21,760 (Blackwell) |
| Power (TGP) | 575W (1000W PSU) |
| Street price | ~$4,300 to $5,000+ (MSRP was $1,999) |
It runs 32B models with comfortable context headroom and chews through 8B models. Where it falls short: a 70B at Q4 still needs about 40GB, so even 32GB cannot fit one on a single card, and the GDDR7 shortage has pushed street pricing to more than double MSRP, with only the occasional $1,999 Founders Edition drop selling out in minutes. The 575W draw also demands a serious power supply and cooling. For a single-card local rig that does everything short of 70B, it is the one to beat. Check RTX 5090 availability on Amazon, since Founders Edition stock comes and goes.
5. NVIDIA RTX 6000 Ada (48GB): 70B on one card
This is where the conversation shifts from gaming cards to workstation silicon. The RTX 6000 Ada Generation puts 48GB of ECC memory on a single 300W board, which is the first point on this list where a 70B model fits on one card without dropping to a punishing quantization. It is the same Ada architecture as the 4090, tuned for reliability and capacity instead of clock speed.
| Spec | RTX 6000 Ada |
|---|---|
| VRAM | 48GB GDDR6 with ECC |
| Memory bandwidth | 960 GB/s |
| CUDA cores | 18,176 (Ada Lovelace) |
| Power (TGP) | 300W |
| Street price | ~$8,400 to $10,000+ |
The 48GB runs a 70B at Q4 with room for the KV cache, and its low 300W draw and dual-slot form make it friendly to multi-GPU workstations. Where it falls short is age: this is 4090-generation silicon at a workstation premium. The value math moved back in its favor during 2026 though, because NVIDIA kept raising the 96GB Blackwell card, so a 6000 Ada found near $8,400 is again the cheaper sane route to a single-card 70B when 96GB is overkill. See RTX 6000 Ada listings on Amazon.
6. NVIDIA RTX PRO 6000 Blackwell (96GB): the no-compromise workstation card
For the buyer who wants the most VRAM that fits in a desktop, the RTX PRO 6000 Blackwell is the answer. It carries 96GB of GDDR7 with ECC, which makes it the only GPU that fits in a tower and holds a 70B model at high precision with enormous context headroom, or a 120B-class model at Q4. It ships in a 600W Workstation Edition and a 300W Max-Q variant with the same memory for dense multi-GPU builds.
| Spec | RTX PRO 6000 Blackwell |
|---|---|
| VRAM | 96GB GDDR7 with ECC |
| Memory bandwidth | ~1,792 GB/s |
| CUDA cores | 24,064 (Blackwell) |
| Power (TGP) | 600W (Max-Q: 300W) |
| Street price | ~$12,100 to $13,250 (list price now $13,250) |
It runs a 70B at Q8 with context to spare, or much larger models at Q4, all on one card with no multi-GPU plumbing. Where it falls short is price, and the direction it keeps moving: NVIDIA has raised the list price repeatedly since launch, to $13,250 at this update, a 55 percent climb driven by the same GDDR7 squeeze hitting the 5090. For a serious single-user AI workstation, nothing else in a tower comes close, and we benchmarked it head to head against the RTX 5090 to quantify exactly what the extra money buys. Check the RTX PRO 6000 Blackwell on Amazon; these are often sold through system integrators, so stock varies.
7. Apple Mac Studio (M3 Ultra): the unified-memory alternative
The Mac Studio with M3 Ultra takes a different route to big-model capacity. It uses unified memory shared between CPU and GPU, 96GB of it at 819 GB/s on a current unit, in a complete computer that draws under 500W and runs nearly silent. That 96GB holds a 70B model with room for context, and a roughly 100B-class model at Q4, in a box you can leave on a desk without hearing it.
| Spec | Mac Studio (M3 Ultra) |
|---|---|
| Unified memory | 96GB (the only configuration Apple still sells) |
| Memory bandwidth | 819 GB/s |
| GPU | 60 or 80-core |
| Max power (whole machine) | 480W |
| Street price | from $5,299 (96GB, 1TB) |
Where it falls short: prompt processing on Apple silicon is slower than a comparable NVIDIA card, the CUDA software ecosystem is not there (you run llama.cpp and MLX, not vLLM), and the big-memory story shrank during 2026. As DRAM prices climbed, Apple dropped the 512GB option in March 2026 and the 256GB option soon after, so a new M3 Ultra now caps at 96GB, Apple raised its price by $1,300 in June, and ship dates have been slipping by weeks. The single Mac that held a 405B model is gone. What remains is a quiet, low-power, all-in-one path to 96GB that handles a 70B comfortably, often for less than a 96GB workstation card alone. Configure one through Mac Studio listings on Amazon and watch the memory tier, since that is the spec that matters for LLMs.
8. AMD Radeon RX 7900 XTX (24GB): the ROCm option
The honest AMD pick is the Radeon RX 7900 XTX. It offers 24GB of GDDR6 at a price that, now that it is end-of-generation, often undercuts a used 3090. The catch has always been software, and the news there is better than it used to be: its gfx1100 architecture is supported in current ROCm, and llama.cpp runs on it through both the ROCm/HIP backend and the vendor-neutral Vulkan backend.
| Spec | RX 7900 XTX |
|---|---|
| VRAM | 24GB GDDR6 |
| Memory bandwidth | ~960 GB/s |
| Compute units | 96 (RDNA 3) |
| Power (TBP) | 355W (800W PSU) |
| Street price | ~$770 to $930 (launched at $999) |
It runs 32B models at Q4 much like a 3090 does. Where it falls short: outside llama.cpp and Ollama, a lot of the AI tooling assumes CUDA, so you will hit more friction with training frameworks, fine-tuning scripts, and newer inference servers than you would on any NVIDIA card. If you run Ollama or llama.cpp and want 24GB cheaply, it is a real option. If you expect to follow every CUDA-first tutorial unchanged, buy NVIDIA. AMD’s own step up is the Radeon AI PRO R9700, a 32GB RDNA 4 workstation card listed at $1,299 that runs the same ROCm stack, though retail supply has been patchy since it launched, so treat it as one to check rather than a default. Browse RX 7900 XTX cards on Amazon.
Tested: LLM inference speed in tokens per second
Spec sheets tell you what fits; they do not tell you how fast it runs. We rented an RTX 3090, RTX 4090, RTX 5090, and L40S (48GB) by the hour and measured generation speed with Ollama on three model sizes, all at Q4_K_M with a fixed 4096-token context. The numbers below are the measured generation rate in tokens per second, warm with the model already loaded.
Here is one run on the RTX 5090, the fastest card we tested. The figure that matters is the eval rate at the bottom:
Collecting that eval rate for every card and model size gives the full picture:
| GPU (VRAM) | Llama 3.1 8B | Qwen2.5 32B | Llama 3.3 70B |
|---|---|---|---|
| RTX 3090 (24GB) | 119 tok/s | 37 tok/s | does not fit |
| RTX 4090 (24GB) | 148 tok/s | 43 tok/s | does not fit |
| RTX 5090 (32GB) | 250 tok/s | 71 tok/s | does not fit |
| L40S (48GB) | 113 tok/s | 33 tok/s | 16 tok/s |
Plotted side by side, the spread is easy to read:
Three things stand out. On the 8B model every card clears 100 tokens per second, well past reading speed, so the difference barely matters for interactive chat. On the 32B model the order changes: the RTX 5090 hit 71 tokens per second, more than double the L40S, even though the L40S has more VRAM. Token generation is bound by memory bandwidth, and the 5090’s GDDR7 is far faster than the L40S’s GDDR6, so more memory does not mean more speed. And the 70B ran only on the 48GB card; the 24GB and 32GB cards cannot hold it at all.
One trap is worth calling out, because it bites people daily. Those 32B numbers used a modest 4096-token context. When we let Ollama fall back to a large context window, the 32B’s KV cache pushed the footprint past 24GB, the 3090 spilled layers onto the CPU, and generation collapsed from 37 tokens per second to under 4. On a 24GB card, keep the context window in check or the model quietly falls off a cliff. A 32GB or larger card sidesteps the problem entirely. You can reproduce any of this with a pull and a verbose run:
ollama pull llama3.1:8b
ollama run llama3.1:8b --verboseThe --verbose flag prints the eval rate (generation tokens per second) after each response, which is the figure in the table. Your own numbers will shift with the backend, the quantization, and the context length, but the ranking between cards holds. The backend gap is real: the same class of card generated meaningfully faster when we ran llama.cpp directly than through Ollama, and vLLM and TensorRT-LLM shift the picture again. If you would rather not buy hardware at all, the same models run on CPU too, and our guide on running local LLMs with Ollama on CPU covers that path.
Data-center GPUs: what the cloud actually runs
You cannot sensibly buy these for a desk, but it helps to know what sits above the workstation tier, because it is what you rent when you go to the cloud. These are the parts behind hosted inference APIs and serious training runs.
| GPU | VRAM | Architecture | Bandwidth | Cloud rental (rough) |
|---|---|---|---|---|
| H100 (SXM) | 80GB HBM3 | Hopper | 3.35 TB/s | ~$2 to $3 / hr |
| H200 | 141GB HBM3e | Hopper | 4.8 TB/s | ~$3 to $4 / hr |
| B200 | 192GB HBM3e | Blackwell | 8 TB/s | ~$5 to $6 / hr |
Rental rates move constantly and vary by provider, region, and commitment, so treat these as ballpark. The point is the capability gap: a single B200 holds nearly as much as eight 24GB consumer cards, with memory bandwidth no desktop part approaches. For serving many users at once, that is the tier you scale into, often with vLLM on a Kubernetes cluster.
Buy or rent? Run the break-even math
A large share of “best GPU for LLM” buyers conclude they should rent instead, and for plenty of workloads that is the right call. The math is simple: divide the purchase price by the hourly rental rate of an equivalent cloud card to get the break-even running time.
A new RTX 4090 around $2,700 against a cloud 4090 at roughly $0.40 an hour breaks even near 6,750 hours, which is about nine months of running it 24 hours a day. If your GPU would otherwise sit idle most of the week, renting wins comfortably. If you run inference or fine-tuning jobs continuously, owning pays for itself and then keeps paying. The crossover ignores electricity, the host machine, and resale value, but it frames the decision: steady heavy use favors buying, bursty or occasional use favors renting.
For occasional jobs, renting a cloud GPU by the hour (for example on Vultr’s cloud GPU instances) skips the upfront cost entirely, and you can automate on-demand GPU jobs so you only pay while a job runs. Renting also lets you try an H100 or B200 for an afternoon to see whether the speed is worth it before spending workstation money, and our notes on renting high-end GPUs from cloud providers cover the gotchas.
Best GPU for fine-tuning an LLM: does training change the answer?
Mostly no, with one twist. For LoRA and QLoRA fine-tuning, the VRAM ladder above still decides everything, just one rung higher than inference for the same model. QLoRA loads the base weights at 4-bit and trains small adapter layers on top, so a 24GB card fine-tunes models up to about 14B comfortably, and our QLoRA fine-tuning walkthrough with Unsloth runs end to end on a single consumer GPU. The twist: training cares about capacity more than bandwidth. A batch either fits or it does not, so for a fine-tuning box the 5090’s 32GB beats a faster hypothetical 24GB card, and a cheap 3090 remains a perfectly good adapter-training workhorse.
Full fine-tuning, where every weight updates, is a different budget. Weights, gradients, and optimizer states together want several times the model’s FP16 footprint, which puts even a 7B full fine-tune into 80GB territory. That is rental hardware, not desk hardware: an H100 by the hour costs less than a workstation card’s monthly depreciation for the handful of runs most projects need. If you want the adapter route without touching code first, Unsloth Studio wraps the same QLoRA flow in a UI.
Frequently asked questions
Can I run a 70B model on a 24GB GPU?
Not entirely in VRAM. A 70B at Q4 needs about 40GB, so a 24GB card has to offload the rest to system RAM, which slows generation to a crawl. To run a 70B properly you want a 48GB card like the RTX 6000 Ada, two 24GB cards together, or Apple unified memory. On a single 24GB card, stick to 32B and smaller.
Is the RTX 5090 worth twice the price of a used 3090?
For speed and the extra 8GB of VRAM, yes, if you can afford it and you generate a lot of tokens. The 5090 is dramatically faster and gives more context room. But a used 3090 fits the same 32B models and costs a fraction as much, so for learning, light use, or a tight budget the 3090 is the smarter buy. Most people are better served by the 3090 or a 4090 than by stretching for a 5090. We break this matchup down in our RTX 3090 vs RTX 5090 comparison.
Do I need an NVIDIA GPU, or does AMD work?
AMD works for inference. The RX 7900 XTX runs Ollama and llama.cpp through ROCm and Vulkan, and 24GB for the price is attractive. NVIDIA remains the safe default because almost all AI tooling targets CUDA first, so you hit far less friction with training, fine-tuning, and newer inference servers. If you only run Ollama or llama.cpp, AMD is fine. For anything else, choose NVIDIA.
How much system RAM do I need, separate from VRAM?
For pure GPU inference, a comfortable rule is system RAM at least equal to your VRAM, and double it if you ever offload layers to the CPU or load models from disk. A 24GB GPU pairs well with 32 to 64GB of RAM. If you run large models partly on the CPU, or run a Mac where memory is unified, RAM capacity becomes the model-size limit directly.
Does quantization hurt model quality?
A little, and usually not enough to notice. Q4_K_M is the standard local quantization because it cuts memory roughly in half versus FP16 while keeping output quality very close to the original. Q8 is nearly indistinguishable from full precision. Drop below Q4 and quality degradation becomes visible, so Q4_K_M is the floor most people should run.
Which is the best GPU for your LLM build?
Work down this short decision path and you will land on the right card:
- On a tight budget, or just learning? A used RTX 3090. Maximum VRAM per dollar, runs everything up to 32B.
- Want something new, cheap, and low-power? The RTX 4060 Ti 16GB. Good for 13B models and an always-on box.
- Want the fastest single card and run mostly 32B and smaller? The RTX 5090. Buy a 4090 used instead if you want to spend less for similar memory.
- Need to run a 70B at home? The RTX 6000 Ada (48GB), or two 24GB cards if you would rather build than buy a workstation part.
- Want the most memory a desktop can hold, with no plumbing? The RTX PRO 6000 Blackwell (96GB) for raw speed, or a Mac Studio M3 Ultra if a quiet, low-power 96GB box matters more than peak throughput.
- Only need a GPU now and then? Do not buy one. Rent by the hour and put the money toward the jobs that actually need it.
For most people building a local LLM machine in 2026, the honest answer is a used 3090 to start and a 5090 when you have outgrown it. Once you have settled on a card, our local AI workstation build turns it into a complete, parts-verified machine across three tiers. Spend on VRAM first, speed second, and everything else last.
