GPU infrastructure, compared honestly
Every option below is supported in our server platforms. Select a model and workload to see what each GPU delivers for your organisation, then compare several side by side.
| GPU | Class | VRAM | Bandwidth | ECC | Power | Fits Llama 3.1 8B? | Recommended count | Expected tokens/s | Hardware price | |
|---|---|---|---|---|---|---|---|---|---|---|
| GeForce RTX 5090 Blackwell · PCIe, 2–3.5 slot (board-partner dependent) |
Professional / high-end consumer GPU | 32 GB | 1,792 GB/s | No | 575 W | Yes (2 GPUs) | 2 | 11,524 | $9,600 $4,800 each | |
| GeForce RTX 5080 Blackwell · PCIe, 2–3 slot |
Professional / high-end consumer GPU | 16 GB | 960 GB/s | No | 360 W | Yes (4 GPUs) | 4 | 11,661 | $9,200 $2,300 each | |
| RTX PRO 6000 Blackwell Server Edition Blackwell · PCIe, dual-slot passive |
Enterprise workstation & server GPU | 96 GB | 1,792 GB/s | Yes | 600 W | Yes (1 GPU) | 1 | 6,402 | $17,000 $17,000 each | |
| NVIDIA L40S Ada Lovelace · PCIe, dual-slot passive |
Enterprise workstation & server GPU | 48 GB | 864 GB/s | Yes | 350 W | Yes (2 GPUs) | 2 | 5,556 | $30,000 $15,000 each | |
| NVIDIA H100 NVL Hopper · PCIe, dual-slot passive, NVLink bridge |
Data-centre accelerator | 94 GB | 3,938 GB/s | Yes | 400 W | Yes (1 GPU) | 1 | 14,069 | $56,000 $56,000 each | |
| NVIDIA H200 NVL Hopper · PCIe, dual-slot passive, NVLink bridge |
Data-centre accelerator | 141 GB | 4,800 GB/s | Yes | 600 W | Yes (1 GPU) | 1 | 17,149 | $64,000 $64,000 each |
Recommended count is the smallest number of GPUs meeting memory, 20% peak concurrency at 10 tokens/s per user and daily demand of 20 prompts per employee. Illustrative until benchmarked.
Side-by-side comparison
GeForce RTX 5090 Professional / high-end consumer GPU | RTX PRO 6000 Blackwell Server Edition Enterprise workstation & server GPU | NVIDIA H200 NVL Data-centre accelerator | |
|---|---|---|---|
| Architecture | Blackwell | Blackwell | Hopper |
| VRAM | 32 GB GDDR7 | 96 GB GDDR7 with ECC | 141 GB HBM3e with ECC |
| Memory bandwidth | 1,792 GB/s | 1,792 GB/s | 4,800 GB/s |
| FP16 tensor (dense) | 209 TFLOPS | 251 TFLOPS | 835 TFLOPS |
| ECC memory | No | Yes | Yes |
| NVLink | No | No | Yes |
| Power | 575 W | 600 W | 600 W |
| Form factor | PCIe, 2–3.5 slot (board-partner dependent) | PCIe, dual-slot passive | PCIe, dual-slot passive, NVLink bridge |
| Cooling | Active (axial fans); needs 3-slot spacing and strong chassis airflow | Passive; server chassis airflow | Passive; data-centre airflow |
| Support terms | Consumer warranty; not covered by enterprise driver and support programmes | Enterprise drivers, 3-year warranty, vendor support | Enterprise support and 5-year NVIDIA AI Enterprise option |
| Trade-offs | Best inference per dollar; 32 GB without ECC limits model size and reliability for continuous service. | Large ECC memory in a standard PCIe server; the usual choice for 70B-class models without HBM pricing. | Largest memory per PCIe card: fewer GPUs for 70B+ models and long contexts. |
| Supported models (INT4, per GPU) | Llama 3.2 3B, Llama 3.1 8B, Gemma 3 12B, Qwen2.5 14B | Llama 3.2 3B, Llama 3.1 8B, Gemma 3 12B, Qwen2.5 14B, Qwen2.5 32B, Llama 3.3 70B, gpt-oss 120B (MoE, 5B active) | Llama 3.2 3B, Llama 3.1 8B, Gemma 3 12B, Qwen2.5 14B, Qwen2.5 32B, Llama 3.3 70B, gpt-oss 120B (MoE, 5B active) |
| Expected concurrent sessions (8B, 8k ctx, 1 GPU) | 24 | 81 | 122 |
| Hardware price (each) | $4,800 | $17,000 | $64,000 |
| Source | Manufacturer page | Manufacturer page | Manufacturer page |
Professional / high-end consumer
GeForce RTX 5090 and 5080. Excellent inference per dollar for 8B–14B models and small teams. Trade-offs: no ECC memory, consumer warranty, active coolers that need 3-slot spacing and strong airflow, and 16–32 GB that caps model size and concurrency. We cap deployments at four cards.
Enterprise workstation and server GPU
RTX PRO 6000 Blackwell and L40S. ECC memory, passive cooling for dense servers, enterprise drivers and multi-year support. 48–96 GB per card runs 32B–70B-class models in standard PCIe servers at a fraction of HBM pricing.
Data-centre accelerators
H100 NVL and H200 NVL. HBM3/HBM3e bandwidth of 3.9–4.8 TB/s and up to 141 GB per card for the highest throughput on 70B+ models and long contexts, with NVLink pairing. Premium cost, power delivery and facilities requirements.