How to Benchmark GPU Cloud Providers for Real Cost

A GPU listed at $1.40 an hour can cost you more than one listed at $2.10. That is not a typo, it is what happens when you buy hours instead of buying work. If you are comparing GPU clouds by sorting a pricing table, you are optimizing for the wrong number, and this guide walks through building a benchmark that gets you the number that actually matters: cost per unit of work.

Step 1: Define your unit of work before you touch a provider

You cannot benchmark what you have not defined. Pick the job that actually represents your spend, not a generic test. If you are training, that means a fixed number of epochs on a fixed dataset slice, measured to a specific loss target or a fixed wall-clock budget. If you are serving inference, it means tokens generated per second at a specific batch size and sequence length, under a load pattern that resembles production.

Write this down as a spec before you provision anything: dataset size, model architecture, precision (fp16, bf16, fp8), batch size, and the exact success criterion. Without this, you will end up comparing apples measured on Tuesday to oranges measured on Thursday, and every provider will look different for reasons that have nothing to do with cost.

Step 2: Run the same job on each provider and record raw throughput

Provision the comparable instance type on each provider, same GPU generation, same GPU count, and run your defined job unchanged. Record wall-clock time to completion, tokens or samples processed per second, and total instance-hours billed.

Do not trust the spec sheet here. Two listings that both say "H100" can perform differently because of interconnect (NVLink versus PCIe), memory bandwidth utilization, host CPU bottlenecks, or how well the provider's stack is tuned for your framework. An older CUDA driver or a noisier multi-tenant host can quietly cost you meaningful throughput on the exact same silicon. Run it yourself, three times if the numbers seem to wobble, and take the median.

At this point you can compute a first-pass number: raw compute cost per unit of work, which is (hourly rate x hours to complete job) / units of work completed. This is already more honest than $/hour alone, but it is still incomplete.

Step 3: Layer in realistic utilization, not theoretical utilization

Here is where most GPU budgets actually leak. Instance-hour billing charges you the full hour whether the GPU sits at 92% utilization or 12%. If your workload is bursty, batch jobs that run for 20 minutes then go idle for two hours, your effective cost per unit of work is a multiple of what the raw benchmark showed. This pairs well with see also: fix look-ahead bias in insider trade backtests.

To layer this in, estimate the utilization rate you will realistically hit on each provider given its autoscaling speed and your job's burst pattern. A provider with slow node scale-up leaves you paying for idle capacity between bursts, while a provider with fast autoscaling lets you scale down aggressively and pay only for active work.

Adjust your cost-per-unit number using this formula:

Effective cost per unit of work = (raw cost per unit of work) / (expected utilization rate)

A workload that only hits 40% utilization on a cheap provider can end up costing more per unit of work than the same job hitting 85% utilization on a pricier one. This single adjustment is usually enough to flip a comparison table upside down.

Sub-step: measure utilization with something you already have

You do not need custom tooling for this. nvidia-smi dmon or a lightweight Prometheus exporter running alongside your job gives you GPU utilization samples over the run. Log the percentage at regular intervals and average it across the full billed window, not just the active compute window, since the gaps between bursts count against you too. We cover related ground in go deeper on redora vs nestjs-redis: which redis tool fits nestjs?.

Step 4: Price the full data path, not just the accelerator

GPUs consume data constantly: training data streams in, checkpoints write out, model weights load at startup. If your data lives in a different region or cloud than your GPU capacity, egress fees and cross-region transfer charges can rival the compute bill itself.

For each provider, add up the egress cost per terabyte moved during your benchmark job, the storage cost for the data your job touches, and any premium the provider charges for co-locating storage with compute. A provider offering a slightly higher hourly rate but free egress within its own storage tier can beat a cheaper provider that nickel-and-dimes you on every gigabyte moved in and out.

Step 5: Account for provisioning time and cold starts

Time the full cycle from requesting a node to your job actually executing its first useful operation. This includes instance boot, driver initialization, and pulling any large model weights into memory. If you churn nodes frequently, spinning up for short jobs and tearing down, this overhead repeats every cycle and eats into both wall-clock time and billed instance-hours.

A provider that takes eight minutes to deliver a usable GPU node effectively taxes every job you run there. Compare this against a provider with near-instant capacity, even if that provider charges a premium rate. Fold this into your cost-per-unit number by adding the provisioning time to the denominator of your throughput calculation, treating it as billed but unproductive time.

Step 6: Factor in eviction risk as a cost, not a footnote

The lowest advertised rates typically come from spot or preemptible capacity pools, and those pools disappear when demand spikes. If eviction hits mid-run and you restart from the last checkpoint, or worse from scratch, the cheap rate just bought you expensive rework.

Also read: how to backtest a polymarket trading bot for slippage — background

Estimate an eviction cost by asking what percentage of runs on that provider's discounted tier you expect to lose, then multiply that by the cost of a full re-run. Add this expected rework cost to your total. A provider with a stable on-demand tier might look 20% more expensive on paper but end up cheaper once you price in the re-runs a flaky spot pool forces on you.

Step 7: Assemble the worksheet and compute your real number

Pull every prior step into one formula per provider:

True cost per unit of work = [(hourly rate x hours to complete job) / expected utilization rate] + data egress and storage cost + (provisioning overhead x billed rate) + (eviction probability x re-run cost), all divided by units of work completed.

Build this as a simple spreadsheet with one row per provider and one column per input: hourly rate, benchmarked throughput, expected utilization, egress cost, provisioning time, and eviction probability. Fill in real numbers from your benchmark runs, not estimates from a sales page, and let the spreadsheet do the sorting.

What to expect next

Once you run this worksheet on two or three providers, expect the ranking to change from what the pricing page suggested, sometimes dramatically. Treat this as a recurring check rather than a one-time exercise: provider pricing, capacity, and performance all shift over time. Pair the benchmark with basic FinOps habits, killing idle non-production GPU pools and scheduling scale-downs during off-hours, and provider choice becomes a smaller lever than simply not wasting what you already rent.