Why Benchmark Results Vary After Reboot: Thermal Throttling

Rerun the exact same benchmark on the exact same machine, and you can get numbers that differ by almost 3x depending on whether the machine just booted or has been grinding for a day straight. That's not measurement noise. That's thermal throttling quietly rewriting your results, and if you benchmark anything disk-bound, it's probably already in your numbers without you knowing it.

Why would a reboot change benchmark results at all?

A cold boot gives your CPU, GPU, and NVMe controller a clean thermal slate. Sustained workloads generate heat, heat raises silicon temperature, and every major chip in your system responds to rising temperature by clocking down to stay inside its thermal envelope. This isn't a bug; it's the designed behavior of dynamic frequency scaling.

The effect compounds when a workload is disk-heavy. A benchmark that involves streaming a large model's weights off an NVMe SSD, for instance, showed a 2.95x slowdown at low memory budgets when comparing a heat-soaked machine (24+ hours of sustained load) to a freshly rebooted, idle one on AC power: 13.7 seconds per token versus 4.64 seconds per token. That same workload run at a higher memory budget, where more computation happens on GPU and less depends on raw disk throughput, showed almost no gap at all: 1.81 seconds versus 1.74 seconds, a 1.04x ratio. The more I/O-bound the test, the more the reboot state matters.

Why does NVMe sequential throughput actually drop under sustained load?

NVMe controllers have their own thermal limits, separate from the CPU. Sustained sequential reads, especially large ones pulled with O_DIRECT to bypass the page cache, keep the controller and flash package busy continuously, and that generates heat inside the drive itself, often inside a laptop chassis with limited airflow.

Measured directly, one workload saw sequential read throughput fall from 5.3 GB/s cold to 4.4 GB/s heat-soaked, a drop of roughly 17%. That's before you even count CPU or memory-bandwidth throttling layered on top. When a benchmark is dominated by expert-cache misses or any other pattern of pulling large chunks off disk repeatedly, that throughput drop propagates directly into your latency numbers, and it does so nonlinearly: a config that's almost pure I/O suffers far more than one where computation dominates.

How do I detect thermal throttling in my own I/O-bound benchmarks?

The most reliable signal is comparing a cold-boot run against a run taken after an hour or more of continuous load, using identical parameters both times. If your numbers move by more than roughly 20%, thermal state is contaminating your results, and you need to control for it explicitly rather than treat it as noise. For more on this, see read about node --build-sea not working? fix it on node 24 lts.

A second, cheaper signal: watch whether your numbers degrade progressively across repeated runs in the same session rather than staying flat. A benchmark that gets slower on run five than run one, with no code changes in between, is telling you the machine is heating up under your own test harness. On laptops especially, check whether the chassis feels warm to the touch or whether a fan has spun up mid-run; both are cheap proxies for a thermal state you haven't measured directly.

Why does this matter more for some configurations than others?

Throttling doesn't apply evenly across a workload. Disk-bound configurations, where the CPU or GPU spends most of its time waiting on I/O rather than computing, take the worst of it because they're exposed to both drive throttling and system throttling simultaneously. Configurations that shift more work onto a GPU, or that fit comfortably in a larger cache and avoid re-reading from disk, are far less exposed. This pairs well with our guide to graphrag vs vector rag: when to use each one.

This is exactly the pattern behind that 2.95x versus 1.04x swing: a memory-constrained run that's almost pure sequential disk I/O sits at the mercy of drive thermals, while a well-provisioned run backed by GPU compute barely notices the difference between cold and heat-soaked. If you're benchmarking a system with a tunable memory or cache budget, don't assume a percentage improvement measured under one thermal state transfers to another. Re-verify at both ends before you trust the number.

How should I actually structure a benchmark so the numbers are trustworthy?

Start every serious benchmarking session from an idle, freshly rebooted machine on stable power, and record that baseline explicitly rather than assuming a prior session's environment still applies. Treat that cold-boot number as your reference point, and if you need to compare against sustained-load behavior, run that as a distinct, labeled condition rather than mixing it into the same table without a note.

For anything you actually care about optimizing at the kernel or function level, don't rely on a full end-to-end run to resolve small improvements. A 70-second generation run, for example, can't reliably distinguish a genuine 12% kernel speedup from thermal noise that swings results by 20% or more on its own. Build a dedicated, isolated microbenchmark that exercises just the code path you changed, run it in a tight loop, and compare medians rather than single samples. That isolation is what let one project pin down that a supposedly cheap memcpy was actually consuming 22% of total cost per operation, a fact that a full pipeline benchmark would have buried in noise.

What should I actually report alongside my numbers?

Report the thermal state the benchmark ran under, not just the number itself. That means noting whether the machine was freshly rebooted or had been under load, roughly how long it had been running, and ideally the ambient conditions like whether it was on battery or AC power. Without that context, a reader has no way to know whether your 2x speedup claim reflects your optimization or just a cooler room.

It's also worth stating your own error bars honestly. A reasonable default, based on the swings observed above, is to treat any single timing as accurate to within about 20% at a fixed thermal state, and to expect swings up to roughly 3x when comparing across thermal states on I/O-heavy workloads. Stating that range up front does more for your credibility than a single suspiciously precise number ever will.

Also read: byzantine quorum size formula: why 3-of-4 beats 2-of-3

Isn't thermal throttling just a hardware problem, not something I need to handle in software?

This is the misconception worth correcting directly: thermal throttling is a hardware behavior, but its effect on your benchmark results is entirely a measurement and reporting problem that software practices can control for. You can't stop a chip from downclocking when it's hot, but you absolutely can control whether your benchmark methodology exposes that variability or hides it.

The fix isn't defeating throttling; it's designing your test protocol around it. That means establishing a cold-boot baseline, isolating the specific code path you're measuring in a microbenchmark rather than a full run, and disclosing the thermal conditions alongside every number you publish. Skip those steps, and you're not measuring your code. You're measuring whatever the room temperature happened to be that day.