Sample Size Calculator for Robot Policy Comparison

You ran Policy A and Policy B for 100 episodes each. A hit 90 successes, B hit 92. Somebody on your team already opened a slide deck titled "Policy B Wins." Here's the uncomfortable math: with those sample sizes, a 95% confidence interval on the difference runs from about -5.9pp to +9.9pp. Zero sits comfortably inside that range, which means your data cannot rule out A actually being better.

This is the point where most teams stop, shrug, and ship the model anyway. The better move is to figure out how many rollouts you actually needed before running the experiment, then use that number going forward. This is a walkthrough for doing exactly that, with a Python snippet you can drop into your evaluation pipeline today.

Why Raw Percentages Mislead You

Success rates on robot benchmarks are Bernoulli outcomes: each episode is a coin flip that lands on success or failure. A single point estimate like "90%" hides a wide plausible range whenever the sample is small. The RoboLab v4 benchmark makes this concrete: at 10 episodes per task, a 90% success rate carries a 95% confidence interval spanning roughly 19 percentage points. Even bumping that up to 100 rollouts only narrows the interval to about six points.

That's not a rounding error. It means two policies separated by 2pp on a 100-rollout benchmark are statistically indistinguishable in most cases. The fix isn't a better point estimate. It's knowing, before you spend compute on rollouts, how many trials you need to detect the effect size you actually care about.

Defining Your Minimum Detectable Effect

The first decision is not statistical, it's engineering. Ask what gap between policies would actually change your deployment decision. If a 1pp difference wouldn't move you to ship Policy B over Policy A, don't design your experiment to detect 1pp. Most teams land on a minimum detectable effect (MDE) between 3pp and 5pp for practical policy comparisons, since that range tends to correspond to differences that matter in production reliability.

Once you pick an MDE, you need an effect size metric that works well for proportions. Cohen's h is the standard choice: it transforms two proportions using an arcsine transformation, which stabilizes variance near 0% and 100% where the raw percentage-point difference behaves poorly. The formula is straightforward:

import numpy as np

def cohens_h(p1: float, p2: float) -> float:
    phi1 = 2 * np.arcsin(np.sqrt(p1))
    phi2 = 2 * np.arcsin(np.sqrt(p2))
    return phi1 - phi2
``` We cover related ground in [see also: best llm quant format for every apple silicon chip](/best-llm-quant-format-for-every-apple-silicon-chip).

For a baseline of 90% and a target of 95% (a 5pp MDE), Cohen's h comes out to roughly 0.18, which statisticians typically classify as a small-to-medium effect. That classification matters: small effects require large samples, full stop. There's no clever test that gets around this.

## Independent Samples: The Sample Size Table You Need

If Policy A and Policy B run on different rollouts, different seeds, and possibly different tasks, you're in independent-sample territory. This is the simpler design conceptually but the more expensive one statistically, because you get no benefit from shared randomness between conditions.

Here's a power analysis function using statsmodels, targeting 80% power at alpha = 0.05, which are standard defaults for engineering decisions: This pairs well with [bar chart vs histogram: when to use each in python in depth](/bar-chart-vs-histogram-when-to-use-each-in-python).

```python
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

def required_n_independent(p1: float, p2: float, alpha: float = 0.05, power: float = 0.8) -> int:
    effect_size = proportion_effectsize(p1, p2)
    analysis = NormalIndPower()
    n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power, ratio=1.0)
    return int(np.ceil(n))

## Example: baseline 90%, MDE of 5pp (target 95%)
n_per_group = required_n_independent(0.90, 0.95)
print(f"Rollouts needed per policy: {n_per_group}")

Running this for a baseline near 90% and a 5pp MDE typically returns a few hundred rollouts per policy, which is manageable for simulation but painful for physical robot trials. Tightening the MDE to 2pp near that same baseline pushes the requirement into the thousands per arm, consistent with the broader finding that resolving a true 2pp gap near 90% success generally demands sample sizes in the low thousands. This is exactly why the earlier 100-vs-100 comparison couldn't settle anything: it was underpowered by an order of magnitude for the effect size it was implicitly trying to detect.

Paired Design: Where You Actually Save Rollouts

If you can run both policies on identical tasks, scene configurations, and random seeds, switch to a paired design. This is where sample size requirements can drop substantially, because you're no longer comparing two noisy averages, you're comparing outcomes on matched trials.

In a paired setup, only the discordant outcomes matter: cases where one policy succeeds and the other fails. Concordant outcomes, both succeed or both fail, contribute nothing to detecting a difference. McNemar's test is built for exactly this structure. Power calculations for paired designs depend heavily on the discordant rate, meaning how often the two policies actually disagree on a given trial:

Also read: related topic: claude on aws: bedrock vs platform security checklist

def required_n_paired(p_discordant_a_wins: float, p_discordant_b_wins: float, alpha: float = 0.05, power: float = 0.8) -> int:
    from statsmodels.stats.power import GofChisquarePower
    # Approximate via McNemar power using discordant proportions
    p_total_discordant = p_discordant_a_wins + p_discordant_b_wins
    effect = abs(p_discordant_a_wins - p_discordant_b_wins) / np.sqrt(p_total_discordant * (1 - p_total_discordant))
    analysis = GofChisquarePower()
    n_discordant = analysis.solve_power(effect_size=effect, alpha=alpha, power=power, n_bins=2)
    return int(np.ceil(n_discordant / p_total_discordant)) if p_total_discordant > 0 else float('inf')

The catch: if your two policies rarely disagree, meaning the discordant rate is low, paired design's efficiency gain shrinks fast. Two policies that behave almost identically on the same tasks produce few informative trials no matter how many total rollouts you run. Paired design also doesn't fix confounded task selection; if certain tasks systematically favor one policy, you still need to report the full contingency table, not just the aggregate.

The practical decision rule looks like this: choose paired design whenever you can control seeds and task assignment, since it typically requires far fewer total rollouts to hit the same power target as independent sampling. Reserve independent-sample analysis for cases where policies run in genuinely separate environments, different robots, different labs, different task generators, where pairing isn't physically possible. Either way, run the sample size calculation before you spend rollout budget, not after you've already collected disappointing 100-vs-100 numbers and are trying to rescue a conclusion. If the required sample size busts your compute budget, raise your MDE threshold, tighten the paired design, or explicitly label the result as exploratory rather than conclusive.