Your agent's moderation gate flags a comment with 0.73 confidence. Do you auto-approve it, block it, or send it to a human? If your answer is "I picked 0.7 because it felt reasonable," you have a production incident waiting to happen.
Most teams building agent decision gates skip straight from "the model returns a probability" to "we chose a cutoff" without checking whether that cutoff means anything. A threshold is only as good as the data used to set it. This guide walks through the calibration workflow: how to log a decision without acting on it, how to turn those logs into an accuracy measurement, and how to pick different thresholds depending on how expensive a wrong answer actually is.
The examples below use a generic probability output — the kind you get from a Choice, Score, or Noul-style decision primitive, or from any classifier that returns a confidence score. The workflow doesn't care which model produced the number.
Step 1: Run the decision in shadow mode
Before any threshold controls live behavior, run the decision silently alongside your existing logic. In shadow mode, the model makes a call, you record it, and then you ignore it — the original code path still decides what actually happens.
This matters because you cannot calibrate a threshold you haven't tested against reality. If you flip on automation before you know the model's accuracy at 0.6 versus 0.9, you're not calibrating. You're guessing with extra steps.
Here's a minimal, unexecuted example of shadow-mode logging for a tool-routing decision:
// Unexecuted example: illustrates schema, not verified output
async function shadowRouteDecision(request, existingRouter) {
const decision = await decisionModel.evaluate({
type: "choice",
options: ["search_tool", "calculator_tool", "escalate"],
instructions: "Which tool should handle this request?",
input: request
});
// The real system still uses the existing router.
const actualRoute = existingRouter(request);
await logDecision({
requestId: request.id,
questionVersion: "route_v3",
modelVersion: "decision-model-2026-08",
proposedAction: decision.choice,
probability: decision.confidence,
actualAction: actualRoute,
timestamp: Date.now()
}); This pairs well with [see also: check if a browser tool uploads your file secretly](/check-if-a-browser-tool-uploads-your-file-secretly).
return actualRoute; // shadow decision never controls behavior
}
Expected output: a log entry per request, with the model's guess sitting next to what actually happened, and zero change in user-facing behavior. Verify this step by confirming your production metrics — latency, error rate, routing outcomes — are identical before and after adding the shadow call. If they shift at all, something is leaking into the live path. That's a bug, not a calibration finding.
Step 2: Log the fields you'll need later, not just the probability
A lone probability number is nearly useless six weeks from now. You won't remember which prompt version produced it, which model served the request, or whether the eventual outcome even confirmed or contradicted the guess.
At minimum, log these fields for every shadowed decision:
- Question version — the exact wording or schema version of the prompt. Wording changes shift calibration, so treat every edit as a new question.
- Model version — which underlying model or model snapshot produced the probability.
- Proposed action — what the decision layer would have done if it were live.
- Probability — the raw confidence score, unrounded.
- Actual outcome — what really happened, either from the existing deterministic path or from downstream ground truth (a user's follow-up action, a moderator's ruling, a later error report).
- Timestamp and request ID — for joining logs across systems and debugging drift over time.
Storing the question version, model version, probability, proposed action, and actual outcome together is what lets you review a labeled sample later. Skipping any of these fields means you'll eventually re-run shadow mode from scratch when you notice the gap.
Step 3: Label a representative sample and measure accuracy by probability band
Once you've accumulated a few weeks of shadow logs, pull a representative sample and label it: was the proposed action actually correct? For some decisions — tool routing, relevance checks — the "actual outcome" field already tells you this. For others, like moderation triage or quality scoring, you may need a human reviewer to confirm ground truth on a sample, since the deterministic fallback path doesn't always reveal whether the model's guess was right.
Then bucket the labeled sample into probability bands — for example 0.5–0.6, 0.6–0.7, 0.7–0.8, 0.8–0.9, 0.9–1.0 — and compute observed accuracy within each band. This is the step teams skip, and it's the one that actually justifies a threshold.
A well-calibrated model should show accuracy roughly matching its stated confidence: decisions at 0.9 should be right about 90% of the time, not 60%. If your 0.9-band decisions are only right 65% of the time, the model is overconfident for this task. Any threshold you set needs to be higher than intuition suggests, or you need a different question altogether. Splitting a compound judgment into atomic questions, rather than asking one vague prompt, tends to produce better-calibrated bands, because each question has a narrower, more checkable claim.
Step 4: Convert probability bands into an actual cutoff
With accuracy-by-band data in hand, the threshold decision becomes concrete instead of arbitrary. You're no longer picking 0.7 because it looks confident. You're picking the lowest band where observed accuracy clears your acceptable error rate for that specific action.
If 0.85 and above shows 97% accuracy, and 0.7–0.85 shows 88% accuracy, your threshold depends entirely on what a wrong answer costs. For a read-only relevance filter, 88% might be fine. For anything that deletes data, it isn't close.
This is where reversibility enters the calculation, and it's the piece most calibration write-ups skip entirely.
Step 5: Set different thresholds based on reversibility, not universal confidence
A single global threshold, applied to every kind of decision, ignores the fact that a wrong guess on a search-relevance filter costs nothing and a wrong guess on an access-revocation gate costs a security incident. The threshold needs to account for what happens when the model is wrong, not just how often it's wrong.
| Action type | Example | Acceptable automation confidence | Fallback behavior | |---|---|---|---| | Read-only | Search relevance, quality scoring, routing to a read-only tool | Moderate threshold acceptable after validation (band-dependent, roughly the 0.7–0.8 range) | Log and monitor; low cost of error | | Reversible | Draft edits, non-destructive state changes, tagging | Higher threshold required; uncertainty should preserve the original state | Default to "no action" or "keep" when below threshold | | Irreversible | Deletion, payments, access revocation | No confidence level should act alone | Always route to human or stronger-model review, regardless of probability |
For read-only actions, automation may be acceptable after validation. For reversible actions, uncertainty should usually preserve the original state. For irreversible effects like deletion or payments, a probabilistic component should never be the sole authority. Notice the third row has no threshold at all — that's intentional. No band of observed accuracy justifies letting a probability model be the only gate on something you can't undo.
Also read: go deeper on how to build a shutdown ritual for remote developers
Step 6: Add a fallback for the cases that don't fit the model
Even a well-calibrated threshold needs an escape hatch. Malformed responses, timeouts, and probabilities that land exactly on your cutoff should all route to the original deterministic path or a human reviewer, not to a coin flip in your code. Building this fallback in from day one means a bad model update or a schema change degrades gracefully instead of silently corrupting production behavior.
Treat this fallback as part of the threshold, not an afterthought. A threshold without a defined failure mode isn't a complete calibration — it's half of one.
What to expect next
Once shadow logging and banded accuracy measurement are running, expect the first few weeks to surface surprises: question wording that performs worse than expected, model versions that drift after an update, or decision categories that need splitting into narrower atomic questions before they calibrate cleanly at all. Re-run the labeling step whenever you change a prompt, swap a model, or notice accuracy drift in production logs. Calibration isn't a one-time setup — it's a recurring maintenance task tied to every version change in your decision layer.
