Your coding agent opened a pile of pull requests this week. How many of its actions did anything other than the agent itself see before they ran? If you can't answer with a number, you don't have an oversight metric yet. You have a feeling.
A frontier lab recently showed what that number looks like. This guide takes the one idea worth copying, that oversight needs a denominator, and turns it into three metrics you can compute from your own agent action logs: pre-execution monitor coverage, flag-to-human review latency, and block/flag/escalation rate.
What the lab reported, and what to check first
Anthropic's post on measuring the pace of AI development inside frontier labs is at https://www.anthropic.com/institute/measuring-pace-of-ai-development. A secondary write-up on dev.to (https://dev.to/neticslabs/anthropics-26-is-a-measurement-story-before-it-is-a-warning-2cdj) summarizes its oversight section.
By that summary, roughly 30,000 agents ran on Anthropic's most-used internal platform in August 2026, and online monitors covered 100% of actions before execution. Over a billion decisions were analyzed, and 0.002% were blocked. About 100,000 transcripts a week were flagged, around 50 of those escalated to a human, and humans reviewed blocked actions within a week.
All of that is self-reported. The dev.to piece is a secondary source, so check each figure against Anthropic's primary post before you quote it anywhere. The write-up also passes along Anthropic's own caveat: it used its own models as evaluators, and a judge model can repeat the mistakes of the model it checks. That caveat shapes Step 5 below.
Your scale is different. You probably run one agent or a few, not 30,000. The method still transfers, because it needs only a log and a count of what should have been in it.
Step 1: Log the fields the metrics need
Write one JSON line per agent action, at the point where the tool call is made. The log has to come from your tool gateway, CI runner, or wrapper, not from the agent's own account of what it did.
| Field | Meaning | |---|---| | action_id | Unique ID, also recorded wherever the action takes effect (commit trailer, CI job ID) | | agent_id | Which agent or bot acted | | tool | What it tried to do (shell, git push, file write, API call) | | verdict | allow, flag, or block, as decided by your monitor | | checked_at | When the monitor ruled, or null if no monitor ran | | executed_at | When the action ran, or null if it never did | | flagged_at, reviewed_at | When it entered the review queue, and when a human closed it | | escalated | True if a human decided it needed follow-up |
A "monitor" can be crude: a regex allowlist, a permission prompt, or a classifier. What matters is that it ruled before execution and left a timestamp.
Step 2: Define the three metrics
| Metric | Formula | Required fields | Warning sign | |---|---|---|---| | Pre-execution coverage | Actions with a verdict recorded at or before execution, divided by all actions that should exist (from an independent source) | action_id, checked_at, executed_at | Coverage looks near 100% only because the denominator is the monitor's own log | | Review latency | Median and max of reviewed_at minus flagged_at, plus count of flags still open | flagged_at, reviewed_at | Median is fine but flags sit open for weeks, or the queue is empty because nothing is ever flagged | | Block, flag and escalation rate | Blocked / total, (flagged + blocked) / total, escalated / (flagged + blocked) | verdict, escalated | A rate of exactly zero, or a rate that drifts down while agent permissions grow |
The denominator in the first row is the whole idea. If you divide by what the monitor logged, you measure the monitor's honesty with itself. Pull the expected action IDs from somewhere the monitor doesn't control, such as your git history for agent-authored commits, CI job records, or the gateway's raw request log.
Step 3: Run a small script over the log
This example is unexecuted. It was written for this guide and has not been run. It needs Python 3.10 or newer and the standard library only, no third-party packages. We cover related ground in see also: sharp vs cloudinary vs imgix for fixed avatar sizes.
## UNEXECUTED example. Python 3.10+, standard library only.
import json
import statistics
from datetime import datetime
def ts(value):
return datetime.fromisoformat(value) if value else None
def covered(a):
checked, ran = ts(a.get('checked_at')), ts(a.get('executed_at'))
return checked is not None and (ran is None or checked <= ran)
def compute(actions, expected_ids):
by_id = {a['action_id']: a for a in actions}
covered_n = sum(1 for i in expected_ids if i in by_id and covered(by_id[i]))
queue = [a for a in actions if a['verdict'] in ('flag', 'block')]
hours = [(ts(a['reviewed_at']) - ts(a['flagged_at'])).total_seconds() / 3600
for a in queue if a.get('reviewed_at')]
open_n = sum(1 for a in queue if not a.get('reviewed_at'))
total = len(actions)
return {
'coverage': covered_n / len(expected_ids),
'coverage_logged_only': sum(covered(a) for a in actions) / total,
'review_median_h': statistics.median(hours) if hours else None,
'review_max_h': max(hours) if hours else None,
'open_flags': open_n,
'block_rate': sum(a['verdict'] == 'block' for a in actions) / total,
'flag_rate': len(queue) / total,
'escalation_rate': (sum(bool(a.get('escalated')) for a in queue) / len(queue)) if queue else None,
}
if __name__ == '__main__':
with open('actions.jsonl') as f:
actions = [json.loads(line) for line in f if line.strip()]
with open('expected_ids.txt') as f:
expected = {line.strip() for line in f if line.strip()}
for key, value in compute(actions, expected).items():
print(key, round(value, 3) if isinstance(value, float) else value)
Sample actions.jsonl (six lines, invented for illustration):
{"action_id":"a1","verdict":"allow","checked_at":"2026-11-16T09:00:01","executed_at":"2026-11-16T09:00:02","flagged_at":null,"reviewed_at":null,"escalated":false}
{"action_id":"a2","verdict":"allow","checked_at":"2026-11-16T09:05:00","executed_at":"2026-11-16T09:05:01","flagged_at":null,"reviewed_at":null,"escalated":false}
{"action_id":"a3","verdict":"flag","checked_at":"2026-11-16T10:00:00","executed_at":"2026-11-16T10:00:01","flagged_at":"2026-11-16T10:00:00","reviewed_at":"2026-11-16T10:30:00","escalated":false}
{"action_id":"a4","verdict":"block","checked_at":"2026-11-16T11:00:00","executed_at":null,"flagged_at":"2026-11-16T11:00:00","reviewed_at":"2026-11-16T17:00:00","escalated":true}
{"action_id":"a5","verdict":"flag","checked_at":"2026-11-14T12:00:00","executed_at":"2026-11-14T12:00:01","flagged_at":"2026-11-14T12:00:00","reviewed_at":"2026-11-16T12:00:00","escalated":false}
{"action_id":"a6","verdict":"allow","checked_at":null,"executed_at":"2026-11-16T14:00:00","flagged_at":null,"reviewed_at":null,"escalated":false}
Sample expected_ids.txt lists a1 through a7, one per line. Action a7 came from the git history but never appeared in the log.
Expected output if the script runs as written:
coverage 0.714
coverage_logged_only 0.833
review_median_h 6.0
review_max_h 48.0
open_flags 0
block_rate 0.167
flag_rate 0.5
escalation_rate 0.333
Step 4: Verify the numbers are right
Do these in order before you trust any dashboard built on the script.
- Hand-count the sample. Five of the seven expected actions have a verdict at or before execution, so coverage is 5/7. If your run prints anything else, the script or your copy of the data is wrong.
- Reconcile the denominator. Count agent-authored commits or CI jobs in an independent system and compare them to the action IDs. A gap is the finding, not noise.
- Check timestamps. Confirm clocks agree across the gateway and the review tool, since review latency subtracts times written by different systems.
- Rerun after a deliberate break. Delete one line from the log and confirm coverage drops while
coverage_logged_onlybarely moves.
The failure case is already in the sample. Action a7 never reached the log, so logged-only coverage reads 83.3% while the real figure is 71.4%.
Action a6 shows the other common leak: the agent called a tool directly and no monitor ruled on it. Both gaps look fine in a dashboard built only on the monitor's records.
Step 5: Handle the self-judge problem
Say a classifier flags your agent's actions, and that classifier comes from the same model family as the agent. It may share the agent's blind spots. The dev.to write-up says Anthropic named this same weakness in its own setup, and it applies one level down to any flagging pipeline: whatever produces your flag rate is itself an AI judge. Add a human spot-check.
Also read: a closer look at rds inaccessible encryption credentials: kms triage guide
- Each week, sample a fixed number of flagged actions and have a person label them correct or incorrect.
- Sample the same number of unflagged actions, chosen at random, and label those too. This is the only way to see what the classifier misses.
- Record the labeler, the date and the verdict next to the action ID so the check is auditable.
- Where you can, use a classifier from a different model family than the agent, or a plain rule-based check as a second opinion.
- Track the miss rate from step 2 as a fourth number beside the other three.
Small samples give loose bounds. As a statistical rule of thumb, finding zero misses in 20 unflagged samples still leaves a plausible miss rate of about 15% at 95% confidence. Sample more if the actions are high stakes.
Where the numbers mislead
A low block rate can mean good agents or a blind monitor. A short median review time can hide a long tail, which is why the script prints the max and the open count. Escalation rate means little if reviewers rubber-stamp, so the spot-check labels matter more than any rate.
Volume is the other trap. At a 50% flag rate, as in the toy sample, no person will keep up once agents multiply. Tune the monitor so humans see a queue they can actually finish.
What to expect next
Your first run may show coverage well below what you assumed, mostly from actions that bypass the gateway. Fix those paths first, since each one is an action nobody supervised. After that, review latency becomes the number to watch, and the spot-check labels will tell you whether your flag rate means anything.



