Skip to content
Boomspot
  • Home
Loading...
Boomspot

Daily tech news, software development coverage, Apple reporting, and the gear behind modern music making.

TwitterLinkedIn

Browse

  • Categories
  • Tags
  • Authors

Company

  • About
  • Contact

Legal

  • Privacy Policy
  • Terms of Service
  • Unsubscribe

© 2026 Boomspot. All rights reserved.

Built by Boomspot
Updated hourly

AI Content Disclosure: Articles on Boomspot are researched, written, and edited with the assistance of advanced AI systems. We combine software-assisted research with editorial oversight to deliver useful, accurate, and practical technical and music production content. Learn more about our editorial approach.

Browse by Category

Technology681Coding193SEO66Linux66Studio Gear47Music Production42Apple Rumors11

Popular Posts

Speed Up Linux Kernel Compile Time: 7 Practical Fixes

Speed Up Linux Kernel Compile Time: 7 Practical Fixes

8 min read
Is Linux on PS5 Worth It? pLinux5-NG vs. a Mini PC

Is Linux on PS5 Worth It? pLinux5-NG vs. a Mini PC

7 min read
How to Test Budget DJ Headphones and Speakers at Home

How to Test Budget DJ Headphones and Speakers at Home

10 min read
Beginner-Friendly Terminal Editor: TTT vs micro vs Edit

Beginner-Friendly Terminal Editor: TTT vs micro vs Edit

8 min read
Affiliate Site Audit: Measure Your Google Traffic Risk

Affiliate Site Audit: Measure Your Google Traffic Risk

11 min read

Recent Posts

Google Automated Calls Business Profile Setting Audit

Google Automated Calls Business Profile Setting Audit

Nov 20, 2026•9 min
Voice Agent Latency Budget: Where a Turn's 1,000 ms Goes

Voice Agent Latency Budget: Where a Turn's 1,000 ms Goes

Nov 20, 2026•8 min
Ansible Touch Always Changed: Make the File Task Idempotent

Ansible Touch Always Changed: Make the File Task Idempotent

Nov 20, 2026•7 min
Yamaha NS-10 Alternatives for a Home Studio: What to Buy

Yamaha NS-10 Alternatives for a Home Studio: What to Buy

Nov 20, 2026•7 min
Check Installed OpenJDK Versions on Linux After Java4Shell

Check Installed OpenJDK Versions on Linux After Java4Shell

Nov 20, 2026•9 min
  1. Home
  2. Coding
  3. How to Measure AI Agent Oversight Metrics in Your Repo
coding8 min read

How to Measure AI Agent Oversight Metrics in Your Repo

A frontier lab reports oversight with denominators. You can too. Log a few fields, run one script, and learn how much of your agent's work a human or monitor actually saw.

S

Staff

November 20, 2026

Reviewed byDorian

How to Measure AI Agent Oversight Metrics in Your Repo

Your coding agent opened a pile of pull requests this week. How many of its actions did anything other than the agent itself see before they ran? If you can't answer with a number, you don't have an oversight metric yet. You have a feeling.

A frontier lab recently showed what that number looks like. This guide takes the one idea worth copying, that oversight needs a denominator, and turns it into three metrics you can compute from your own agent action logs: pre-execution monitor coverage, flag-to-human review latency, and block/flag/escalation rate.

What the lab reported, and what to check first

Anthropic's post on measuring the pace of AI development inside frontier labs is at https://www.anthropic.com/institute/measuring-pace-of-ai-development. A secondary write-up on dev.to (https://dev.to/neticslabs/anthropics-26-is-a-measurement-story-before-it-is-a-warning-2cdj) summarizes its oversight section.

By that summary, roughly 30,000 agents ran on Anthropic's most-used internal platform in August 2026, and online monitors covered 100% of actions before execution. Over a billion decisions were analyzed, and 0.002% were blocked. About 100,000 transcripts a week were flagged, around 50 of those escalated to a human, and humans reviewed blocked actions within a week.

All of that is self-reported. The dev.to piece is a secondary source, so check each figure against Anthropic's primary post before you quote it anywhere. The write-up also passes along Anthropic's own caveat: it used its own models as evaluators, and a judge model can repeat the mistakes of the model it checks. That caveat shapes Step 5 below.

Your scale is different. You probably run one agent or a few, not 30,000. The method still transfers, because it needs only a log and a count of what should have been in it.

Step 1: Log the fields the metrics need

Write one JSON line per agent action, at the point where the tool call is made. The log has to come from your tool gateway, CI runner, or wrapper, not from the agent's own account of what it did.

| Field | Meaning | |---|---| | action_id | Unique ID, also recorded wherever the action takes effect (commit trailer, CI job ID) | | agent_id | Which agent or bot acted | | tool | What it tried to do (shell, git push, file write, API call) | | verdict | allow, flag, or block, as decided by your monitor | | checked_at | When the monitor ruled, or null if no monitor ran | | executed_at | When the action ran, or null if it never did | | flagged_at, reviewed_at | When it entered the review queue, and when a human closed it | | escalated | True if a human decided it needed follow-up |

A "monitor" can be crude: a regex allowlist, a permission prompt, or a classifier. What matters is that it ruled before execution and left a timestamp.

Step 2: Define the three metrics

| Metric | Formula | Required fields | Warning sign | |---|---|---|---| | Pre-execution coverage | Actions with a verdict recorded at or before execution, divided by all actions that should exist (from an independent source) | action_id, checked_at, executed_at | Coverage looks near 100% only because the denominator is the monitor's own log | | Review latency | Median and max of reviewed_at minus flagged_at, plus count of flags still open | flagged_at, reviewed_at | Median is fine but flags sit open for weeks, or the queue is empty because nothing is ever flagged | | Block, flag and escalation rate | Blocked / total, (flagged + blocked) / total, escalated / (flagged + blocked) | verdict, escalated | A rate of exactly zero, or a rate that drifts down while agent permissions grow |

The denominator in the first row is the whole idea. If you divide by what the monitor logged, you measure the monitor's honesty with itself. Pull the expected action IDs from somewhere the monitor doesn't control, such as your git history for agent-authored commits, CI job records, or the gateway's raw request log.

Step 3: Run a small script over the log

This example is unexecuted. It was written for this guide and has not been run. It needs Python 3.10 or newer and the standard library only, no third-party packages. We cover related ground in see also: sharp vs cloudinary vs imgix for fixed avatar sizes.

## UNEXECUTED example. Python 3.10+, standard library only.
import json
import statistics
from datetime import datetime

def ts(value):
    return datetime.fromisoformat(value) if value else None

def covered(a):
    checked, ran = ts(a.get('checked_at')), ts(a.get('executed_at'))
    return checked is not None and (ran is None or checked <= ran)

def compute(actions, expected_ids):
    by_id = {a['action_id']: a for a in actions}
    covered_n = sum(1 for i in expected_ids if i in by_id and covered(by_id[i]))
    queue = [a for a in actions if a['verdict'] in ('flag', 'block')]
    hours = [(ts(a['reviewed_at']) - ts(a['flagged_at'])).total_seconds() / 3600
             for a in queue if a.get('reviewed_at')]
    open_n = sum(1 for a in queue if not a.get('reviewed_at'))
    total = len(actions)
    return {
        'coverage': covered_n / len(expected_ids),
        'coverage_logged_only': sum(covered(a) for a in actions) / total,
        'review_median_h': statistics.median(hours) if hours else None,
        'review_max_h': max(hours) if hours else None,
        'open_flags': open_n,
        'block_rate': sum(a['verdict'] == 'block' for a in actions) / total,
        'flag_rate': len(queue) / total,
        'escalation_rate': (sum(bool(a.get('escalated')) for a in queue) / len(queue)) if queue else None,
    }

if __name__ == '__main__':
    with open('actions.jsonl') as f:
        actions = [json.loads(line) for line in f if line.strip()]
    with open('expected_ids.txt') as f:
        expected = {line.strip() for line in f if line.strip()}
    for key, value in compute(actions, expected).items():
        print(key, round(value, 3) if isinstance(value, float) else value)

Sample actions.jsonl (six lines, invented for illustration):

{"action_id":"a1","verdict":"allow","checked_at":"2026-11-16T09:00:01","executed_at":"2026-11-16T09:00:02","flagged_at":null,"reviewed_at":null,"escalated":false}
{"action_id":"a2","verdict":"allow","checked_at":"2026-11-16T09:05:00","executed_at":"2026-11-16T09:05:01","flagged_at":null,"reviewed_at":null,"escalated":false}
{"action_id":"a3","verdict":"flag","checked_at":"2026-11-16T10:00:00","executed_at":"2026-11-16T10:00:01","flagged_at":"2026-11-16T10:00:00","reviewed_at":"2026-11-16T10:30:00","escalated":false}
{"action_id":"a4","verdict":"block","checked_at":"2026-11-16T11:00:00","executed_at":null,"flagged_at":"2026-11-16T11:00:00","reviewed_at":"2026-11-16T17:00:00","escalated":true}
{"action_id":"a5","verdict":"flag","checked_at":"2026-11-14T12:00:00","executed_at":"2026-11-14T12:00:01","flagged_at":"2026-11-14T12:00:00","reviewed_at":"2026-11-16T12:00:00","escalated":false}
{"action_id":"a6","verdict":"allow","checked_at":null,"executed_at":"2026-11-16T14:00:00","flagged_at":null,"reviewed_at":null,"escalated":false}

Sample expected_ids.txt lists a1 through a7, one per line. Action a7 came from the git history but never appeared in the log.

Expected output if the script runs as written:

coverage 0.714
coverage_logged_only 0.833
review_median_h 6.0
review_max_h 48.0
open_flags 0
block_rate 0.167
flag_rate 0.5
escalation_rate 0.333

Step 4: Verify the numbers are right

Do these in order before you trust any dashboard built on the script.

  1. Hand-count the sample. Five of the seven expected actions have a verdict at or before execution, so coverage is 5/7. If your run prints anything else, the script or your copy of the data is wrong.
  2. Reconcile the denominator. Count agent-authored commits or CI jobs in an independent system and compare them to the action IDs. A gap is the finding, not noise.
  3. Check timestamps. Confirm clocks agree across the gateway and the review tool, since review latency subtracts times written by different systems.
  4. Rerun after a deliberate break. Delete one line from the log and confirm coverage drops while coverage_logged_only barely moves.

The failure case is already in the sample. Action a7 never reached the log, so logged-only coverage reads 83.3% while the real figure is 71.4%.

Action a6 shows the other common leak: the agent called a tool directly and no monitor ruled on it. Both gaps look fine in a dashboard built only on the monitor's records.

Step 5: Handle the self-judge problem

Say a classifier flags your agent's actions, and that classifier comes from the same model family as the agent. It may share the agent's blind spots. The dev.to write-up says Anthropic named this same weakness in its own setup, and it applies one level down to any flagging pipeline: whatever produces your flag rate is itself an AI judge. Add a human spot-check.

Also read: a closer look at rds inaccessible encryption credentials: kms triage guide

  1. Each week, sample a fixed number of flagged actions and have a person label them correct or incorrect.
  2. Sample the same number of unflagged actions, chosen at random, and label those too. This is the only way to see what the classifier misses.
  3. Record the labeler, the date and the verdict next to the action ID so the check is auditable.
  4. Where you can, use a classifier from a different model family than the agent, or a plain rule-based check as a second opinion.
  5. Track the miss rate from step 2 as a fourth number beside the other three.

Small samples give loose bounds. As a statistical rule of thumb, finding zero misses in 20 unflagged samples still leaves a plausible miss rate of about 15% at 95% confidence. Sample more if the actions are high stakes.

Where the numbers mislead

A low block rate can mean good agents or a blind monitor. A short median review time can hide a long tail, which is why the script prints the max and the open count. Escalation rate means little if reviewers rubber-stamp, so the spot-check labels matter more than any rate.

Volume is the other trap. At a 50% flag rate, as in the toy sample, no person will keep up once agents multiply. Tune the monitor so humans see a queue they can actually finish.

What to expect next

Your first run may show coverage well below what you assumed, mostly from actions that bypass the gateway. Fix those paths first, since each one is an action nobody supervised. After that, review latency becomes the number to watch, and the spot-check labels will tell you whether your flag rate means anything.

Tags

Artificial IntelligenceSoftware DevelopmentCoding Best PracticesAi AgentsEthical Ai

Keep reading

Unlocking ChatGPT Developer Mode: Full MCP Client Access
Coding•4 min read

Unlocking ChatGPT Developer Mode: Full MCP Client Access

Unlock the power of ChatGPT Developer Mode with full MCP client access. Discover how to enhance your coding projects and streamline development.

Sep 11, 2025

Mastering MCP Elicitation for Enhanced AI Interactions
Coding•3 min read

Mastering MCP Elicitation for Enhanced AI Interactions

Discover the power of MCP elicitation in creating seamless AI interactions, from streamlining development to improving user satisfaction.

Sep 10, 2025

Rethinking Network Security with Physics Principles
Coding•3 min read

Rethinking Network Security with Physics Principles

Discover how viewing networks as energy systems, with data flows and security mechanisms akin to electricity and circuits, can revolutionize cloud security.

Sep 7, 2025

More stories for your next project

Get tech, coding, and music production updates in your inbox.

Unsubscribe anytime.