Skip to content
Boomspot
  • Home
Loading...
Boomspot

Daily tech news, software development coverage, Apple reporting, and the gear behind modern music making.

TwitterLinkedIn

Browse

  • Categories
  • Tags
  • Authors

Company

  • About
  • Contact

Legal

  • Privacy Policy
  • Terms of Service
  • Unsubscribe

© 2026 Boomspot. All rights reserved.

Built by Boomspot
Updated hourly

AI Content Disclosure: Articles on Boomspot are researched, written, and edited with the assistance of advanced AI systems. We combine software-assisted research with editorial oversight to deliver useful, accurate, and practical technical and music production content. Learn more about our editorial approach.

Browse by Category

Technology680Coding189Linux63SEO62Studio Gear43Music Production40Apple Rumors11

Popular Posts

Speed Up Linux Kernel Compile Time: 7 Practical Fixes

Speed Up Linux Kernel Compile Time: 7 Practical Fixes

8 min read
Is Linux on PS5 Worth It? pLinux5-NG vs. a Mini PC

Is Linux on PS5 Worth It? pLinux5-NG vs. a Mini PC

7 min read
How to Test Budget DJ Headphones and Speakers at Home

How to Test Budget DJ Headphones and Speakers at Home

10 min read
Beginner-Friendly Terminal Editor: TTT vs micro vs Edit

Beginner-Friendly Terminal Editor: TTT vs micro vs Edit

8 min read
Affiliate Site Audit: Measure Your Google Traffic Risk

Affiliate Site Audit: Measure Your Google Traffic Risk

11 min read

Recent Posts

Suno Bridge with Ableton Live 12: Try the Beta or Wait?

Suno Bridge with Ableton Live 12: Try the Beta or Wait?

Nov 17, 2026•7 min
Google Places API 120 Results Limit Workaround (Python)

Google Places API 120 Results Limit Workaround (Python)

Nov 17, 2026•9 min
Why SpaceX Wants to Catch Starship's Upper Stage, Explained

Why SpaceX Wants to Catch Starship's Upper Stage, Explained

Nov 16, 2026•6 min
Wine on ARM64 AVX Support: Will Your Windows Apps Run?

Wine on ARM64 AVX Support: Will Your Windows Apps Run?

Nov 16, 2026•6 min
Do Audio Plugins Need a GPU? Check Before You Buy It

Do Audio Plugins Need a GPU? Check Before You Buy It

Nov 16, 2026•7 min
  1. Home
  2. SEO
  3. Block GPTBot, Allow OAI-SearchBot: A robots.txt Audit
seo9 min read

Block GPTBot, Allow OAI-SearchBot: A robots.txt Audit

You can keep OpenAI's training crawler out and still let its search crawler in. Here's the robots.txt, the three policies, and the checks that prove a CDN isn't undoing it.

S

Staff

November 16, 2026

Reviewed byDorian

Block GPTBot, Allow OAI-SearchBot: A robots.txt Audit

Blocking GPTBot does not remove your site from ChatGPT search. A different crawler, OAI-SearchBot, controls that, and a wildcard rule, a CDN, or a firewall can quietly block the wrong one. This audit walks you through picking a policy, writing the rules, and proving they work.

What the two settings control

OpenAI uses separate crawlers for training and for search. GPTBot collects training data. OAI-SearchBot is the search crawler. According to Ahrefs' guide to where ChatGPT gets its information, the two settings are independent, and blocking GPTBot doesn't remove you from ChatGPT's search results. The same guide says OpenAI may use a single crawl for both purposes if you allow both.

That last detail matters when you read your logs later. With both allowed, you may see less separate traffic from each bot than you expect.

Crawler names and behavior can change. Before you ship anything, confirm the current user-agent strings and IP range lists in OpenAI's crawler documentation. Check whether it lists other agents you want rules for, too.

Step 1: Pick a policy

You have three realistic options.

  1. Allow both. You want ChatGPT to learn your brand and fetch your pages for live answers.
  2. Block training only. You don't want your content used to train future models, but you want OAI-SearchBot to reach your pages.
  3. Block both. You want no OpenAI crawling, and you accept the visibility cost.

The training tradeoff is real. The Ahrefs guide says blocking means future models learn less about you from your own site, and that if you want ChatGPT to know your brand, you probably want to leave it open. The same guide calls GPTBot the most-blocked AI crawler in a study of about 140 million websites, so blocking is common. Big publishers often do it to keep leverage in licensing deals. A small brand with no such leverage has a weaker case.

Step 2: Write the rules (illustrative before and after)

The example below is illustrative. The site, the paths, and the existing rules are hypothetical, so adapt them to your own file.

Before: the file most sites start with

## ILLUSTRATIVE EXAMPLE: example.com robots.txt (before)
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

This file blocks neither bot from public pages. Both GPTBot and OAI-SearchBot fall under the wildcard group, so they follow only the admin and cart restrictions.

After, policy A: allow both

You don't need to add anything. If you want an explicit record for the next person who edits the file, name the bots and leave them open:

## ILLUSTRATIVE: allow both
User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /cart/

After, policy B: block training only

## ILLUSTRATIVE: block training, allow search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /cart/

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot gets its own group here. A crawler follows the most specific group that matches its name and generally ignores the wildcard group once it finds one. So repeat any path restrictions you still want in each named group. See more on how to audit third-party scripts after the debugbear hack for additional background.

After, policy C: block both

## ILLUSTRATIVE: block both
User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

The mistake to avoid

Some files have User-agent: * followed by Disallow: /, common on staging sites and locked-down setups. That blocks OAI-SearchBot unless it has its own group, as in policy B.

Also avoid a partial path block on GPTBot if your goal is to opt out of training. Only the paths you disallow are covered.

Step 3: Verify the live file

Don't audit the file in your repo. Audit what the internet receives.

  1. Fetch it from outside your network: curl -s https://example.com/robots.txt
  2. Check the status code with curl -I https://example.com/robots.txt. You want a 200, not a redirect chain, a 403, or a 5xx.
  3. Confirm the content matches what you deployed. CDNs sometimes cache an old copy, and a staging file pushed to production is a classic failure.
  4. Check each hostname you serve (www, non-www, subdomains). Every host needs its own file.

If you manage several environments, diff the live file against your intended version in a script. It takes a minute and catches silent regressions.

Step 4: Test each user-agent against the rules

Run a robots.txt parser or tester and see how each name resolves. Test these five cases:

  • GPTBot against /
  • GPTBot against a key article URL
  • OAI-SearchBot against /
  • OAI-SearchBot against a key article URL
  • OAI-SearchBot against /admin/

Under policy B, both GPTBot tests and the /admin/ test should come back blocked. The two OAI-SearchBot tests on public pages should come back allowed. This pairs well with ai performance review questions for seo teams that work in depth.

Then test the server layer, which robots.txt can't speak for. Send requests with each user-agent string:

curl -I -A 'OAI-SearchBot' https://example.com/some-article/
curl -I -A 'GPTBot' https://example.com/some-article/

A spoofed user-agent from your laptop isn't the real crawler, and bot-management products often verify by IP. A 200 for your spoof is encouraging, not proof. A 403 or a challenge page is a strong warning sign.

Step 5: Review CDN and WAF bot rules

This is where the quiet failures live. A robots.txt that says "allow" means nothing if the firewall returns a 403 before the crawler reads a page.

Open your CDN or WAF dashboard and look for these:

  1. A managed "block AI bots" or "AI crawlers" toggle. Some block by category, which can catch OAI-SearchBot along with GPTBot.
  2. Custom rules that match on user-agent substrings, such as "bot" or "GPT".
  3. Rate limits and bot-score thresholds that challenge unfamiliar crawlers.
  4. Country or ASN blocks that might cover the ranges OpenAI crawls from.
  5. Any allowlist built on IP ranges. Compare it with the ranges OpenAI currently publishes, because stale allowlists cause intermittent blocks.

For policy B, the cleanest setup is usually a firewall rule that blocks GPTBot explicitly, plus an allow rule for OAI-SearchBot placed above any generic bot rule. Rule order varies by vendor, so test after every change.

Step 6: Scan your server logs

Logs show what actually happened. Search your access logs for each name separately:

Also read: related topic: how to audit brand presence on reddit: a worked example

grep -i 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c
grep -i 'GPTBot' access.log | awk '{print $9}' | sort | uniq -c

The status code column tells the story. Under policy B, OAI-SearchBot should show mostly 200s. GPTBot should either vanish or show 403s or similar blocks at the server layer. A well-behaved crawler that reads a disallow typically stops requesting pages, though it still fetches robots.txt itself, so don't worry about robots.txt hits.

Two cautions. User-agent strings can be faked, so confirm that hits come from the IP ranges in OpenAI's documentation before drawing conclusions. And if you allow both bots but see only one, that may reflect the shared-crawl behavior the Ahrefs guide describes, not a block. Look at a few weeks of logs, not a day.

Decision table

| Goal | Setting | Limits | |---|---|---| | Brand knowledge in future models | Allow GPTBot | Slow and indirect. Content shows up only in later models, and you can't see what was learned. | | Live search visibility | Allow OAI-SearchBot | Guarantees access, not citations. OpenAI says placement isn't guaranteed. | | Content protection from training | Block GPTBot | Voluntary compliance. Doesn't remove what past crawls collected or stop other AI companies. | | Protection plus search presence | Block GPTBot, allow OAI-SearchBot | Future models know less about you. Search access still isn't a promise of being cited. | | No OpenAI crawling | Block both | You give up OAI-SearchBot access. Other routes to your brand can still exist. |

What this setup cannot do

Robots.txt is voluntary. Reputable crawlers honor it, but it isn't access control. If content must stay private, put it behind authentication.

Allowing OAI-SearchBot guarantees nothing about citations. The Ahrefs guide says OpenAI typically rewrites a prompt into targeted searches and may send more specific ones to other providers. It also cites an academic study in which Google's results and OpenAI's own search API combined matched only about 27% of the pages ChatGPT cited.

So your pages can be cited through results that never involved OAI-SearchBot, and a page OAI-SearchBot can fetch may never be cited. The guide also reports that only about a quarter of the pages cited for a prompt were cited again the next day, so expect volatility.

One more signal from that guide: about 6.8% of fan-out queries used the site: operator, and three in four of those went to brand websites. That suggests ChatGPT sometimes checks brand sites directly, which is a reason to keep OAI-SearchBot out of your firewall's way. That's an inference, not a promise.

Blocking GPTBot is a tradeoff you make once and should review often. Revisit it when your licensing position, your brand goals, or OpenAI's documented crawlers change.

What to expect next

After you deploy, give it a few weeks of log review before judging the result. Crawl timing varies, and you can't force a visit. Re-run the six steps whenever you change CDN settings, migrate hosts, or edit robots.txt. Check OpenAI's documentation regularly for renamed or added crawlers, because a rule written for an old user-agent name protects nothing.

Tags

OpenaiArtificial IntelligenceWeb DevelopmentAi DevelopmentCoding Best PracticesSEO Strategy

Keep reading

WebAssembly: Unleashing Native Speed in Web Browsers
Coding•4 min read

WebAssembly: Unleashing Native Speed in Web Browsers

WebAssembly is transforming web development with near-native performance, enabling more complex and efficient applications.

Sep 6, 2025

OpenAI and Nvidia Partner to Deploy 10GW of AI Systems
Technology•3 min read

OpenAI and Nvidia Partner to Deploy 10GW of AI Systems

OpenAI and Nvidia's partnership to deploy 10GW of systems marks a milestone in AI technology, promising enhanced capabilities across industries.

Sep 22, 2025

OpenAI Enhances Voice AI for Enterprise with GPT-Realtime
Business•3 min read

OpenAI Enhances Voice AI for Enterprise with GPT-Realtime

OpenAI's gpt-realtime sets a new voice AI standard with naturalistic voices and advanced capabilities, aiming to transform enterprise applications.

Sep 10, 2025

More stories for your next project

Get tech, coding, and music production updates in your inbox.

Unsubscribe anytime.