Your share links can carry a noindex header and still show up in Google. A robots.txt Disallow stops Googlebot from fetching the page, so it never reads the header.
Google's updated spam guide now tells platforms to apply noindex or block crawling by default on pages made for sharing, according to Search Engine Journal's report on the Oct. 26 update. The phrasing sounds like an either/or choice, but the two controls are not equal. This walkthrough shows how to audit your share-link patterns, pick a control for each content type, and verify the result.
Why noindex and robots.txt are not equal choices
Robots.txt controls crawling. Noindex controls indexing. Those are different jobs.
Google's robots.txt introduction, as quoted in the report, says a disallowed page can still be indexed if other sites link to it. Google does not crawl the content, but the bare URL, and sometimes the anchor text of links pointing to it, can appear in results without a description. The same introduction says robots.txt "is not a mechanism for keeping a web page out of Google."
The second problem catches teams off guard. Google's noindex documentation says the rule works only when robots.txt does not block the page, because otherwise "the crawler will never see the noindex rule." From Google's side, a blocked URL with a perfect noindex header is a URL with no readable instructions.
That is why the Page Indexing report has a status called "Indexed, though blocked by robots.txt." Google's help for that status says to remove the block and use noindex instead.
Decision table: content type vs recommended control
The report notes that Google's "Control what you share" page lists robots.txt as applicable to images and video, and noindex as applicable to all content types. The table below builds on those two rules. The recommendations are my inference, not a Google-published matrix.
| Content type | Recommended control | Reason | |---|---|---| | HTML share page | Noindex (meta robots or X-Robots-Tag), crawlable | Googlebot must fetch the page to read the rule | | Generated output page (HTML) | Noindex by default, crawlable | Same mechanism; opt in to indexing only if the output is meant for search | | Generated file (PDF, text, JSON) | X-Robots-Tag header, crawlable | No HTML head exists for a meta tag | | Image or video served from share pages | Robots.txt block, or X-Robots-Tag noindex | Google's documentation lists robots.txt as applicable here | | App view behind a link or login | Noindex on anything publicly reachable; keep login walls too | Inference: a login does not replace an explicit rule on public responses | | Already-indexed share URLs | Staged: lift the block, serve noindex, then reconsider | Googlebot has to recrawl to see the rule |
The short version: for pages, default to noindex on a crawlable URL. Reserve robots.txt for media, or for crawl-load reasons where an indexed bare URL is acceptable.
Step-by-step audit checklist
- Inventory your share URL patterns. List every route that creates a public link: share tokens, published outputs, embeds, export URLs and preview pages. Group them by pattern, such as
/s/{token}or/o/{id}, not by individual URL. - Read robots.txt for each pattern. Fetch the live file and look for matching
Disallowlines. Check the robots.txt report in Search Console too, since a CDN or subdomain may serve a different file than you expect. - Check the response headers. Request a real share URL and look for
X-Robots-Tag. Note the value exactly. Google treats "none" as equal to noindex plus nofollow, per the report. - Check the HTML for a meta robots tag. Look in the
<head>of the rendered page, not only the template. Client-side frameworks can inject or drop tags after load. - Compare the signals. A pattern with both a
Disallowand a noindex has an invisible noindex. A pattern with neither is open to indexing. - Decide a fix per pattern using the table above.
- Deploy, then verify with URL Inspection and the Page Indexing report.
Commands for steps 2 to 4
Run these against your own domain:
curl -s https://example.com/robots.txt
curl -sI https://example.com/s/SAMPLE_TOKEN
curl -s https://example.com/s/SAMPLE_TOKEN | grep -i 'name="robots"'
Curl shows what your server returns to curl. If you vary responses by user agent or IP, Googlebot may get something different, so confirm with the URL Inspection live test. See more on google indexing api alternative for job postings: 5 steps for additional background.
Hypothetical worked audit
This example is illustrative. The domain, paths and findings are invented, and no real outcomes are implied.
Imagine a collaboration tool at example.com with share pages at /s/{token}, generated summaries at /o/{id}, and preview images at /img/.
Check robots.txt. The file contains Disallow: /s/ and nothing for /o/. Share pages are blocked from crawling, and generated outputs are open.
Check headers and meta. A curl request to a share page returns X-Robots-Tag: none. A request to an output page returns no robots header, and the HTML has no meta robots tag.
Interpret. The share pages carry a noindex-equivalent that Googlebot cannot read because of the Disallow. If other sites link to any of those URLs, they can still surface as bare URLs. The output pages have no control at all.
Decide. Remove Disallow: /s/. Keep the none header, or switch to noindex if you want links followed. Add X-Robots-Tag: noindex to /o/ responses. Leave /img/ blocked in robots.txt, since Google's documentation names robots.txt for images.
Verify. Re-run the three curl commands. Then run URL Inspection on a few share URLs and confirm the live test reports that indexing is not allowed because of noindex, and that robots.txt does not block crawling.
Search Engine Journal's author described a real-world case of this kind. On July 27, the author confirmed that Claude's share path was blocked by a Disallow in robots.txt, while the share pages also returned none in the X-Robots-Tag header. The author could not determine what Google saw when it first found those links. The lesson: while a block is in place, the header alone proves nothing.
Clean up URLs that are already indexed
Order matters here. Adding noindex to a blocked URL does nothing, so the block comes off first.
Also read: go deeper on how to label free articles on a paywalled site: test plan
Start with the Page Indexing report and filter for "Indexed, though blocked by robots.txt." Export the sample URLs to find the patterns. Remove the matching Disallow, deploy noindex on those responses, and request a recrawl of a few examples through URL Inspection.
Then wait. Google has to recrawl each URL before it drops, and Google's documentation gives no timeline, so this guide promises none.
Once the URLs have dropped, you can decide whether to restore a robots.txt block for crawl-load reasons. That is a judgment call. The risk is that new links to unseen tokens produce fresh bare-URL listings.
Monitor so it stays fixed
The spam guide also asks platforms to verify the site or subdomain in Search Console. It recommends reviewing the Page Indexing and Search performance reports for unexpected spikes in indexed pages, impressions or queries. A new bullet suggests tracking creation and indexing at the platform level, with alerts for spikes in newly published public pages or "a small number of users generating a large share of indexable URLs."
Wire that into your own data. A weekly count of newly indexable URLs by pattern can catch a template regression before Search Console does.
Limitations
Noindex can be delivered as a meta tag or an X-Robots-Tag header. The header works for non-HTML files, while the meta tag works only in HTML. Pick one source of truth, because conflicting signals are hard to debug.
Google's documentation does not say how quickly Google will reprocess a given site, and nothing here confirms whether any specific platform's share URLs are indexed. It also does not spell out how Google handles every edge case. Test on a small pattern first.
What to expect next
After the block lifts and noindex goes live, expect the "Indexed, though blocked by robots.txt" count to fall gradually as Google recrawls, not overnight. Watch the report weekly and spot-check with URL Inspection. Keep the curl checks in your deploy pipeline so a new share route cannot ship without a readable rule.
