# nearbyuk. The site is meant to be crawled, by search engines and by AI.
#
# A page a crawler is forbidden to fetch is a page whose noindex it can never
# read, and Google will still index a disallowed URL it finds linked elsewhere
# and show it with no title and no snippet, with no way back out. So index
# control happens where a crawler can see it: a robots meta tag on the page.
# Nothing here is hidden from crawlers.
#
# ---------------------------------------------------------------------------
# READ THIS BEFORE EDITING
#
# Cloudflare prepends its own managed block ABOVE this file, and that block is
# what names and refuses the AI TRAINING crawlers: GPTBot, ClaudeBot, CCBot,
# Google-Extended, Applebot-Extended, Amazonbot, Bytespider, meta-externalagent.
# Confirmed on the sister site on this same account.
#
# Never add an Allow group here for an agent Cloudflare already names. Two
# groups for one agent is a file nobody can reason about. The sections below
# name only agents Cloudflare does NOT touch. Everything Cloudflare does not
# name and this file does not name falls to the wildcard at the foot, which
# allows it, so the file is correct whether or not that managed block is on.
#
# Whether to allow AI TRAINING crawlers as well is a business decision, not a
# technical one, and rule 03 flags it as the owner's. It is currently
# Cloudflare's default rather than a choice. See seo/overlay.md.
# ---------------------------------------------------------------------------

# --- Traditional search -----------------------------------------------------
# The crawlers that build a search index. Googlebot is also what feeds Google's
# AI Overviews and AI Mode, which is a separate thing from Google-Extended.

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

# --- AI search and live lookup: THE CITATION DRIVERS -------------------------
# These fetch a page at the moment somebody asks a question and then answer
# with a link back. They are the reason this site can be recommended by an
# assistant at all, and they are a different thing entirely from the training
# crawlers. Allowed deliberately and by name.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

# --- Discovery --------------------------------------------------------------
# Crawlers that build datasets and answer engines feed from. Diffbot and YouBot
# are not in Cloudflare's managed list, so they are named here.

User-agent: Diffbot
Allow: /

User-agent: YouBot
Allow: /

# --- Everything else --------------------------------------------------------
# Nothing here is admin, paginated into infinity, or crawl waste, so there is
# nothing to disallow. /search/ is a client-side filter page held out of the
# index by its robots meta tag, not by hiding it from crawlers: a disallowed
# page is a page whose noindex is never read.
#
# Content-Signal states what this site permits, per rule 03 [HIGH]. search=yes
# and ai-input=yes because being found and being quoted in an answer is the
# entire point of publishing this.
#
# ai-train=no mirrors what is actually happening rather than a preference: the
# Cloudflare managed block above already refuses the training crawlers, so
# declaring yes here would be a permission the site does not in fact grant. If
# the owner decides to allow training, both change together, and this is the
# one word that changes here.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://nearbyuk.com/sitemap-index.xml