# jobshifu.com — the PUBLIC marketing site. (The app at app.jobshifu.com is separately noindex:
# a <meta name="robots"> tag with crawling ALLOWED so the tag can be read, never Disallow: /.
# See apps/web/public/robots.txt.) This site SHOULD be indexed.
#
# AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, ...) are DELIBERATELY
# allowed by the wildcard below. Answer engines are a retrieval surface we want to be in, and
# llms.txt at the site root exists for exactly them (generated by scripts/build-blog.mjs).
#
# ⛔ There is deliberately NO `Disallow: /admin` for the CMS. A Disallow stops the crawler
# FETCHING the page, so it never reads the noindex - which is how a disallowed url still gets
# listed url-only. /admin is kept out of the index by `X-Robots-Tag: noindex` in _headers plus
# a <meta name="robots"> tag in admin/index.html, both of which require the fetch to work.
#
# `Content-Signal` (2026-08-17) states the ask we actually have, which is NOT the same as the
# access the wildcard grants: index us, quote us, cite us — `use=reference` — but the corpus
# behind the research pages is not training material. It is advisory, unenforceable, and costs
# nothing; it exists so the intent is on the record rather than inferred from a bare Allow.
User-agent: *
Content-Signal: search=yes,ai-input=yes,use=reference
Allow: /

# ⛔ A JOB URL CARRYING A QUERY IS NOT A PAGE (2026-09-11, with the hub pages).
# `/jobs/<uuid>` is a posting and `/jobs/<slug>` is a hub; both are finite sets the Worker
# enumerates. A query string on either is a filter permutation, i.e. an unbounded crawl space
# that yields near-duplicates of pages already in the sitemaps. The Worker already refuses to
# render one (it passes through to the marketing site), and this keeps the crawl budget off them
# in the first place.
Disallow: /jobs/*?*

# ── AI-crawler tiering — WRITTEN, DELIBERATELY NOT ACTIVE ────────────────────────────────────
# ⛔ Do not uncomment this block yet. Today jobshifu.com is entirely marketing pages plus
# aggregate research, and every one of them is a page we WANT an answer engine to read and cite.
# Restricting AI crawlers now would cost retrieval reach and withhold nothing — a pure loss.
#
# ACTIVATION CONDITION, and the only one: the day we publish ROW-LEVEL pages (per-posting or
# per-company detail — Phase 4 of docs/plans/public-research-program.md). At that point the two
# audiences genuinely diverge: search engines should index the long tail, while AI crawlers
# should see the aggregates and not the inventory, because the inventory is the scrapeable
# asset and an aggregate is not. Competitor precedent for exactly this split is recorded in
# docs/archive/corpus-public-surface-analysis.md §2.
#
# When that day comes, uncomment and keep the Allow list in sync with what /research publishes:
#
# User-agent: Amazonbot
# User-agent: Applebot-Extended
# User-agent: Bytespider
# User-agent: CCBot
# User-agent: ClaudeBot
# User-agent: FacebookBot
# User-agent: Google-Extended
# User-agent: GPTBot
# User-agent: PerplexityBot
# User-agent: meta-externalagent
# Allow: /$
# Allow: /research
# Allow: /blog
# Allow: /llms.txt
# Disallow: /
#
# ⚠️ Two traps if you do activate it. (1) A named User-agent group REPLACES the wildcard group
# for that crawler — it does not layer on top — so anything still wanted must be re-Allowed
# inside the block, including llms.txt. (2) `Allow: /research` prefix-matches /research-foo
# too; use `/research/` plus `/research$` if that ever matters.

Sitemap: https://jobshifu.com/sitemap.xml
# The crawl pilot's job pages (jobshifu.com/jobs/<id>, one country, served by the
# job-page-render Worker; docs/architecture/public-job-page.md §5). Separate file: it is dynamic and
# not part of the static sitemap the deploy gate audits.
Sitemap: https://jobshifu.com/sitemap-jobs.xml
# The HUB pages (jobshifu.com/jobs/<slug>), the durable half of that surface: a posting expires
# and a hub does not. Served by the same Worker, and it publishes only hubs that actually held
# inventory when the file was built, so a registry row that never fills is never announced.
Sitemap: https://jobshifu.com/sitemap-hubs.xml