# archiveris.com
#
# Everything under / is public and may be crawled, indexed, summarised and
# used to answer questions about Archiveris. That includes the AI crawlers
# (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and the
# rest): being quoted accurately is worth more to us than being invisible.
# No robots.txt existed before this file, so permitting everything is the
# behaviour the site already had, now stated rather than implied.
#
# /preview/ held eight positioning experiments, each arguing a different
# case for the same company. Disallowing them was not enough — not every
# crawler honours robots.txt or <meta robots noindex>, and one of those
# pages was quoted back at us as current a week after it went dead — so
# they are no longer published at all. Every /preview/ URL now returns 404.
# The rule stays because it still does something: a crawler holding an old
# link stops asking instead of retrying a page that will never return.

User-agent: *
Allow: /
Disallow: /preview/

Sitemap: https://archiveris.com/sitemap.xml