# topoquest.com robots policy
#
# Block the dynamic map renderers' infinite crawl space. Every map.php view is a
# unique lat/lon/zoom/datum/size/coord URL, and each page links to 8 neighbors +
# zoom in/out - a spider trap - with each request triggering an expensive
# MapServer/GDAL render.
#
# Only the INFINITE-param-space endpoints are blocked. Bare /map.php (the landing
# page, listed in the sitemap) stays crawlable, and the real content stays fully
# open so it keeps ranking: the /place/... clean URLs, place-detail.php, and
# map-detail.php (USGS map metadata: name, state, scale - a finite catalog, no
# param explosion; it 301s to the clean /map/... URLs).
#
# The bounded image renderers place-detail-map.php (?id=) and map-detail-preview.php
# (?usgs_cell_id=) are intentionally ALLOWED: each renders one image per place / per
# USGS map (no param explosion, not a spider trap), so letting Googlebot-Image fetch
# them lets the location and USGS-map images appear in Google Image search.
# Wikipedia/GeoHack human links are unaffected (robots.txt does not apply to real
# visitors).
#
# "Disallow: /map.php?" is a literal prefix match: it blocks /map.php?lat=... but
# not bare /map.php, and works for every crawler (no wildcard support required).

# Amazonbot and Applebot crawl the /place catalog aggressively (together ~half of all bot
# traffic, 18 Jun 2026) and fetch each page's decorative map_from_tiles.php previews, each of
# which renders source tiles - the dominant load on dev3. Throttle them harder than the
# general 10s. A named User-agent block REPLACES the `*` block for that bot, so the
# spider-trap Disallows are repeated here. (Both honor Crawl-delay.)
User-agent: Amazonbot
Disallow: /map.php?
Disallow: /map2.php?
Disallow: /map_image.php
Disallow: /map_post.php
Disallow: /map_post2.php
# Amazonbot has no human-facing web-search product (Alexa / model training only), so
# rendering our decorative previews has zero referral value - block them outright.
Disallow: /map_from_tiles.php
Crawl-delay: 30

User-agent: Applebot
Disallow: /map.php?
Disallow: /map2.php?
Disallow: /map_image.php
Disallow: /map_post.php
Disallow: /map_post2.php
# TEMPORARY (19 Jun 2026): also blocking the decorative previews while dev3's preview cache
# warms. UNLIKE Amazonbot, Applebot feeds human-facing search (Siri/Spotlight/Safari), so
# REVISIT and remove this one Disallow once the cache is warm, to let Apple image search
# index the previews. (See docs/ABUSE-MITIGATION.md.)
Disallow: /map_from_tiles.php
Crawl-delay: 30

User-agent: *
Disallow: /map.php?
Disallow: /map2.php?
Disallow: /map_image.php
Disallow: /map_post.php
Disallow: /map_post2.php
Crawl-delay: 10

# The index ties together sitemap.xml, the static .txt dumps, and all the
# 50k-URL google-sitemap-{gnis,maps}.php chunks (2.3M+ clean URLs) - making
# the full URL surface crawler-discoverable, not just GSC-submitted.
Sitemap: https://topoquest.com/sitemap-index.php