ai.robots.txt: drop-in files to block the AI crawlers you do not want
A community-maintained, continuously updated list of AI crawlers, shipped as drop-in robots.txt, htaccess, Nginx and Caddy files so you can block or allow AI bots in minutes.
Filed under
robots.txt
Every Vault entry filed under robots.txt, across every shelf.
The Tuesday Briefing
One editor-curated email. The updates that matter, the noise filtered out.
Free. Occasionally includes sponsored messages from our partners. Unsubscribe anytime.