advertools: the Python toolkit that replaces a drawer of SEO scripts
An open-source Python package for crawling, sitemap and robots.txt analysis, SERP tooling and log-file parsing, all under one pip install.
The SEO Bench
Cloudflare AI Crawl Control: see which AI bots read your site, and gate them
A Cloudflare feature, free on every plan, that shows exactly which AI crawlers hit your site and lets you allow or block them per bot with one click.
"Is ChatGPT reading us? Is Perplexity?" is now a question editors ask as often as any classic ranking query, and answering it used to mean pulling raw server logs and grepping user agents by hand. If your site sits behind Cloudflare, you already have the answer sitting in the dashboard: AI Crawl Control.
Why it made the cut
It is the most accessible route into GEO-era crawler visibility we have found, because there is nothing to install and nothing to configure to start seeing data. The dashboard breaks down exactly which AI crawlers, GPTBot, ClaudeBot, PerplexityBot, and more, have hit your site and what they fetched, then lets you flip any one of them on or off without touching a robots.txt file or a server config.
How to use it
- In the Cloudflare dashboard, open your site and find AI Crawl Control.
- Review the crawler activity: which bots, how often, which paths.
- Set per-bot rules across the Search, Agent and Training categories, allow the ones that help your visibility, block the ones that only train a model on your work.
- Revisit periodically, new AI crawlers appear regularly and your allow list will need updating.
Worth knowing
AI Crawl Control is available on every Cloudflare plan, including Free, so this is not a paid add-on. The separate pay-per-crawl monetisation option is still in private beta and is a different feature. One date worth flagging: from 15 September 2026, Cloudflare is changing its default AI-crawler blocking behaviour for newly onboarded domains, so a Cloudflare site set up after that date may start more restrictive than an existing one. Check your own defaults rather than assuming.
More Like This
ai.robots.txt: drop-in files to block the AI crawlers you do not want
A community-maintained, continuously updated list of AI crawlers, shipped as drop-in robots.txt, htaccess, Nginx and Caddy files so you can block or allow AI bots in minutes.
Search Console Bulk Export to BigQuery: unsampled search data, past the row cap
Google Search Console's own feature that schedules a free daily export of full, unsampled search performance data into a BigQuery dataset you own, past the thousand-row UI cap.