advertools: the Python toolkit that replaces a drawer of SEO scripts
An open-source Python package for crawling, sitemap and robots.txt analysis, SERP tooling and log-file parsing, all under one pip install.
The SEO Bench
The reference tutorial for bulk-checking indexing status through Google's URL Inspection API in the SEO Spider, including working within the 2,000 URLs per day quota.
The URL Inspection tool in Search Console tells you everything about one URL and nothing about a thousand. Google's API behind it lets you ask the same question at scale, but wiring up authentication and working within a 2,000-URL-per-property-per-day quota is fiddly enough that most people give up before finishing. This tutorial from Screaming Frog is the guide that gets you through it, connecting the SEO Spider to the API so a crawl reports live indexing status, canonical resolution and coverage verdicts against real Google data rather than assumptions.
When large-scale index monitoring comes up in practitioner conversation, this is the guide that actually gets linked, not a vague "check Search Console" gesture. It comes from the most trusted name in technical SEO crawling software, it is kept current with the tool's own updates, and it treats the quota limitation honestly instead of glossing over it, telling you how to plan a crawl around 2,000 checks a day rather than promising something the API cannot deliver.
The 2,000 URLs per day per property quota is Google's hard limit, not Screaming Frog's, so a large site needs a deliberate sampling or rotation plan rather than one full-site pass. This is a setup tutorial, not a monitoring product: scheduling repeat checks and tracking change over time is left to you.
More Like This
An open-source Python package for crawling, sitemap and robots.txt analysis, SERP tooling and log-file parsing, all under one pip install.
A community-maintained, continuously updated list of AI crawlers, shipped as drop-in robots.txt, htaccess, Nginx and Caddy files so you can block or allow AI bots in minutes.
A Cloudflare feature, free on every plan, that shows exactly which AI crawlers hit your site and lets you allow or block them per bot with one click.