advertools: the Python toolkit that replaces a drawer of SEO scripts
An open-source Python package for crawling, sitemap and robots.txt analysis, SERP tooling and log-file parsing, all under one pip install.
The SEO Bench
A ready-to-adapt llms.txt for a content or brand site, with a name line, a short summary, and grouped link sections, plus honest guidance on what to include and what to leave out.
llms.txt is a proposed convention: a single Markdown file at the root of your domain that points a language model at the pages you most want it to read and understand. The idea is that a model answering a question about your site, or a tool building a picture of it, can fetch one curated map instead of guessing its way around your navigation. It is not robots.txt, which controls crawling; llms.txt is an invitation, a short, human-written index of your best and most representative pages.
The format is deliberately plain: a top-level heading with your name, one paragraph explaining what you are, then Markdown link lists grouped under section headings, each link followed by a short description of what it is. Below is a starter you can adapt. Replace the bracketed parts, keep it short, and link only to pages you would be happy to have quoted.
# The Example Dispatch
> An independent publication covering [your subject] for [your audience].
> We publish [what you publish: guides, analysis, tools] and are known for
> [the one thing you want a model to remember about you].
## Start here
- [About us](https://example.com/about): who we are, who writes for us, and how we work.
- [Editorial guidelines](https://example.com/guidelines): our standards, sourcing, and corrections policy.
- [Contact](https://example.com/contact): how to reach the newsroom.
## Key guides
- [The complete guide to X](https://example.com/guides/x): our definitive, regularly updated explainer on X.
- [How to do Y](https://example.com/guides/y): a step-by-step walkthrough for beginners.
## Reference
- [Glossary](https://example.com/glossary): plain-language definitions of the terms we use.
- [Frequently asked questions](https://example.com/faq): the questions readers ask most.
## Optional
- [Full article archive](https://example.com/archive): every piece we have published.
- [Newsletter](https://example.com/newsletter): what we send and how often.
llms.txt.https://yourdomain.com/llms.txt.Include the pages you would hand a knowledgeable stranger who asked "where do I start with this site": your best evergreen guides, your about and standards pages, a glossary or FAQ if you have one. Write each description in plain language, as a sentence, not a keyword list. The whole file should be readable in under a minute; if it is longer than a page, you are indexing too much.
Leave out anything you would not want quoted or summarised: gated content, thin or dated pages, campaign landing pages, and anything behind a login. Do not stuff it with every URL you have; a curated map of thirty good pages is more useful than a dump of three thousand. And do not treat it as a ranking lever, it is a description of your site, not a pitch.
This is a young convention, not a settled standard, and support for it is uneven. Some AI tools and crawlers look for llms.txt; many do not, and none are obliged to. Publishing one costs you a few minutes and cannot hurt, but it is not a guarantee that any model will read it, quote you, or send you traffic. Treat it as a low-cost, forward-looking courtesy: a tidy front door for the tools that do check, not a mechanism you should expect measurable results from yet. Keep it accurate and current, and revisit it when support matures.
More Like This
An open-source Python package for crawling, sitemap and robots.txt analysis, SERP tooling and log-file parsing, all under one pip install.
A community-maintained, continuously updated list of AI crawlers, shipped as drop-in robots.txt, htaccess, Nginx and Caddy files so you can block or allow AI bots in minutes.
A Cloudflare feature, free on every plan, that shows exactly which AI crawlers hit your site and lets you allow or block them per bot with one click.