What your robots.txt allows to be indexed
The robots.txt file tells search crawlers where they may go. A single line can shut the whole site out of Google while the server keeps answering normally, so ordinary monitoring notices nothing. We check whether the file is cutting you out of the results and which AI crawlers it lets in or keeps out.
What exactly we check
- Whether the file shuts the whole site out of search engines (User-agent: *, Disallow: /)
- Which AI model crawlers the file bars: GPTBot, ClaudeBot, CCBot, Google-Extended and others
- Whether the file points to a sitemap (a Sitemap: line)
- Whether the file exists at all at /robots.txt
We read the robots.txt file itself. noindex tags on individual pages are checked by the content review in the panel, because that needs a visit to each address in turn.
Want to build or fix the file? robots.txt generator ›
How to read the result
The whole site is closed
The line "User-agent: * / Disallow: /" tells every search engine that it may not fetch a single page. The site works for visitors, but over time it drops out of Google. Most often this is a file from a staging environment that shipped to production with a release.
AI crawlers blocked
Separate lines can refuse entry to AI model crawlers such as GPTBot or ClaudeBot while the site stays open to ordinary search engines. This is sometimes a deliberate choice: not everyone wants their content feeding the models. We show the state; the decision is yours.
No sitemap
A "Sitemap:" line hands the search engine the full list of addresses. Its absence is not a fault, and a sitemap can also be submitted directly in webmaster tools, but the line in robots.txt is the simplest route.
No file
When the file is missing, crawlers index everything by default. That is not a fault, though the file is still useful, if only to point to the sitemap or to close off the back end.
Most common causes
A staging file on production
Staging environments keep "Disallow: /" so they never reach Google. A release that copies files wholesale carries that line onto the live site. It is the most common cause of a sudden drop from the results.
A "do not index" box ticked in the CMS
In systems like WordPress a single "discourage search engines from indexing" box flips the whole site to invisible. It gets left on after a migration or after maintenance, and nobody looks at it.
Someone else's robots.txt copied over
A file pasted from another site blocks directories you do not have, or bots you want to let in. It is worth reading line by line rather than trusting that it fits.
Questions and answers
Is robots.txt the same as noindex?
No. robots.txt asks a crawler not to enter an address. A noindex tag lets it in but forbids showing the page in the results. An address blocked in robots.txt can even be shown as a bare link, because the search engine never entered and never saw the noindex.
I blocked AI crawlers. Will that hurt my Google ranking?
No. Google-Extended and GPTBot are crawlers separate from Googlebot, which indexes the site for search. Blocking the former does not touch your visibility in the results.
Does robots.txt hide private pages?
No. It is a request, not a lock, and the file itself is public. An address listed under Disallow is often looked for precisely there. Closing off content is the job of a login and permissions, not robots.txt.
You checked once. We check every five minutes.
The free plan watches your home page, the SSL certificate and the domain expiry date. No card, no expiry date on the plan itself.
- A pulse check every 5 minutes, with a second check confirming an outage
- A warning before the certificate and the domain expire
- Email notifications with no limit
- Seven days of statistics, one account
How we watch this day to day: Content checks ›
Other tools
Is my website down
Check from the outside whether the site responds, and whether the problem is on your end.
SSL certificate
Expiry date, issuer, name coverage and everything else that makes the padlock disappear.
Domain expiry
When the domain expires, who the registrar is and whether anyone can move it without your knowledge.
SPF, DKIM and DMARC
Why your email lands in spam, and whether someone can send invoices from your address.
Security headers
Eight items you set on the server in a quarter of an hour, and every audit asks about them.