What your robots.txt allows to be indexed
The robots.txt file tells search crawlers where they may go. A single line can shut the whole site out of Google while the server keeps answering normally, so ordinary monitoring notices nothing. We do not leave you with a printout of the rules: we say what each one does, what stays out of the results, and whether the sitemap it names really exists.
What exactly we check
- Whether the file shuts the whole site out of search engines (User-agent: *, Disallow: /)
- Whether any search engine is barred by name despite an open general rule — Googlebot, Bingbot and others
- Which paths stay out of reach of crawlers, that is, what will not be in the results
- Which AI model crawlers the file bars: GPTBot, ClaudeBot, CCBot, Google-Extended and others
- Whether the file points to a sitemap — and whether a sitemap, not an error page, really sits at that address
- Whether a sitemap exists even though nobody named it in the file (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml)
- Whether the file exists at all at /robots.txt
We read the robots.txt file and fetch the sitemap it names — a "Sitemap:" line proves nothing on its own, because it looks the same when it points to an address left over from a previous system. noindex tags on individual pages are checked by the content review in the panel, because that needs a visit to each address in turn.
Want to build or fix the file? robots.txt generator ›
How to read the result
The whole site is closed
The line "User-agent: * / Disallow: /" tells every search engine that it may not fetch a single page. The site works for visitors, but over time it drops out of Google. Most often this is a file from a staging environment that shipped to production with a release.
AI crawlers blocked
Separate lines can refuse entry to AI model crawlers such as GPTBot or ClaudeBot while the site stays open to ordinary search engines. This is sometimes a deliberate choice: not everyone wants their content feeding the models. We show the state; the decision is yours.
Barred by name
A "User-agent: Googlebot" group replaces the general rule entirely rather than adding to it. So a file can let everyone in with "*" and shut out Googlebot alone in a separate group. The general rule then looks innocent, and the site is not in Google.
A sitemap that is not there
A "Sitemap:" line hands the search engine the full list of addresses, but it is only a declaration: the address may be left over from a previous system and return 404, may be relative (this line needs a full address), may be closed off by a Disallow rule of its own. So we fetch the address and say what sits there. A missing line is not a fault — a sitemap can be submitted in webmaster tools — but the line in robots.txt is the simplest route.
No file
When the file is missing, crawlers index everything by default. That is not a fault, though the file is still useful, if only to point to the sitemap or to close off the back end.
Most common causes
A staging file on production
Staging environments keep "Disallow: /" so they never reach Google. A release that copies files wholesale carries that line onto the live site. It is the most common cause of a sudden drop from the results.
A "do not index" box ticked in the CMS
In systems like WordPress a single "discourage search engines from indexing" box flips the whole site to invisible. It gets left on after a migration or after maintenance, and nobody looks at it.
Someone else's robots.txt copied over
A file pasted from another site blocks directories you do not have, or bots you want to let in. It is worth reading line by line rather than trusting that it fits.
Questions and answers
Is robots.txt the same as noindex?
No. robots.txt asks a crawler not to enter an address. A noindex tag lets it in but forbids showing the page in the results. An address blocked in robots.txt can even be shown as a bare link, because the search engine never entered and never saw the noindex.
I blocked AI crawlers. Will that hurt my Google ranking?
No. Google-Extended and GPTBot are crawlers separate from Googlebot, which indexes the site for search. Blocking the former does not touch your visibility in the results.
Does robots.txt hide private pages?
No. It is a request, not a lock, and the file itself is public. An address listed under Disallow is often looked for precisely there. Closing off content is the job of a login and permissions, not robots.txt.
That was one area. Site condition has four.
This check is a slice of one of the four areas of site condition. The free review in the wizard combines them into a single 0–100 score and shows what takes points away and where to start.
- A provisional 0–100 score right after the review, with a reason next to every point taken away
- Then ongoing care: a check every 5 minutes on the free plan, an alert once a follow-up check confirms the problem
- A warning before the certificate and the domain expire
- Email notifications with no limit
How we watch this day to day: Content checks ›
Other tools
Is my website down
Check from the outside whether the site responds, and whether the problem is on your end.
SSL certificate
Expiry date, issuer, name coverage and everything else that makes the padlock disappear.
Domain expiry
When the domain expires, who the registrar is and whether anyone can move it without your knowledge.
SPF, DKIM and DMARC
Why your email lands in spam, and whether someone can send invoices from your address.
Security headers
Eight items you set on the server in a quarter of an hour, and every audit asks about them.
WCAG accessibility check
Find out whether someone using a screen reader or no mouse can get through your page. A 0–100 score and the main barriers with fix instructions.