Test your robots.txt
Fetches your robots.txt, parses every group and rule, and tells you what each crawler is allowed to see. Catches the classic mistakes: a stray Disallow: / left over from staging, a sitemap line that points at the wrong host, rules that block CSS and JS so Google renders a blank page.
Any public page. Try .
What to do about it
The check found it. These are the ways to fix it, keep it fixed, or run the same check for every business you serve.
TagFrog Pro
Server-side tracking, consent and every ad platform API, proven with a real test order and kept current. Watch included when it launches.
See TagFrog ProWatch
Re-runs this check every week and emails you the moment the answer changes: site down, tag gone, score dropped, certificate expiring.
Join the waitlistWebsite
No website yet? $499, live in two days, or rebuild a tired one for the same price. Hosting and TagFrog on from day one, Watch when it launches, changes by chat, never a call. Have an agency? They can resell this.
Get online, $499OpsBlu
This same check, run for every local business in a territory. Leads with a reason to call, as CSV, API and MCP.
Go to opsblu.comWhat we check
- robots.txt exists and answers 200 with text
- User-agent groups and their Allow / Disallow rules
- Whether the homepage and main sections are crawlable by Googlebot
- Sitemap: lines and whether they resolve
- Blocked CSS, JS or image paths that break rendering
- AI crawler rules (GPTBot, ClaudeBot, CCBot) if present
Cost of one run: 1 request. Results for the same URL are kept for a few minutes, so running twice is free and instant.
Questions
Do I need a robots.txt at all?
No. A missing file means everything is crawlable, which is the right default for most sites. You need one when you want to keep crawlers out of search results pages, carts, admin paths, or to point them at your sitemap.
Will Disallow keep a page out of Google?
Not reliably. Disallow stops crawling, not indexing; a page linked from elsewhere can still appear with no snippet. To keep a page out of results, let it be crawled and add a noindex meta tag or header.
Why does the tester flag my blocked /wp-includes/ or /assets/ path?
Google renders pages like a browser. If the CSS and JavaScript are blocked, it sees a broken page and may rank it accordingly. Allow the static assets even when you block the rest of the directory.