robots.txt
Last updated: 2026-11-16
Google recommendation
What it is #
robots.txt is a plain-text file at https://yourdomain.com/robots.txt that tells crawlers which paths they may fetch. It's also where you advertise your sitemap.
Why it matters #
Google doesn't require one: with no file, everything can be crawled. But a real file lets you keep crawlers out of private areas (your app dashboard, admin pages), and its Sitemap: line is one of the ways Google finds your sitemap. Google doesn't guess /sitemap.xml.
How we check it #
We fetch /robots.txt and pass this check only if it returns 200 with plain text, not an HTML page.
Single-page apps (React, Vue, Next.js exports) behind a CDN often return the app's index.html with a 200 for every unknown path. That looks like a robots.txt to a naive checker but gives crawlers no rules at all. We reject HTML bodies for exactly this reason.
How to fix #
- Create a plain-text file:
User-agent: * Allow: / Disallow: /dashboard Sitemap: https://yourdomain.com/sitemap.xml
- Serve it from the site root. build output as-is. the file for you.
- Vite / Create React App: put it in
public/robots.txt; it's copied to the - Next.js:
public/robots.txt, orapp/robots.ts. - Webflow / WordPress / Squarespace: use the SEO settings panel, which generates
- Vite / Create React App: put it in
- Deploy, then check it (below).
Verify it yourself #
curl -s -i https://yourdomain.com/robots.txt | head -5You want 200 and content-type: text/plain, followed by your rules, not <!doctype html>.
Tips #
- Don't block what Google needs to render. Blocking
/assets/or your JS/CSS bundles stops Google from rendering your pages. - robots.txt isn't privacy. Blocked URLs can still be indexed if other sites link to them. Use authentication or
noindexfor anything sensitive. - Blocking AI crawlers is a separate decision.
Google-Extended,GPTBotandClaudeBotonly affect those products. BlockingGooglebotremoves you from Google Search and its AI features. - What we learned fixing gtmrun.co: until 2026-09-29,
/robots.txton gtmrun.co returned our React app's HTML. Our own scanner scored it as present, which is the bug that made us add the HTML check. We fixed it with a two-linepublic/robots.txt.
Sources #
- Google: Introduction to robots.txt
- Google: Create and submit a robots.txt file