GTMRun

robots.txt

Last updated: 2026-11-16

Google recommendation

What it is #

robots.txt is a plain-text file at https://yourdomain.com/robots.txt that tells crawlers which paths they may fetch. It's also where you advertise your sitemap.

Why it matters #

Google doesn't require one: with no file, everything can be crawled. But a real file lets you keep crawlers out of private areas (your app dashboard, admin pages), and its Sitemap: line is one of the ways Google finds your sitemap. Google doesn't guess /sitemap.xml.

How we check it #

We fetch /robots.txt and pass this check only if it returns 200 with plain text, not an HTML page.

Single-page apps (React, Vue, Next.js exports) behind a CDN often return the app's index.html with a 200 for every unknown path. That looks like a robots.txt to a naive checker but gives crawlers no rules at all. We reject HTML bodies for exactly this reason.

How to fix #

  1. Create a plain-text file:
    User-agent: *
    Allow: /
    Disallow: /dashboard
    
    Sitemap: https://yourdomain.com/sitemap.xml
  1. Serve it from the site root. build output as-is. the file for you.
    • Vite / Create React App: put it in public/robots.txt; it's copied to the
    • Next.js: public/robots.txt, or app/robots.ts.
    • Webflow / WordPress / Squarespace: use the SEO settings panel, which generates
  2. Deploy, then check it (below).

Verify it yourself #

curl -s -i https://yourdomain.com/robots.txt | head -5

You want 200 and content-type: text/plain, followed by your rules, not <!doctype html>.

Runs GTMRun's own checks only — seconds, no third-party calls.

Tips #

  • Don't block what Google needs to render. Blocking /assets/ or your JS/CSS bundles stops Google from rendering your pages.
  • robots.txt isn't privacy. Blocked URLs can still be indexed if other sites link to them. Use authentication or noindex for anything sensitive.
  • Blocking AI crawlers is a separate decision. Google-Extended, GPTBot and ClaudeBot only affect those products. Blocking Googlebot removes you from Google Search and its AI features.
  • What we learned fixing gtmrun.co: until 2026-09-29, /robots.txt on gtmrun.co returned our React app's HTML. Our own scanner scored it as present, which is the bug that made us add the HTML check. We fixed it with a two-line public/robots.txt.

Sources #