Generate a robots.txt file visually. Add allow/disallow rules per user-agent, block AI crawlers (GPTBot, CCBot, Claude-Web), set your sitemap URL, and download the ready-to-use file.
Free, no sign-up — one of the SaaSQuickStart Content & SEO tools.
The robots.txt file is a plain-text file placed at the root of your domain ( https://yourdomain.com/robots.txt ) that instructs web crawlers which paths they are allowed or disallowed to access. This tool groups rules by user-agent and renders the correct User-agent:, Allow:, and Disallow: directives in the format specified by the Robots Exclusion Standard (RFC 9309). The Sitemap directive points crawlers to your XML sitemap.
For well-known AI crawlers that respect robots.txt (GPTBot, Anthropic's Claude-Web, CCBot, Google-Extended), yes. However, robots.txt is a voluntary protocol — malicious scrapers and bots that don't respect it will ignore it. For truly sensitive content, use authentication rather than relying on robots.txt.
Disallow tells a crawler not to access a path. Allow overrides a broader Disallow rule for a specific sub-path. For example, "Disallow: /" combined with "Allow: /public/" blocks everything except /public/. More specific rules take precedence over broader ones.
Place the file at the root of your domain: https://yourdomain.com/robots.txt. In most web frameworks, this means placing it in the public/ or static/ directory. In Next.js 13+ you can use the built-in robots.ts config. Download the file from this tool and upload it there.
Blocking Googlebot from crawling a page with Disallow prevents it from being indexed and ranking. However, Google may still show the URL in search results (without a description snippet) if other pages link to it. To prevent indexing entirely, use a noindex meta tag or X-Robots-Tag response header on the page itself.