InBuild
Free tool · SEO

Robots.txt Generator

Toggle the rules you want, get a valid robots.txt with line-by-line explanations. Includes presets for blocking AI training crawlers (GPTBot, ClaudeBot, Perplexity, Google-Extended) and SEO scrapers.

Googlebot ignores Crawl-delay. Bing/Yandex respect it.

robots.txt
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /preview

Sitemap: https://example.com/sitemap.xml
Show line-by-line explanation
User-agent: *

Applies the rules below to every crawler that doesn't have its own group.

Allow: /

Allows crawling of the site root by default.

Disallow: /admin

Blocks /admin and everything beneath it.

Disallow: /api

Blocks /api and everything beneath it.

Disallow: /preview

Blocks /preview and everything beneath it.

Sitemap: https://example.com/sitemap.xml

Tells crawlers where to find the XML sitemap. Always use an absolute URL.

Skip generating this by hand next time.

Every InBuild export already ships an app/robots.ts route — no separate file to maintain or re-paste after a redesign.

See real output first
See plans

How to use this generator

  1. Toggle the rules you need. Block-all for staging, or pick disallow/allow paths, AI crawler blocking, SEO scraper blocking, a crawl delay, and sitemap URLs.
  2. Read the line-by-line explanation. Every generated line gets a plain-English annotation next to it, so you know exactly what each directive does before you ship it.
  3. Copy it to yourdomain.com/robots.txt. The generator only produces the text — deploying it (at the domain root, not a subfolder) is a manual step.

What this generator actually blocks

The AI-crawler and SEO-scraper toggles aren't generic placeholders — each one emits a named block for a specific, real user-agent string:

  • Block AI crawlers — GPTBot, ChatGPT-User, OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Google's AI training — separate from Googlebot itself), CCBot (Common Crawl, used to train many models), Bytespider (ByteDance), and Amazonbot.
  • Block SEO scrapers — AhrefsBot, SemrushBot, DotBot, MJ12bot, rogerbot, and BLEXBot.
  • Crawl-delay — only emitted if you set it above 0. Bing, Yandex, and Yahoo honor it; Googlebot ignores it outright.
  • Host — a non-standard directive Yandex respects and Google ignores. Omitted unless you fill it in.

Frequently asked questions

Where do I put the robots.txt file?

At the root of your domain: yourdomain.com/robots.txt. Subfolder paths (like /blog/robots.txt) are ignored by crawlers. In Next.js App Router, you can also generate it programmatically via app/robots.ts.

Will blocking AI crawlers stop my content from showing up in ChatGPT?

It stops future training crawls — content already crawled stays in the model. The 'AI Search' crawlers (OAI-SearchBot, ChatGPT-User) are separate from training crawlers (GPTBot). Blocking GPTBot keeps you out of future training; blocking OAI-SearchBot keeps you out of ChatGPT's live citations.

Should I block SEO scrapers like Ahrefs and Semrush?

Usually no. Blocking them stops your own SEO research on competitors and prevents your site from appearing in their datasets. The exception is if your content is high-value and you don't want competitors mining your structure.

What's the difference between Disallow in robots.txt and noindex?

robots.txt tells crawlers not to fetch the page. noindex (in a meta tag or HTTP header) tells crawlers they CAN fetch it but shouldn't show it in search results. For sensitive pages, use both. For pages you want crawled but not indexed (like paginated archives), use only noindex.

Does Googlebot respect Crawl-delay?

No. Googlebot ignores Crawl-delay entirely. Bing, Yandex, and Yahoo respect it. To slow Googlebot, use Google Search Console's crawl rate settings instead.

Does this tool deploy the file for me, or check my existing robots.txt?

Neither — it's a generator, not a validator or a deploy pipeline. It builds the text and lets you copy it; you still have to put it at yourdomain.com/robots.txt yourself. It also doesn't fetch or check a robots.txt you already have — if you want to verify what's live right now, view yourdomain.com/robots.txt directly in a browser.

What exactly gets blocked when I turn on 'Block AI crawlers'?

Ten named user-agents get a Disallow: / block: GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Google's AI training, separate from Googlebot itself), CCBot (Common Crawl, used to train many models), Bytespider (ByteDance), and Amazonbot. "Block SEO scrapers" is a separate toggle for six different bots: AhrefsBot, SemrushBot, DotBot, MJ12bot, rogerbot, and BLEXBot.

Inspect a sitemap →Run a full page audit →Check a canonical tag →robots.txt glossary →What's crawl budget? →