Robots.txt Builder SEO

Create robots.txt files with crawler directives and sitemaps.

Generate a robots.txt file to control how search engines crawl your website. Customize user-agents, allow/disallow paths, and add sitemaps.

How to use this tool

Privacy & Safety

All processing is done locally in your browser. No data is uploaded or stored.

Why choose this tool?

Complete robots.txt directives, multi-user agent support, sitemap inclusion, and instant copy/download.

Crawler Directives

User-agent

Crawl Delay (seconds)

Sitemap URL

Disallowed Paths (one per line) Example: /admin/ blocks access to admin directory

Allowed Paths (one per line) Only used if you need to allow specific paths within disallowed directories

Generated robots.txt

User-agent: Specifies which crawler the rules apply to (* means all)

Disallow: Tells crawlers which URLs to avoid crawling

Allow: Overrides disallow rules for specific paths

Crawl-delay: Instructs crawlers to wait between requests

Sitemap: Points search engines to your XML sitemap

No history yet. Generate robots.txt to see it here.

What is robots.txt?

Robots.txt is a file that tells search engine crawlers which pages to crawl or avoid on your website.

Where should I upload this file?

Place the robots.txt file in your website's root directory (e.g., https://example.com/robots.txt).

Is this tool free?

Yes – completely free, no registration required.