SEO Tools

Robots.txt Generator

Build a clean robots.txt file from simple options — allow all, block all or custom rules, with user-agent, disallow/allow paths and a sitemap. Generated in your browser; nothing is uploaded.

Runs entirely in your browser — nothing you type is uploaded.

This tool runs in your browser. Please enable JavaScript to use it — the guide below explains what it does.

Note: the file is generated in your browser with JavaScript. With JavaScript off you can read this guide, but the tool controls will not appear.

Create a clean robots.txt file

This tool builds a valid robots.txt file entirely in your browser — no upload, no network request and no server-side writing. Choose whether to allow all, block all or set custom rules, pick a user-agent, list disallow and allow paths, add a sitemap, then copy the result or download the file.

How it works

  1. Enter your website URL (optional) and choose Allow all, Block all or Custom.
  2. Pick a user-agent — all crawlers, Googlebot, Bingbot, or a custom name.
  3. For Custom, add disallow and allow paths (one per line, each starting with /), and an optional sitemap URL.
  4. Click Generate robots.txt, then copy the text or download the file.

Benefits and features

  • Valid output — clean, plain-text directives crawlers understand.
  • Flexible rules — allow all, block all, or precise custom paths.
  • Safe input — blank lines ignored, duplicates removed, paths normalised to start with /.
  • Copy or download — grab the text or save a ready robots.txt file.
  • Private and offline — nothing is uploaded or published.

Practical tips

Place the finished file at the root of your site so it is reachable at https://example.com/robots.txt — crawlers only read it there. Remember that robots.txt controls crawling, not indexing or access: it is a guideline for well-behaved bots and should never be used to hide sensitive pages. Adding your sitemap URL helps search engines discover your pages.

Limitations

This tool only generates the text; it does not upload, publish or verify the file, and it cannot confirm that crawlers will obey it. Malicious or non-compliant bots may ignore robots.txt entirely, so it is not a security control. Always review the output and test it in your search console after uploading.

Privacy

Everything runs locally in your browser. Your input is never uploaded, stored or shared, and the tool makes no network requests.

Frequently Asked Questions

What does a robots.txt file do?
It tells search-engine crawlers which parts of your site they may or may not crawl. It is a guideline that well-behaved crawlers follow; it is not a security measure and does not hide pages from people who have the URL.
Does this tool publish the file for me?
No. It only generates the text in your browser. You then upload the generated robots.txt to the root of your website, for example https://example.com/robots.txt. The tool does not upload, publish or verify anything.
What is the difference between Allow all, Block all and Custom?
Allow all produces an empty Disallow (crawlers may crawl everything). Block all produces Disallow: / (crawlers should crawl nothing). Custom lets you list specific Disallow and Allow paths for the chosen user-agent.
How should paths and the sitemap be written?
Paths should start with a slash, e.g. /admin/. Enter one path per line; blank lines are ignored and duplicates removed. The sitemap must be an absolute URL such as https://example.com/sitemap.xml.
Is my input uploaded anywhere?
No. The file is built entirely in your browser's memory. Your input is never uploaded, stored or shared, and the tool makes no network requests.

Related SEO Tools tools

Meta Tag Generator

Create clean, ready-to-copy HTML meta tags — title, description, keywords, robots, canonical, Open Graph and Twitter cards — right in your browser.

SEO Tools Open Tool

Keyword Density Checker

Check how often a keyword appears in your text and its density percentage — with whole-word, substring and case options. Runs locally in your browser; your text is never uploaded.

SEO Tools Open Tool

Open Graph Generator

Generate Open Graph and Twitter/X meta tags for rich social sharing previews — title, description, image, type and card. Built and escaped in your browser; nothing is uploaded.

SEO Tools Open Tool
Scroll to Top