Free SEO tool
Build a robots.txt without guessing
Choose which crawlers get which parts of your site, see the exact file as you build it, and download it. Everything happens in this tab, and we never fetch your website.
- Runs in your browser
- Your site is never crawled or sent anywhere
- Presets you can edit afterwards
- Download a ready-to-upload file
1. Start from a preset
Or skip this and build the file by hand below. A preset replaces what is in the form, and everything stays editable afterwards.
2. Say who, and what
Each group names one or more crawlers and the paths they may or may not fetch. A crawler reads the single group that names it best, so one group per crawler.
3. Point at your sitemap
Optional, and the one line here that helps crawlers find more rather than less. It has to be a full URL.
Optional. Written at the top of the file as a # comment, for whoever reads it next.
Your robots.txt
Updates as you type. Nothing leaves your browser.
User-agent: *
Disallow:
Upload it to the root of your site, so it answers at https://yourdomain.com/robots.txt. It is a request crawlers choose to honour, not a lock: anything that must stay private needs a password.
What to do with the file
Generating it is the quick part. These three things are what make it work.
- 1
Upload it to your root
Save it as robots.txt and put it at the root of your site, so it answers at yourdomain.com/robots.txt. A crawler looks nowhere else, and a file in a subfolder does nothing.
- 2
Check what Google sees
Test it before you trust it. Google Search Console has a robots.txt report that shows what Googlebot actually reads, which is the only opinion that counts.
- 3
Revisit it after a rebuild
Change it when your site changes. A Disallow left over from a rebuild is the classic way to make a section of a live site disappear from search.
Common questions
- What is robots.txt?
- It is a plain text file at the root of your site that asks crawlers which paths to leave alone. Well-behaved crawlers read it and honour it. Nothing enforces it.
- Does it keep private pages secure?
- No, and this is the mistake that costs people the most. It is a public file that anyone can read, so it advertises the paths you named. Crawlers that ignore it are unaffected. Anything that genuinely must stay private needs a password or an access control rule on the server.
- Does Disallow remove a page from Google?
- No. Disallow asks a crawler not to fetch a page. A page it never fetches can still be listed, using only the links pointing at it, because the crawler cannot read the page to find out otherwise. To keep something out of the results, let it be crawled and add a noindex tag, or require a login.
- Does Crawl-delay work?
- Google ignores Crawl-delay entirely. Bing and Yandex read it. Google removed the crawl-rate setting from Search Console in January 2024, so the way to slow Googlebot down now is to return 429 or 503 on the requests you cannot serve.
- Do you crawl the site I am writing this for?
- No. The tool builds the file from what you type and nothing else. It never fetches your site, and the URLs you enter stay in your browser. Close the tab and they are gone.