Write a correct robots.txt in a minute.
A robots.txt file tells crawlers which parts of a site they may visit. This free generator writes one from a few choices, adds your sitemap, can ask AI crawlers to stay out, and warns you about the mistakes that hide a site from search.
Your robots.txt
User-agent: * Allow: /
Save it as robots.txt in the root of your site.
How to use it.
Choose who may crawl
Most live sites allow crawlers. Block everything only for a site that is not ready or not public.
List paths to keep out
Add pages that add nothing to search, such as a basket or internal search results, one per line.
Add your sitemap
Give its address so crawlers find every page you want found.
Copy and save the file
Save it as robots.txt in the root of your site, so it answers at yourbusiness.com/robots.txt.
What robots.txt does, and does not do.
A robots.txt file sits at the root of a website and gives crawlers their instructions before they visit. It can keep them away from areas that waste their time, and point them to the sitemap. It is one of the first files a search engine reads.
It is also easy to get wrong. A single line, Disallow: /, asks every crawler to leave the whole site, which is exactly what you want for a site under construction and exactly what you do not want after launch. The file is public and it is a request, not a lock: it does not protect private pages.
Good habits
- Check the file on the live address after you publish it, and again after every redesign
- Never rely on robots.txt to hide private pages. Use a login for anything that must stay private
- Remember that a blocked page can still appear in results if other sites link to it. To keep a page out, use a noindex instruction instead
- Keep staging and test sites blocked, and remove the block on the day you launch
- Decide on AI crawlers deliberately: blocking them can also keep you out of AI-written answers
Keep going.
Guides
Services
More free tools are on the tools page, including the free website audit.
Questions, answered.
What is a robots.txt file?
It is a plain text file at the root of a site that tells crawlers which paths they may or may not visit. Well-behaved crawlers read it first and follow it.
Where does the robots.txt file go?
In the root of the site, so that it opens at your domain followed by /robots.txt. A file placed anywhere else is ignored.
Can robots.txt keep a page out of Google?
Not reliably. It stops crawling, but a blocked page can still be listed if other sites link to it. To keep a page out of results, use a noindex instruction and leave the page crawlable so it can be read.
Does robots.txt protect private information?
No. The file is public, and anyone can read it. Private pages need a login, not a robots.txt line.
Should I block AI crawlers?
It is your choice. Blocking asks crawlers that gather text for AI to stay out, but it can also keep your pages out of AI-written answers. Crawlers that do not follow the rules ignore it either way.
