· 7 min read

How to Create a robots.txt File: Syntax, Examples, and Common Mistakes

A small text file called robots.txt can make the difference between a site that search engines crawl efficiently and one that wastes time on the wrong pages, or worse, can’t be crawled at all. Learning how to create a robots.txt file is simple, but a single wrong line can hide your whole site from Google.

This guide explains what robots.txt does, how to write it, shows safe examples you can adapt, and lists the mistakes to avoid.

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website, for example https://www.example.com/robots.txt. It tells crawlers, such as the ones used by search engines, which parts of your site they may or may not request.

It follows the Robots Exclusion Protocol. Reputable crawlers respect it, but it is not a security system: anyone can read it, and bad bots can ignore it.

What robots.txt can and can’t do

It canIt can’t
Ask crawlers not to visit certain folders or URL patternsGuarantee that a page stays out of search results
Reduce crawling of low-value pages, such as internal search resultsProtect private or sensitive content
Point crawlers to your XML sitemapRemove a page that is already indexed
Set different rules for different crawlersForce bad bots to obey

The key point: robots.txt controls crawling, not indexing. A blocked URL can still appear in search results, usually without a description, if other sites link to it. To keep a page out of search results, use a noindex directive, which only works if the page is not blocked, or put the page behind a login.

The basic syntax

A robots.txt file is made of groups of rules. Each group starts with a user-agent line and is followed by rules.

  • User-agent: names the crawler the rules apply to. An asterisk * means all crawlers.
  • Disallow: a path the crawler should not request.
  • Allow: a path that is allowed, often used to make an exception inside a blocked folder.
  • Sitemap: the full URL of your XML sitemap.

Two special characters are widely supported by major search engines: * matches any sequence of characters, and $ marks the end of a URL. Paths are case sensitive, and lines starting with # are comments.

How to create a robots.txt file step by step

  1. Decide what to block. Make a short list of sections that have no value in search, such as admin areas, carts, internal search results, and endless filter combinations.
  2. Open a plain text editor. Use a basic editor, not a word processor, and save in UTF-8.
  3. Write your rules using the syntax above.
  4. Add your sitemap line. It helps crawlers find your important pages.
  5. Name the file exactly robots.txt in lowercase.
  6. Upload it to the root of your domain, so it loads at yourdomain.com/robots.txt.
  7. Test it before and after publishing, as described below.

Remember that each subdomain needs its own file. A robots.txt on www.example.com doesn’t apply to blog.example.com.

Robots.txt examples you can adapt

Allow everything

This is the simplest valid file. An empty Disallow means nothing is blocked.

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Block a folder

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Block internal search and filter URLs

User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://www.example.com/sitemap.xml

Block a folder but allow one file inside

User-agent: *
Disallow: /private/
Allow: /private/public-guide.html

Set rules for one specific crawler

User-agent: ExampleBot
Disallow: /

User-agent: *
Disallow: /admin/

Block a whole site (for a staging or test site only)

User-agent: *
Disallow: /

Warning: never use this on your live website. It asks all crawlers to stay away from everything.

How to test your robots.txt file

  1. Open it in a browser. Visit yourdomain.com/robots.txt and make sure it loads as plain text.
  2. Check Google Search Console. The robots.txt report shows the files Google found, when it last fetched them, and any problems.
  3. Inspect important URLs. Use the URL inspection tool to confirm that your key pages are not blocked.
  4. Re-test after every change, especially after a redesign, migration, or plugin update.

For a complete walkthrough of everything that can affect crawling and indexing, read our technical SEO audit guide.

Common robots.txt mistakes

  • Blocking the whole site by accident. A leftover Disallow: / from a development site is a classic cause of vanishing rankings.
  • Blocking CSS and JavaScript files. Search engines need them to render and understand your pages.
  • Using robots.txt to hide pages from search results. Blocked URLs can still be indexed. Use noindex or authentication instead.
  • Blocking a page and adding noindex. If the crawler can’t visit the page, it can’t see the noindex tag.
  • Wrong location or file name. The file must be at the root and called exactly robots.txt.
  • Typos and wrong case. Paths are case sensitive, so /Admin/ and /admin/ are different.
  • Forgetting subdomains. Each one needs its own file.
  • Relying on unsupported directives. Google ignores Crawl-delay, for example.
  • Putting secrets in it. Since anyone can read the file, don’t list private URLs there.

Robots.txt and AI or other crawlers

Many site owners now add rules for specific crawlers, including those used by AI companies, by naming their user-agent in a separate group. Each company publishes the user-agent names it uses, so check the official documentation before writing rules. Keep in mind that compliance is voluntary.

Robots.txt, sitemaps, and noindex: what to use when

GoalBest tool
Reduce crawling of low-value sectionsrobots.txt
Help crawlers find your important pagesXML sitemap, also referenced in robots.txt
Keep a page out of search resultsnoindex directive, with the page left crawlable
Protect private contentLogin or server-level access control
Point duplicate pages to the main oneCanonical tags or redirects

Check your technical SEO basics

Robots.txt is one piece of the technical foundation. A quick audit can also reveal indexing, speed, and mobile issues that hold your pages back. Analyzaa analyzes your pages in seconds and scores SEO, performance, trust, UX, and conversions, with a prioritized list of what to fix first.

Try the Technical SEO Audit Tool or the AI SEO Score Checker. For the bigger picture, read how to improve website ranking and our 25 SEO tips.

Start your free audit →

FAQ

Where do I put the robots.txt file?

At the root of your domain, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

Do I need a robots.txt file?

Not always. If you don’t have one, crawlers assume they can visit everything. It’s useful when you want to limit crawling of certain sections or point to your sitemap.

Does robots.txt remove pages from Google?

No. It only asks crawlers not to visit a URL. To keep a page out of search results, use a noindex directive on a crawlable page, or restrict access with a login.

Can I block Google from my site with robots.txt?

Yes, by disallowing all paths for its crawlers, but this will stop your pages from being crawled and can cause you to lose visibility. Do it only for sites you don’t want in search, such as staging environments.

How do I check if my robots.txt file is working?

Open it in your browser, use the robots.txt report and the URL inspection tool in Google Search Console, and confirm that your important pages are not blocked.


Want to test your website right now?

Run a free instant audit across SEO, speed, UX, and conversions with Analyzaa.

Free SEO Audit →

← Back to blog