In plain English
It controls crawling, not indexing. A disallowed URL will not be fetched, but if other sites link to it, it can still appear in results without a description.
It is also public, plain text, and the single easiest file to break a site with. One stray disallow line can remove a whole section from search.
What to know
Controls fetching
Disallow prevents requests to a path. It does not remove anything already indexed.
Not access control
Anything sensitive needs authentication, not a robots rule.
Public
Anyone can read it, including the paths you decided to hide.
Order matters
Rules are matched by specificity, and mistakes are easy to make quietly.
Why it matters
Robots.txt causes more accidental traffic loss than almost any other single file, usually through a staging rule that shipped to production. It is worth checking after every deployment.
Common mistakes
×Disallowing the whole site after a launch and not noticing for weeks.
×Using it to try to deindex pages, which prevents the noindex being read.
×Blocking CSS and JavaScript the page needs to render.
×Assuming it hides anything. It is a public list of what you did not want crawled.
FAQs
Will robots.txt remove a page from Google?
No. Use noindex, or remove the page and return the correct status.
Where does it go?
At the root of the domain, and nowhere else.
WRITTEN BY TARIQ SALLAM
Marketing Consultant. Entrepreneur. Content Creator.
I'm a marketing consultant, entrepreneur and content creator. I help businesses grow through practical marketing, websites, SEO, content and AI.
More About Tariq →