What Is Meta Robots? Page-Level Crawling and Indexing Directives
DICTIONARY · SEO

What Is the Meta Robots Tag?

The meta robots tag is a page-level instruction telling search engines how to treat that specific page.

In plain English

Where robots.txt governs whether a URL can be fetched, meta robots governs what happens to the page once it has been. Its values are instructions, not hints.

The two most used are noindex, keeping a page out of the index, and nofollow, telling search engines not to follow the links on it.

What to know

Page-level
Applies to one page, unlike site-wide robots.txt rules.
Common values
index, noindex, follow, nofollow, noarchive, max-snippet.
Header equivalent
X-Robots-Tag delivers the same directives for PDFs and other non-HTML files.
Requires crawling
The page must be fetchable for the tag to be read at all.

Why it matters

Meta robots is where indexing is actually controlled. Most cases of a page inexplicably absent from search, or an unwanted page persistently present, come down to this tag or its absence.

Common mistakes

×Assuming robots.txt does what meta robots does.
×Site-wide noindex applied by a plugin setting nobody checked.
×Using nofollow on internal links to sculpt authority.
×Forgetting X-Robots-Tag for PDFs, which then get indexed.

FAQs

Do I need a meta robots tag on every page?

No. Its absence means index and follow, which is usually what you want.

What about noarchive and max-snippet?

Useful in publishing and licensing contexts to control how much of the page is shown.

WRITTEN BY TARIQ SALLAM
Marketing Consultant. Entrepreneur. Content Creator.

I'm a marketing consultant, entrepreneur and content creator. I help businesses grow through practical marketing, websites, SEO, content and AI.

More About Tariq →