What Is a Crawler? Bots, Spiders and User Agents

In plain English

A crawler is a program that fetches pages, extracts links and follows them. It identifies itself with a user agent string, which is how you can tell one from another in your logs.

The set of crawlers that matter has widened. Alongside search engine bots there are now AI training and retrieval crawlers, and deciding which to allow is a real editorial decision.

What to know

User agents

Each crawler identifies itself, which lets you allow, block or measure it separately.

Not a browser

Some crawlers execute JavaScript, some do not, and some do so slowly.

Rules apply

Well-behaved crawlers respect robots.txt. Not all crawlers are well-behaved.

Server load

Aggressive crawling can affect site performance and shows up in server logs.

Why it matters

Knowing which crawlers visit and what they do lets you make deliberate decisions: which to welcome, which to slow down, and whether AI crawlers should be allowed to use your content. That is now a business question, not just a technical one.

Common mistakes

×Blocking useful crawlers by accident with a broad robots rule.

×Assuming every crawler renders JavaScript.

×Never looking at server logs to see who is actually visiting.

×Blocking AI crawlers without considering the visibility cost.

FAQs

Should I block AI crawlers?

It depends on whether you want to appear in AI answers. Blocking removes the content and the citation.

Do all crawlers obey robots.txt?

Reputable ones do. It is a request, not an enforcement mechanism.

WRITTEN BY TARIQ SALLAM

Marketing Consultant. Entrepreneur. Content Creator.

I'm a marketing consultant, entrepreneur and content creator. I help businesses grow through practical marketing, websites, SEO, content and AI.

More About Tariq →