What Is Pretraining? Where a Model's Knowledge Comes From

In plain English

A very large body of text is used to teach the model language, reasoning patterns and a great deal of general knowledge. This is the expensive stage, and it happens once.

It is also why models have a knowledge cutoff. Anything published after training finished is unknown unless the model can search or is told.

What to know

Done once, at scale

Enormously expensive, performed by the model's developer.

Source of general knowledge

Whatever was in the training data, with all its gaps and biases.

Creates the cutoff

Events after training are unknown without retrieval.

Followed by tuning

Instruction tuning and alignment come after pretraining.

Why it matters

Pretraining explains both what a model knows and what it cannot know. It is also why what does it know about my business is usually the wrong question: the answer is very little, and the fix is to tell it rather than to train it.

Common mistakes

×Assuming a model knows recent events.

×Expecting knowledge of a niche business that was never written about publicly.

×Confusing pretraining with fine-tuning.

×Relying on training knowledge for anything that changes.

FAQs

What is a knowledge cutoff?

The point after which the model was not trained on anything, so it has no knowledge of later events.

Can I add to pretraining?

Not practically. Use retrieval, or fine-tune for behaviour.

WRITTEN BY TARIQ SALLAM

Marketing Consultant. Entrepreneur. Content Creator.

I'm a marketing consultant, entrepreneur and content creator. I help businesses grow through practical marketing, websites, SEO, content and AI.

More About Tariq →