A modern AI chatbot is the product of several training stages, each doing a different job. This article explains what happens in each stage, what the foundational research papers actually report, and which questions the public record still leaves open.
Why is a chatbot trained in stages?
A large language model, the technology behind most chatbots, is a program that predicts which piece of text is likely to come next. Predicting the next word sounds modest, but doing it well across a huge range of text forces the system to pick up grammar, facts, writing styles and some problem-solving patterns.
Predicting text is not the same as being a good assistant, though. A system trained only to continue text may answer a question with another question, or continue a request in a rambling way, because that is what its training data sometimes looks like. Developers therefore add later stages that shape behavior. In broad terms, the pipeline looks like this:
| Stage | What the model learns from | Main purpose |
|---|---|---|
| Pretraining | Very large collections of text | General language ability and knowledge |
| Supervised fine-tuning | Example conversations written or chosen by people | Following instructions in a useful format |
| Preference training | Rankings of competing answers | Preferring answers people judge better and safer |
| Evaluation and testing | Held-out tests and adversarial probing | Finding weaknesses before release |
Company practices differ, and many details are not published. The sections below stick to what the research literature documents.
What happens during pretraining?
Pretraining is the most expensive stage. The model reads an enormous amount of text and repeatedly tries to predict the next token, a small chunk of text such as a word or part of a word. When it guesses wrong, its internal numbers, called parameters, are nudged so that the same mistake becomes less likely next time. Repeated across vast amounts of data, this adjusts billions of parameters.
A landmark 2020 paper, “Language Models are Few-Shot Learners,” described GPT-3, a model with 175 billion parameters. According to the paper’s abstract, that was 10 times more than any previous non-sparse language model. The authors reported that scaling up improved performance on tasks the model was never explicitly trained for. “Few-shot” means the model is shown a handful of examples inside the prompt, with no further training. The abstract states that the model was applied without gradient updates or fine-tuning, meaning its parameters were not changed for each task.
The same abstract is candid about limits. It says few-shot learning still struggles on some datasets, and that some datasets raise methodological issues related to training on large web corpora. In plain terms, when a model learns from a huge slice of the internet, test questions can end up in its training material, which makes some scores harder to interpret. The companion article on how AI benchmarks work and why scores mislead covers that problem in more depth.
Where does the training text come from?
Public descriptions generally say that pretraining text is drawn from web pages, books, code repositories and other sources, then filtered and de-duplicated. The 2020 GPT-3 paper itself refers to training on large web corpora. Beyond that general description, the composition of training sets for current commercial models is often not disclosed in detail, which is one reason debates over copyright, privacy and data quality continue. Readers should treat claims about exactly what any given model was trained on with caution unless the developer or an independent audit has published specifics.
How does a raw model become an assistant?
The next step is supervised fine-tuning. People write or select high-quality example responses to a wide variety of prompts, and the model is trained further to imitate them. The goal is not to teach new facts but to teach a manner of behaving: answer the question asked, follow the requested format, and decline clearly inappropriate requests.
The best-known description of this approach is the March 2022 paper “Training language models to follow instructions with human feedback,” whose lead authors include Long Ouyang and Jeff Wu. The paper describes collecting labeler demonstrations for supervised fine-tuning of GPT-3. The abstract does not state how many examples were used, so any specific figures quoted elsewhere should be checked against the full paper.
What is reinforcement learning from human feedback?
Showing a model good examples only goes so far. It is often easier for a person to say which of two answers is better than to write the ideal answer from scratch. Reinforcement learning from human feedback, usually shortened to RLHF, takes advantage of this.
The process works roughly as follows:
- The model produces several candidate answers to a prompt.
- Human labelers rank those answers from best to worst.
- A separate model, called a reward model, is trained to predict those rankings.
- The chatbot is then trained with reinforcement learning, a technique in which a system adjusts its behavior to earn higher scores, using the reward model as the scorer.
The 2022 paper reported a striking result. According to its abstract, outputs from a 1.3 billion parameter model trained this way were preferred by human evaluators over outputs from the 175 billion parameter GPT-3, even though the smaller model had about 100 times fewer parameters. Check the arithmetic: 175 divided by 1.3 is roughly 135, so “100 times fewer” is a conservative round figure that the authors use. The abstract also reports improved truthfulness and reduced toxic output, with minimal performance regressions on public language-processing datasets.
It adds a caution that deserves equal weight: the resulting models still make simple mistakes. Preference training makes answers more agreeable and more useful on average. It does not guarantee accuracy, which is why chatbots can still produce confident errors, a problem explained in why chatbots make things up.
Can AI systems give the feedback instead of people?
Human labeling is slow and costly, and it raises questions about whose judgments are being encoded. One alternative, described in a December 2022 paper titled “Constitutional AI: Harmlessness from AI Feedback,” replaces much of the human labeling with a written list of principles, often called a constitution.
According to the paper’s abstract, the method has two phases. In the supervised phase, the model generates responses, critiques them against the principles, revises them, and is then fine-tuned on the revisions. In the reinforcement phase, the model produces pairs of responses and another model judges which one better follows the principles. Those AI-generated preferences train a preference model that becomes the reward signal. The authors call this reinforcement learning from AI feedback.
The authors report that the resulting assistant was harmless but non-evasive, meaning it explains its objections to harmful requests rather than simply refusing. Note that this is a claim from the developers of the method, published as a research paper. Independent replication and long-term testing matter before treating any single training recipe as settled.
What can training not fix?
Training shapes tendencies, not guarantees. Several limits are worth keeping in mind:
- Knowledge is frozen at a point in time. A model only knows what was in its training data up to a cutoff, unless it is connected to outside sources. Approaches such as retrieval-augmented generation exist partly to address that gap.
- Feedback reflects the people giving it. Preferences from a particular group of labelers, or a particular written constitution, will not match every user’s values or every cultural context.
- Optimizing a proxy can drift. A reward model is only an estimate of what people want. A system can learn to please the scorer rather than to be correct.
- Training does not remove errors. Both the 2020 and 2022 papers acknowledge remaining weaknesses in their own results.
What is still unknown?
Much of the practical detail of frontier model training is not public: the exact data mixtures, the size and composition of human feedback datasets, how many training runs are discarded, and how much compute each stage uses. The foundational papers cited here are several years old, and commercial systems have moved on in ways their developers describe only partially. Newer approaches combine more reasoning-focused training and automated feedback, but independent descriptions are thinner than for the early methods.
For readers, the useful question to ask about any chatbot claim is which stage it relates to. A statement about knowledge usually concerns pretraining data. A statement about tone or refusals usually concerns fine-tuning and preference training. A statement about reliability needs independent evaluation, not just a description of how the model was trained.
Reading training claims with fewer surprises
The three-stage picture is a useful mental model: absorb text, learn to follow instructions, then be steered by judgments about what a good answer looks like. Each stage adds capability and introduces trade-offs, and none removes the need for checking important answers. As model makers publish more detail, and as standards bodies and researchers develop better evaluation methods, the gap between what is claimed and what can be verified should become easier to measure.
This article is general information about how AI systems work and is not professional advice.
Sources: arXiv
