A large language model, or LLM, is a computer program, specifically a neural network, that has been trained on enormous amounts of text to predict which piece of text is likely to come next. Chatbots, writing assistants and code helpers are built by repeating that one prediction many times, one small chunk of text after another.
This guide explains each step in plain terms, what the key research papers report, and which common beliefs about these systems are wrong.
What is a large language model, in one paragraph?
A language model assigns probabilities to text. Given the words “The capital of France is”, a good model rates “ Paris” as far more likely than “ banana”. A neural network is a mathematical structure made of layers of simple calculations whose behavior is controlled by numbers called parameters (also called weights). “Large” refers mainly to how many parameters the model has and how much text it learned from. There is no official size at which a model becomes “large”; the term is a convention.
Researchers at Stanford’s Institute for Human-Centered AI use a broader label, “foundation models”, for models “trained on broad data at scale” that can be adapted to many tasks. Language models such as GPT-3 are examples. The same researchers note that these systems are not yet fully understood, which is a theme worth keeping in mind throughout.
For a wider introduction to the field, see this plain-English guide to artificial intelligence.
How does a large language model work, step by step?
The process from a typed question to a printed answer has five main steps.
- Split the text into tokens. The model does not read letters or whole words directly. A tokenizer breaks text into tokens, which are chunks such as a short word, part of a longer word, or punctuation.
- Turn tokens into numbers. Each token is mapped to a list of numbers, called an embedding, that places it in a mathematical space where related tokens tend to sit near each other.
- Mix information across the text. Layers of the network, built on a design called the transformer, let every token take account of the other tokens around it.
- Score every possible next token. The final layer outputs a probability for each token in the model’s vocabulary.
- Pick one, append it, repeat. The system selects a token, adds it to the text so far, and runs the whole process again until the answer is complete.
That loop is why responses appear to stream out word by word: they are literally generated one token at a time.
Why did the transformer change everything?
Before 2017, leading language systems typically relied on recurrent or convolutional networks, designs that process text in sequence or in small local windows. In June 2017 a team of eight researchers, including Ashish Vaswani and Noam Shazeer, posted the paper “Attention Is All You Need” on arXiv, the open research repository. It proposed the Transformer, a network based solely on a mechanism called attention, “dispensing with recurrence and convolutions entirely,” in the authors’ words.
Attention lets the model weigh how relevant each other token is when processing a given one. In the sentence “The trophy did not fit in the suitcase because it was too big”, attention is how a model can connect “it” to the trophy rather than the suitcase. The paper reported results on machine translation: 28.4 BLEU (a standard translation-quality score) on English-to-German, and 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs, graphics processors commonly used for AI work. The authors also said the design was more parallelizable, meaning many calculations can run at once, which is a large part of why models could later be scaled up so far.
How is the model trained?
Training starts with pretraining. The model is shown vast quantities of text with the next token hidden, makes a guess, and is scored on how wrong it was. A procedure called gradient descent then nudges the parameters so the same mistake becomes slightly less likely. Repeating this across a huge collection of text is what builds up the model’s apparent knowledge of grammar, facts and style.
Scale matters. The 2020 paper “Language Models are Few-Shot Learners” described GPT-3, an autoregressive model (one that generates text left to right) with 175 billion parameters, which its authors described as ten times more than any previous non-sparse language model. They reported that it could perform tasks such as translation and three-digit arithmetic from a few examples placed in the prompt, without any further training.
Size is not the only lever. A 2022 paper from DeepMind researchers, “Training Compute-Optimal Large Language Models,” trained more than 400 models and concluded that existing large models were significantly undertrained: model size and the number of training tokens should be scaled equally. Their test model, Chinchilla, had 70 billion parameters and used the same compute budget as the 280 billion parameter Gopher but four times more data. The authors report that it outperformed Gopher and GPT-3 across a broad range of evaluation tasks. Note that these are the authors’ own reported results on their chosen benchmarks.
The training of a finished assistant has further stages after pretraining. Those are covered in how AI chatbots are trained, from pretraining to feedback.
A worked example of next-token prediction
The numbers below are illustrative arithmetic, not output from a real model. Suppose the prompt is “The capital of France is” and the model assigns these probabilities to candidate next tokens:
| Candidate token | Illustrative probability |
|---|---|
| Paris | 0.90 |
| a | 0.04 |
| located | 0.03 |
| Lyon | 0.01 |
| all other tokens combined | 0.02 |
The probabilities sum to 1.00, as they must (0.90 + 0.04 + 0.03 + 0.01 + 0.02). If the system always takes the top choice, it outputs “Paris”. Many products instead sample from the distribution with a setting often called temperature, so that over many runs “Paris” appears about 90 times in 100 and other tokens occasionally appear. That randomness is why asking the same question twice can produce different wording, and, on questions where the probabilities are closer together, different answers.
Does a language model understand what it says?
This is the most contested question in the field, and the honest answer is that researchers disagree about what “understand” should mean. What can be said plainly is how the system is built: it is trained to produce plausible text, not to look facts up in a verified database. The Stanford-led foundation model report says that researchers lack a clear picture of how these models work, when they fail and what they are capable of.
Two practical consequences follow:
- Fluent is not the same as correct. A model can write a confident, well-formed sentence that is false. This is the problem described in why chatbots make things up.
- Skill is uneven. The GPT-3 authors themselves noted that its few-shot performance still struggled on some datasets, and that training on large web collections raises methodological issues, such as test material leaking into the training data.
What are the most common misconceptions?
“It searches the internet for every answer.” A standard model answers from patterns stored in its parameters. Some products add a separate search or document-retrieval step, an approach explained in retrieval-augmented generation, but that is an add-on, not the core model.
“It is a database of facts.” The parameters are numbers that encode statistical patterns. Facts are not stored as retrievable entries, which is why recall can be patchy.
“It learns from my conversation as we chat.” In normal use the parameters do not change during a conversation. The model uses the text in the current exchange, held in what is called its context window, the limited amount of text it can consider at once. Whether a company later uses conversations to improve future models is a policy question for that provider.
“Bigger always means better.” The Chinchilla result suggests that balance between model size and training data matters, and a smaller, well-trained model can beat a larger, undertrained one on the tests the authors used.
“Every chatbot is a different technology.” Many products are built on the same few underlying models. The Stanford researchers call this homogenization and warn that defects in a shared base model are inherited by all the models adapted from it.
What changes by model, provider and date?
Almost everything about specific systems varies. Parameter counts, training data, context window length, the cutoff date after which the model knows nothing, and the safeguards applied differ by developer and change with each release. Many developers publish few details about their newest systems, so figures quoted in marketing should be treated as company claims unless confirmed by independent evaluation. The papers cited here are from 2017 to 2022; they explain the foundations well but do not describe any current commercial product.
Using the idea sensibly
The most useful mental model is a very well-read autocomplete: it is good at producing text that fits the pattern of a request, which makes it helpful for drafting, summarizing and explaining, and unreliable as a sole source for facts, figures or citations. Checking important claims against primary sources remains necessary, however fluent the answer sounds.
This article is general information about how AI systems work and is not professional advice.
Sources: arXiv



