
When you ask ChatGPT, Gemini or Claude a question, the answer sounds like it came from someone who knows the subject. On the other side, though, is a language model: a program trained to estimate, very accurately, what the next word in a piece of text should be.
That one idea, repeated billions of times, is what lets a chatbot write an email, summarize a contract or explain quantum physics to a ten-year-old. It also explains why the model gets things wrong so confidently, and that in turn shapes how you should use it.
A next-word predictor
A language model calculates the probability of a word (or a piece of a word) showing up after a stretch of text. Google’s machine learning course uses the blank “When I hear rain on my roof, I ___ in my kitchen.” The model gives each possible continuation its own probability, such as “cook soup” or “warm up a kettle”, and picks among the likeliest.
The models behind today’s chatbots are large language models, or LLMs. “Large” refers to two things: they are trained on enormous amounts of text, and they have a huge number of parameters, the internal numbers that store what the model has learned. The same kind of model also summarizes, translates, classifies and analyzes text. When a new chatbot version ships, as in our coverage of ChatGPT with GPT-6, what changed is the model underneath the conversation.
Tokens: how the model reads text
Computers work with numbers, not letters, so text gets chopped into tokens, pieces that can be a whole word, part of a word or a single character. Each token gets a number, and the model calculates with those numbers.
In English, Google says one token is about 4 characters, or three quarters of a word, so 400 tokens come to roughly 300 words. Wikipedia adds that some other languages need noticeably more tokens for the same text, which is one reason non-English prompts can hit usage limits sooner.
Three stages of training
- Pretraining: the model reads a gigantic amount of text (books, websites, articles, code) and tries to predict what comes next. Every miss nudges the parameters slightly. Nobody writes rules of grammar or fact by hand; they emerge from the patterns.
- Instruction tuning: the model is trained on examples of requests paired with good answers, so it learns to follow instructions instead of just continuing the text.
- Reinforcement learning from human feedback (RLHF): people rate answers, a second model learns those preferences, and the main model is adjusted toward answers people prefer. This is the stage that turns a text autocomplete into a polite assistant.
Most current models are built on the transformer architecture, introduced in a 2017 Google paper. Its attention mechanism relates every part of a text to every other part, even distant ones, instead of reading strictly word by word.
Training is expensive. According to Wikipedia, GPT-2 (1.5 billion parameters, 2019) cost about $50,000 to train, and PaLM (540 billion parameters, 2022) about $8 million. Those are published estimates; the newest models do not disclose their costs reliably.
The context window sets what it can see
The context window is the number of tokens the model can consider at once: your request, the conversation so far and the answer being written. Anything outside it has no effect on the reply, which is why a very long chat can make the assistant “forget” something you said at the start. GPT-2 had a window of only about 1,000 tokens. Current models handle far more, and the exact limit depends on the product and plan.
A context window is also not permanent memory. Unless the app stores information between chats, the model starts fresh each time.
Why it makes things up
The model does not look facts up in a database. It generates the text that seems most likely, and most of the time the most likely text is also correct. Sometimes it isn’t.
When information is missing, the model rarely says “I don’t know”. It fills the gap with something plausible: a book that was never written, a wrong date, a law with the wrong number. Researchers call this a hallucination. IBM describes it as output that sounds plausible but is false or fabricated. It turns up most in recent events that happened after training, in obscure details such as the name of a small town’s council member, and in requests for exact citations with page numbers, links or word-for-word quotes.
To reduce the problem, companies use retrieval-augmented generation (RAG): before answering, the system searches trusted documents and hands that material to the model as the basis for its reply. That is what happens when a chatbot searches the web and shows its sources. It helps a lot, but the risk remains, because the model can still summarize what it read badly. Web search, clear instructions and human review all reduce hallucinations, and none of them guarantees correct answers every time.
Six habits for using a chatbot
- Give context. “Write an email” gets you little; “write a short, polite email to my landlord asking them to fix the garage light” gets you much more.
- Ask for sources when the topic matters, then open the links and check they say what the model claims.
- Allow an “I don’t know”. Telling the model in your prompt to say so when it is unsure cuts down on inventions, according to IBM’s guidance, though it does not eliminate them.
- Use it for what it does well: drafting, rewriting, summarizing, translating, brainstorming and explaining concepts.
- Be suspicious of very round or very specific numbers that show up without an explanation.
- Keep a person in the final decision on health, money and legal matters.
A useful picture is a fast, extremely well-read intern who never admits not knowing something. With supervision, it saves you hours; without it, it can land you in an awkward spot. The lineup of models changes quickly, as our coverage of Meta Muse Wonder shows, but the mechanics described here apply to all of them.
Frequently asked questions
Is a language model the same thing as artificial intelligence?
No. Artificial intelligence is the broad field; a language model is one kind of AI, focused on text. ChatGPT, Gemini and Claude are assistants built on language models.
Does the model understand what it writes?
It reproduces patterns of language very skillfully, which often looks like understanding. But it checks whether something is likely, not whether it is true. Checking remains your job.


