LLMs & transformers.
How ChatGPT actually works.
You've used ChatGPT. You've seen it write code, answer questions, tell jokes, and — sometimes — confidently state things that are wrong. This lesson explains what's actually happening: the next-token prediction loop, what temperature does, why hallucinations are inevitable, what a context window really is, and how to write prompts that get better results. No magic. Just the mechanism.
An LLM is a transformer trained to predict the next token. That's the entire objective. Given a prompt, it predicts the most likely next token, appends it, and repeats. That loop — generation — produces everything from a haiku to working code. The model doesn't "think of the whole answer" — it predicts one token at a time, 10-50 times per second. Understanding this explains hallucinations, temperature, context limits, and why some prompts work better than others.
Before this lesson: complete Lesson 4. You need to understand tokens, embeddings, and attention — they're the foundation for everything here.
🎯 Next-token prediction — the entire objective
The whole training objective is one sentence. Everything else emerges from it.
TRAINING OBJECTIVE (during training, on billions of documents): "Given these tokens, predict the next one." That's it. That's the entire loss function. Text: "The capital of France is ___" Model predicts: "Paris" If wrong, adjust weights. Repeat billions of times. GENERATION (what happens when you use ChatGPT): Step 1: You type: "Write a haiku about rain" Step 2: Model tokenizes → [write, a, haiku, about, rain] Step 3: Model predicts next token: → "Soft" (most likely given the prompt) Step 4: Append "Soft" to the sequence Sequence becomes: [write, a, haiku, about, rain, Soft] Step 5: Predict next token: → "raindrops" Step 6: Append "raindrops": Sequence: [write, a, haiku, about, rain, Soft, raindrops] Step 7: Predict next token: → "fall" ...repeat until end-of-sequence token or length limit... Final output: "Soft raindrops fall Whispering on window glass Nature's gentle song" EVERY RESPONSE IS GENERATED THIS WAY. Every poem, every line of code, every answer. One token at a time. 10-50 tokens per second. No "thinking ahead." No "planning the whole answer." WHY THIS MATTERS: → Hallucinations (next section) → Temperature (next section) → Why prompts matter → Why LLMs fail at math All follow from this single mechanism.
Autoregressive
The model predicts one token, then uses that token as input for predicting the next. Each token depends on all previous tokens. This is why generation is fast but not perfect — errors compound.
Greedy vs sampled
Greedy: always pick the most likely token. Sampled: randomly pick from the distribution. ChatGPT uses sampling with temperature — that's why the same prompt gives different answers sometimes.
End-of-sequence
The model learns to emit a special end token when the response is complete. It's not deciding "I'm done" — it's predicting that this particular token is most likely next.
Generation speed
Modern models generate 10-50 tokens per second. A 1,000-token response takes 20-100 seconds. That's the delay before you see the full answer — it's not "thinking," it's generating.
Try this: type "The sky is" and let ChatGPT autocomplete it 5 times. Different completions? That's temperature and sampling at work. Same completion every time? That's greedy decoding. Most LLMs allow you to control this via the API. Understanding next-token prediction makes all of this predictable rather than mysterious.
🌡️ Temperature — the randomness knob
Controls how "creative" or "focused" the model is. The most important setting in the API.
After the model produces a probability distribution over the vocabulary, temperature changes the distribution before sampling. PROMPT: "The best programming language is" RAW PROBABILITIES (before temperature): Python 0.30 JavaScript 0.25 Rust 0.15 Go 0.10 Java 0.08 C++ 0.07 ...other 0.05 TEMPERATURE = 0 (Greedy) Always pick the highest. → "Python" every time. Deterministic. Repeatable. Use for: factual answers, code, structured output. TEMPERATURE = 0.7 (Default, balanced) Sample proportionally to probabilities. → Usually "Python" (0.30 chance) → Often "JavaScript" (0.25 chance) → Sometimes "Rust" (0.15 chance) → Rarely "Go" (0.10 chance) → Almost never the others Use for: general chat, most applications. TEMPERATURE = 1.5 (High, creative) Flatten the distribution — more likely to pick lower-probability tokens. → "Python" still favored (0.15 chance) → "Go" gets (0.12 chance) — much higher than before → Surprises happen Use for: creative writing, brainstorming. TEMPERATURE = 3.0 (Extreme, near-random) Distribution is nearly uniform. Sampling becomes almost random. → Output usually incoherent. Don't use. THE RULE OF THUMB: task requires precision (facts, code) → temperature 0-0.3 task requires creativity (writing, ideas) → temperature 0.7-1.0 task requires chaos (rarely useful) → temperature 1.5+ ANALOGY: Temperature 0 = "give me the answer" Temperature 0.7 = "give me a good answer" Temperature 1.5 = "surprise me"
Temperature 0
Greedy — always pick the most likely token. Deterministic, repeatable, but sometimes bland. Use for factual answers, code generation, structured output, unit tests.
Temperature 0.7
The typical default for chat. Balances creativity and coherence. Good for most general use — conversation, moderate creativity, general Q&A.
Temperature 1.0+
Creative — more likely to pick less obvious tokens. Use for brainstorming, poetry, fiction, alternative ideas. Watch out: too high makes the output incoherent.
Temperature is per-token
It's applied at each generation step, not to the whole response. A high temperature can drift wildly even in a single response. That's why high-temperature outputs feel inconsistent.
Temperature 0 does not guarantee correctness. It guarantees determinism — the same input always produces the same output. But if the model's most likely next token is wrong, temperature 0 will confidently pick the wrong answer. Lower temperature improves consistency, not accuracy. For accuracy, you need better training data, RAG, or verification.
👻 Hallucinations — why LLMs confidently lie
Not a bug. An inherent property of next-token prediction. Here's why.
An LLM has no mechanism to check whether what it says is true. It generates the most plausible next token based on patterns in training data. Most of the time, plausible equals true — the training data contains mostly accurate text, so the statistical patterns align with reality. But when the model encounters a query outside its training distribution, or a query where the training data is contradictory, it still generates a confident-sounding response. That's a hallucination.
PROMPT: "What is the population of the small town of Xyzton?" (Assume "Xyzton" is a made-up name.) The model has never seen "Xyzton" in training data. But it still generates a response: Step 1: Most plausible next token after "The population of Xyzton is": → "approximately" (high probability, grammatically correct) Step 2: Next token: → "12,000" (plausible number for a small town) Step 3: Next token: → "." (sentence end) Output: "The population of Xyzton is approximately 12,000." → CONFIDENT. → SPECIFIC. → COMPLETELY FABRICATED. WHY THIS HAPPENS: The model isn't "checking" whether Xyzton exists. It's generating the most plausible sequence of tokens given the prompt "What is the population of..." Pattern matches: "What is the population of ___?" → answer pattern "The population of ___ is approximately ___,___." The model fills in the blank with plausible content. It has no mechanism to say "I don't know" unless "I don't know" is the most plausible next token — which it usually isn't, because training data is full of confident answers, not uncertain ones. TYPES OF HALLUCINATIONS: 1. Fabricated facts (invented numbers, dates, names) 2. Fabricated citations (invented paper titles, authors) 3. Fabricated sources (invented URLs, book titles) 4. Wrong details (real entity, wrong attribute) 5. Confident errors (right format, wrong content) HOW TO MITIGATE: • RAG (retrieval-augmented generation) — give the model actual documents to reference, not just its training memory • Verification — check facts against a real source • Lower temperature — reduces creativity but not hallucinations • "I don't know" prompting — ask the model to admit uncertainty • Ask for citations — then verify them (models often hallucinate citations too, so this is imperfect)
Not fixable
Hallucinations are inherent to next-token prediction. Larger models hallucinate less, but they still do. The mechanism that produces good answers also produces confident lies. They're the same mechanism.
RAG helps
Retrieval-Augmented Generation: give the model real documents to reference. Instead of relying on training memory, the model quotes from provided sources. This is the standard fix for factual tasks.
Trust but verify
Never trust LLM output for critical facts — numbers, dates, names, citations. Always verify against a real source. The model sounds confident whether it's right or wrong.
The confidence trap
LLMs are confident even when wrong. This is because confident-sounding text is well-represented in training data. A hesitant answer would have low probability, so it's rarely generated. Don't use tone as a proxy for accuracy.
Never trust LLM-generated citations without verification. ChatGPT has fabricated legal cases, scientific papers, and book titles — with authors and page numbers that look real. Lawyers have been sanctioned in court for citing hallucinated cases. If a model gives you a citation, verify it against the actual source. If you can't find it, it probably doesn't exist. This is not paranoia — it's the standard practice for anyone using LLMs for research.
📦 Context window — the model's working memory
Everything the model can "see" at once. Your prompt, chat history, and the response all count.
THE CONTEXT WINDOW INCLUDES: ┌─────────────────────────────────────────┐ │ 1. System prompt │ │ (instructions for the model) │ │ │ │ 2. Full conversation history │ │ (every previous user message │ │ + every previous model response) │ │ │ │ 3. Current prompt │ │ (what you just typed) │ │ │ │ 4. Reserved for response │ │ (space for the model to generate) │ │ │ └─────────────────────────────────────────┘ ALL of this together must fit within the model's context window. Exceed it and something has to give. CONTEXT WINDOW SIZES (2026): GPT-3.5-turbo: 4,096 tokens (~3,000 words) GPT-4-turbo: 128,000 tokens (~96,000 words) Claude 3.5: 200,000 tokens (~150,000 words) Gemini 1.5 Pro: 2,000,000 tokens (~1,500,000 words) WHAT HAPPENS WHEN YOU EXCEED IT: Older models: Error, request fails Modern models: Truncate oldest messages (chat "forgets" the beginning) Some models: Reject and warn WHY THIS MATTERS: • Long chat = model forgets early messages • Long document = must be chunked into pieces • Long code base = can't fit all at once • API cost is per token — long context = expensive PRACTICAL RULE: For long documents, chunk into 3,000-5,000 token pieces and process each piece separately, or use a model with a larger context window. Never assume the model "sees" everything in a long chat — it sees what fits. VISUAL: Total conversation length ├────────────────────────────────────────┤ │← fits in context →│← cut off →│ │ │ │ │ Model can see this │ Model can't │ │ │ see this │ └────────────────────────────────────────┘ ↑ context limit When new messages push length past the limit, the oldest messages disappear. The model literally cannot remember them.
Everything counts
System prompt + chat history + your message + the response. All tokens count. A 10-message chat uses much more context than a single question.
Chat memory is not real
The model doesn't have persistent memory. It "remembers" the conversation only because the conversation is included in the context each turn. When the context truncates, the memory is gone.
Bigger isn't always better
Larger context windows cost more (per token) and can slow down inference. For most tasks, a smaller window plus RAG is more efficient than a giant context.
The "lost in the middle" problem
Even within a large context window, models tend to attend less to information in the middle. Critical instructions should go at the start or end, not buried in a 100-page document.
The most common LLM application bug: forgetting that chat history counts against the context. Build a customer support bot that remembers 50 messages of history and the context fills up fast. The model starts "forgetting" recent messages because old ones are still in the context. Fix: summarize the conversation periodically, or use a retrieval system that pulls only relevant history into each turn. The context window is a hard limit, not an infinite scroll.
✍️ Prompt engineering — getting better outputs
Not magic. Clear communication with a statistical model.
1. BE SPECIFIC ✗ "Write about Python" ✓ "Write a 200-word intro to Python variables for absolute beginners, using analogies" 2. PROVIDE CONTEXT ✗ "Fix this code" (which code? what's broken?) ✓ "Fix this Python function — it's supposed to return the average, but it crashes on empty lists. [code]" 3. USE EXAMPLES (few-shot) ✗ "Classify these reviews as positive or negative" ✓ "Classify reviews. Example: 'Love it!' → positive Example: 'Terrible product' → negative Now: 'Best purchase I've made' → ?" 4. SPECIFY FORMAT ✗ "Give me user data" ✓ "Return user data as JSON with fields: name (string), age (number), email (string)" 5. ITERATE First prompt rarely works. Refine based on output. "Good, but make it shorter" "Good, but add examples" "Good, but change the tone to be more formal" ADVANCED PATTERNS: CHAIN OF THOUGHT: "Think step by step" or "explain your reasoning" Forces the model to generate reasoning-like text, which improves accuracy on multi-step problems. ROLE PROMPTING: "You are an expert Python developer reviewing code" Changes the distribution of likely responses. NEGATIVE PROMPTS: "Do NOT include filler phrases like 'Certainly!'" Explicitly excludes certain patterns. STRUCTURED OUTPUT: "Respond ONLY with valid JSON, no markdown, no prose" Forces the model into a specific format. REAL IMPROVEMENT: Vague prompt: ~50% useful output Specific prompt: ~80% useful output Few-shot prompt: ~90% useful output The improvement is real, but it's not magic — it's just giving the model clearer patterns to match against.
Clarity
The model can't read your mind. Every detail you don't specify is a gap it fills with the most common pattern. Be explicit about what you want.
Examples
One good example is worth 100 words of description. "Here's an example of what I want" is often the fastest way to get what you want.
Chain of thought
"Think step by step" forces the model to generate reasoning text before the answer. This improves math, logic, and multi-step problems by 20-40% in accuracy.
Iteration
The first output is a draft, not the answer. Refine, ask for changes, push back. The best LLM users treat prompts as conversations, not one-shot requests.
Chain-of-thought prompting works for a mechanical reason. When the model generates "Let me think step by step," it produces reasoning-like text. That reasoning text becomes part of the context for the actual answer — and reasoning-like text is more likely to be correct than conclusion-like text, because the training data contains more correct reasoning steps than correct conclusions. You're not making it "think" — you're forcing it to generate a better context window for the final answer. It's a trick that exploits the mechanism, not a claim about intelligence.
🏆 Challenge
Three tasks. Each exercises a different LLM concept.
Task 1 — Predict the next token
For each prompt, predict the most likely next token and explain why:
1. "The capital of Japan is"
2. "2 + 2 ="
3. "Once upon a"
4. "The best pizza topping is"
5. "def hello():"
For each, is temperature 0 or temperature 1 better for the task? Why?
Task 2 — Spot the hallucination
Here are 5 statements. For each, decide: is this probably true, probably false, or possibly a hallucination? How would you verify?
1. "Python was created by Guido van Rossum in 1991."
2. "The 2023 paper 'Attention is All You Need II' extended the original transformer architecture."
3. "JavaScript's let was introduced in ES6 (2015)."
4. "The Rust programming language was created by Apple in 2008."
5. "GPT-4 has approximately 1.7 trillion parameters."
Which ones can you verify from memory? Which need a search? Why does the model sound confident either way?
Task 3 — Rewrite the prompt
Rewrite each weak prompt into a strong one. What changes would you make?
1. "Write a blog post about AI"
2. "Fix my code"
3. "Explain databases"
4. "Write a cover letter"
5. "Give me ideas"
For each, apply the 5 principles: be specific, provide context, use examples, specify format, plan to iterate. Write the improved prompt. Then explain why the original would have failed.
⚠️ Common mistakes
The LLM misconceptions that catch every beginner.
LLMs are confident whether right or wrong. Confidence is not accuracy. Verify anything critical — numbers, dates, names, citations. Especially citations.
Temperature 0 means deterministic, not accurate. If the model's highest-probability next token is wrong, temperature 0 will confidently pick wrong. It improves consistency, not accuracy.
Every previous message counts against the context window. Long chat = model forgets. Long document = must chunk. The context window is a hard limit, not an infinite scroll.
The first prompt is a draft. Refine, ask for changes, push back. "Good, but shorter" and "Good, but more examples" are the fastest way to improve output.
The model doesn't "know" anything. It predicts tokens. When it "knows" Python, it's producing Python-looking text because Python is well-represented in training data. No understanding, just pattern matching.
LLMs fabricate citations — papers, authors, page numbers, URLs. All look real, most are invented. Always verify against the actual source. This has caused lawyers to be sanctioned.
LLMs struggle with arithmetic, especially large numbers. They generate plausible answers, not computed ones. Use a calculator, or ask the model to write code that computes the answer — then run it.
If the task requires factual accuracy about specific documents, RAG (retrieval-augmented generation) is the right approach. Feeding the model the actual documents is far more reliable than asking it to remember.
🎓 AI track complete
Five lessons. One complete mental model. Here's what you can do now.
You understand the mechanism
Tokens, embeddings, attention, transformers, next-token prediction, temperature, context windows, hallucinations. Every concept that appears in AI articles and product pitches is now legible.
You can evaluate claims
"AGI by 2027," "AI will replace developers," "AI is just statistics." You can read any AI claim and assess whether it's plausible, hype, or nonsense.
You can use AI better
Better prompts, lower expectations for facts, higher for structure. You know when to trust the model and when to verify. You know how to iterate.
You can go deeper
PyTorch, Hugging Face, RAG, fine-tuning, prompt engineering, LLM applications. The foundation is built — everything else is discovery.
What to build before moving to advanced AI. Build one small LLM-powered app: a summarizer, a chatbot that answers questions about a PDF, a code explainer. Use the OpenAI API or a local model via Ollama. Handle the context window carefully, verify the outputs, and use RAG for anything factual. That one project will teach you more than the next ten tutorials — because the failure modes only become real when you hit them.
➡️ What's next
You've finished the AI track. Three directions from here.
❓ Frequently asked questions
What is an LLM?
An LLM (Large Language Model) is a neural network trained to predict the next token in a sequence of text. That's the entire objective — no more, no less. GPT-4, Claude, Gemini, and Llama are all trained this way. The 'large' refers to the model size — billions of parameters — and the amount of training data.
How does an LLM generate text?
Autoregressive generation: give the model a prompt, it predicts the most likely next token, appends it to the prompt, then predicts the next, then the next. This loop continues until the model predicts an end-of-sequence token or hits a length limit. Every response you've ever gotten from ChatGPT was generated one token at a time.
What is temperature in LLMs?
Temperature controls randomness in token selection. Temperature 0 always picks the most likely token — deterministic, repeatable, but sometimes bland. Temperature 1 samples from the distribution — creative, unpredictable. Use lower temperature (0-0.3) for factual answers, higher (0.7-1.0) for creative writing.
Why do LLMs hallucinate?
LLMs are trained to produce plausible text, not true text. They have no mechanism to verify facts. When the training data doesn't contain the answer, the model still produces a confident-sounding response. This isn't a bug that will be 'fixed' — it's inherent to how the architecture works. Fix: use RAG (retrieval-augmented generation) to ground responses in real documents.
What is a context window?
The context window is the maximum number of tokens an LLM can consider at once. Your prompt, the conversation history, and the generated response all count against it. GPT-3.5: 4,096 tokens. GPT-4: 128,000. Claude 3: 200,000. When you exceed the limit, the model either truncates the input or errors out.
What is the difference between GPT-4, Claude, and Gemini?
All three are transformers trained on next-token prediction. The differences are: training data, scale, fine-tuning, context window, and safety training. For most tasks, they perform similarly. The choice comes down to API pricing, latency, specific capabilities, and how well each handles your use case.
What is prompt engineering?
Prompt engineering is the practice of writing prompts that get better LLM outputs. Core techniques: be specific, provide context, use examples, specify format, and iterate. It's not magic — it's clear communication with a statistical model.
Can LLMs reason?
Partially. LLMs can follow logical steps and solve math problems — but they don't 'reason' the way humans do. They pattern-match against reasoning-like text in their training data. Chain-of-thought prompting (asking the model to 'think step by step') helps because it forces the model to generate reasoning-like text, which improves accuracy.