NLP basics.
How machines read text.
Text is meaningless to a computer. It only sees numbers. So how do ChatGPT, Claude, and Google Translate actually process language? They solve three problems: turning text into numbers (tokenization), giving those numbers meaning (embeddings), and understanding the relationships between them (attention). Those three ideas power every modern NLP system. This lesson explains all three — no math PhD required.
Three operations transform text into understanding: tokenization splits text into pieces (tokens). Embeddings convert each token into a vector of numbers that captures meaning — similar words have similar vectors. Attention lets the model weigh which tokens matter most for the current context. Together, they're the foundation of every LLM — from BERT to GPT-4. No other mechanism in modern AI has had more impact.
Before this lesson: complete Lesson 3. You need to understand neurons and layers before you can understand embeddings and attention.
🔤 Tokenization — split text into pieces
Words aren't tokens. Tokens are pieces. The difference is crucial.
TEXT:
"The quick brown fox jumps over the lazy dog"
NAIVE TOKENIZATION (split by space):
["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]
→ 9 tokens
SUBTLE PROBLEM: What about "don't" or "jumps" or "unhappiness"?
Splitting by space fails on:
• Punctuation ("don't" → "don", "t"?)
• Compound words ("firefly" vs "fire fly"?)
• Rare words ("antidisestablishmentarianism")
MODERN SUBWORD TOKENIZATION:
Rare words get split into common pieces.
"unhappiness" → ["un", "happi", "ness"] (3 tokens)
"running" → ["run", "ning"] (2 tokens)
"the" → ["the"] (1 token — very common)
This means:
✓ The vocabulary stays small (typically 50,000 tokens)
✓ Rare words are still representable (as combinations)
✓ The model can handle new words it's never seen
REAL EXAMPLES (GPT-4 tokenizer):
"Hello, world!" → 4 tokens
"antidisestablishment" → 6 tokens
"😂" → 2 tokens
" ChatGPT" → 2 tokens
"supercalifragilistic" → 8 tokens
WHY THIS MATTERS:
• Every model has a max token limit (GPT-4: ~8,000-128,000)
• Pricing is per token, not per word
• Long documents need to be chunked into token-sized pieces
• "1 word ≈ 0.75 tokens" is the rule of thumb (English)
Token ≠ word
A token is a piece of text — usually a word, part of a word, or punctuation. "Running" might be one token or two. "The" is always one. Counting tokens is not the same as counting words.
Subword tokenization
Rare words are split into common pieces. "Unhappiness" → "un" + "happi" + "ness". This lets the model handle words it's never seen — it just needs to know the pieces.
Model-specific
Every model has its own tokenizer. GPT-4's tokenizer is different from Claude's and different from Llama's. You can't use one model's tokens with another model.
Token limits
Every model has a context window — max tokens it can process at once. GPT-3.5: 4,096. GPT-4: 128,000. Claude 3: 200,000. The limit includes input AND output.
One word ≈ 0.75 tokens is the rule of thumb for English. So a 1,000-word article is about 750 tokens. A 10,000-word document is about 7,500 tokens. This math matters when you're working with token limits or API pricing — most LLM APIs charge per token, and the input+output combined count against your quota.
🧭 Embeddings — give tokens meaning
Words become vectors. Similar words become close vectors. That's how machines "understand."
EACH TOKEN BECOMES A VECTOR (typically 100-4096 numbers):
"cat" → [0.2, -0.5, 0.8, ..., 0.1] (768 numbers)
"dog" → [0.3, -0.4, 0.7, ..., 0.1]
"kitten" → [0.25, -0.45, 0.75, ..., 0.15]
"car" → [-0.6, 0.9, -0.2, ..., 0.5]
"truck" → [-0.5, 0.85, -0.15, ..., 0.45]
THE MAGIC: SIMILAR WORDS HAVE SIMILAR VECTORS
Distance("cat", "dog") = small (0.3)
Distance("cat", "kitten") = smaller (0.2) ← closer
Distance("cat", "car") = large (2.1) ← unrelated
You can measure meaning with arithmetic:
cat is closer to kitten than to car.
The model learns this without being told what
cats and dogs are. It learns from patterns in text:
"cat" appears in similar sentences as "kitten."
THE FAMOUS EXAMPLE:
king - man + woman ≈ queen
Because the "gender" direction in vector space
is consistent, and "royalty" is consistent:
king_vector - man_vector = [royalty direction]
queen_vector - woman_vector = [royalty direction]
So: king - man + woman ≈ queen. It really works.
WHY THIS MATTERS:
Every NLP task — sentiment, translation, search, chat —
starts with embeddings. Once text is vectors, you can
do arithmetic, compare, cluster, and feed into
neural networks.
Vector representation
Each token becomes a list of numbers — usually 100 to 4096 of them. "Cat" might be [0.2, -0.5, 0.8, ...]. The whole vector captures the word's meaning.
Similarity = distance
Similar words have close vectors. You can measure semantic similarity with cosine distance. "Cat" is closer to "kitten" than to "car" — even though the letters are more similar to "car."
Arithmetic on meaning
king − man + woman ≈ queen. This works because the "gender" direction is consistent across the vector space. Embeddings capture relationships, not just words.
Contextual embeddings
Modern models don't use fixed embeddings — the same word gets a different vector depending on context. "Bank" in "river bank" vs "money bank" have different vectors. This is why transformers are so powerful.
Embeddings inherit the biases of their training data. If the training text contains "doctor" more often with male pronouns and "nurse" with female pronouns, the embeddings will position them that way. The model doesn't know this is wrong — it just captures statistical patterns. This is the root of most AI bias issues. Any production system using embeddings should test for and mitigate bias.
👁️ Attention — the breakthrough
The idea that turned NLP from a sequence problem into a parallel one. And made LLMs possible.
Before attention, NLP models read text one word at a time, like a person with a single finger scanning a page. The problem: they forgot earlier context by the time they reached the end. Attention changed this. It lets the model look at all parts of the input at once, and weigh which parts matter most for the current token. This parallel processing is what made training 100x faster — and what made GPT possible.
SENTENCE: "The cat sat on the mat, and it was happy." QUESTION: What does "it" refer to? WITHOUT ATTENTION (older models): Reads left to right. By the time it reaches "it," it has "forgotten" what "cat" was. It guesses. Often wrong. WITH ATTENTION: Every token can "look at" every other token. For the token "it", the attention mechanism computes how much each other token matters. Attention weights for "it": "The" → 0.02 "cat" → 0.72 ← strongest weight "sat" → 0.05 "on" → 0.03 "the" → 0.01 "mat" → 0.14 "and" → 0.02 "was" → 0.01 The model "knows" that "it" refers to "cat" because the attention weight for "cat" is highest. HOW IT'S COMPUTED (simplified): Each token produces three vectors: Q, K, V Q = query (what am I looking for?) K = key (what do I contain?) V = value (what do I pass forward?) Attention weight = how well each Q matches each K Output = weighted sum of V vectors All tokens process in parallel — no more "forgetting." This is the breakthrough that made GPT possible. MULTI-HEAD ATTENTION: The model runs several attention mechanisms in parallel, each learning a different type of relationship: • Head 1: grammatical subjects • Head 2: pronouns to referents • Head 3: nearby words • Head 4: long-range dependencies GPT-3 has 96 attention heads per layer. GPT-4 has even more.
Query, Key, Value
Each token produces three vectors. Query = what it's looking for. Key = what it contains. Value = what it passes on. Attention scores how well each query matches each key.
Parallel processing
Unlike RNNs (which process sequentially), attention processes all tokens in parallel. This is 100x faster on GPUs — and the single biggest reason modern LLMs became possible.
Multi-head
Multiple attention mechanisms run in parallel, each learning a different relationship (pronouns, grammar, long-range deps). More heads = more ways to look at context. GPT-3: 96 heads per layer.
Context resolution
Attention is why LLMs can disambiguate. "Bank" in a river context vs a money context gets different attention patterns. The model "understands" through context — not through definitions.
The paper that changed everything: "Attention Is All You Need" (2017). It introduced the transformer architecture, removed the sequential processing of RNNs, and enabled parallel training. Every LLM you've used — ChatGPT, Claude, Gemini, Llama — is a direct descendant of that one paper. Reading it is optional; understanding attention is essential. It's the single most important idea in modern AI.
🔄 Transformers — the architecture
Stack attention + feedforward. Repeat 96 times. That's GPT-3.
ONE TRANSFORMER LAYER: Input tokens (embeddings) ↓ ┌─────────────────────────────────────────┐ │ Multi-head self-attention │ ← which tokens matter to each other? │ (each token looks at all others) │ └─────────────────────────────────────────┘ ↓ Add & normalize (residual connection) ↓ ┌─────────────────────────────────────────┐ │ Feedforward network │ ← transform each token individually │ (2 linear layers + ReLU) │ └─────────────────────────────────────────┘ ↓ Add & normalize (residual connection) ↓ Output (same shape as input) ↓ NEXT LAYER A FULL MODEL: Input text ↓ Tokenization → [101, 8945, 2341, ...] ↓ Embedding layer → vectors ↓ ┌─────────────────┐ │ Transformer │ │ Layer 1 │ ├─────────────────┤ │ Transformer │ │ Layer 2 │ ├─────────────────┤ │ ... │ │ (N times) │ ├─────────────────┤ │ Transformer │ │ Layer N │ └─────────────────┘ ↓ Final feedforward layer ↓ Output probabilities over vocabulary ↓ Pick the most likely next token SCALE: GPT-1 (2018): 12 layers, 117M parameters GPT-2 (2019): 48 layers, 1.5B parameters GPT-3 (2020): 96 layers, 175B parameters GPT-4 (2023): ~120 layers, ~1.7T parameters Same architecture. Only the scale changed. That's the entire story of modern AI progress.
The layer
Each layer does two things: multi-head attention (context), then a feedforward network (transformation). Residual connections and layer norm stabilize training. That's the entire pattern.
Stacked N times
GPT-3 stacks 96 of these layers. Each layer refines the representation. By the final layer, the model "knows" what the next token should be.
Next-token prediction
After the final layer, the model produces a probability for every token in the vocabulary. The highest-probability token is the output. That's all "generation" is — predicting next tokens repeatedly.
Scale is everything
GPT-1 → GPT-4 uses the same architecture. Only the size changed. More layers, more parameters, more training data. The capability emerged from scale, not from architectural breakthroughs.
"Predict the next token" is not the same as "understand." A transformer generates text by predicting the most likely next token over and over. It doesn't have a model of the world, a goal, or beliefs. It has a very good statistical model of what text looks like. This is why LLMs "hallucinate" — they generate plausible-sounding text without knowing if it's true. Understanding this mechanism is the key to using AI effectively.
🏆 Challenge
Three tasks. Each exercises a different NLP concept.
Task 1 — Tokenization
Imagine a tokenizer that splits text into words and common subwords. How would it tokenize these?
1. "unbelievable"
2. "ChatGPT"
3. "running"
4. "The dog's toy"
5. "😀👍🏽"
For each, estimate the number of tokens. Then estimate the tokens for a 500-word article. Remember: ~0.75 tokens per word is the rule of thumb.
Task 2 — Embedding reasoning
Which pair of words is closest in embedding space? Which is furthest?
1. (dog, puppy) vs (dog, cat)
2. (doctor, hospital) vs (doctor, firefighter)
3. (happy, joyful) vs (happy, sad)
4. (Paris, France) vs (Paris, Texas)
Explain why each pair is close or far. Then answer: what does "king - man + woman ≈ queen" tell us about how embeddings work? Why is it not just a coincidence?
Task 3 — Attention analysis
For each sentence, predict which word "it" should attend to most:
1. "The cat chased the mouse, and it was fast."
2. "The cat chased the mouse, and it was frightened."
3. "The bank was closed, so it was quiet."
4. "She tried to solve the puzzle, but it was too hard."
For each sentence, "it" refers to something different based on context. Attention is what lets the model resolve this. If you were training a model, how would you know it's making the right prediction?
⚠️ Common mistakes
The NLP misconceptions that catch every beginner.
They're not. "The" is one token. "Antidisestablishmentarianism" is 6 tokens. Counting words for context limits or pricing is almost always wrong. Use a tokenizer.
Modern models use contextual embeddings — the same word gets different vectors in different contexts. "Bank" is different in "river bank" vs "money bank." This is why transformers work.
Attention lets the model weigh context. It's not comprehension. The model is still predicting tokens based on statistical patterns, not "understanding" the sentence. This matters for evaluating AI claims.
A 175B model isn't always better than a 7B one for a specific task. Fine-tuning a small model on task-specific data often beats prompting a huge one. Scale matters — but it's not the only lever.
Every model has a maximum context window. GPT-3.5: 4,096 tokens. GPT-4: 128,000. Claude 3: 200,000. Feed it more and it fails silently or truncates. Know the limit before you build.
Whitespace, HTML tags, special characters, encoding issues — messy data feeds into messy models. Preprocessing is 80% of applied NLP work. Don't skip it.
Different embedding models produce different vectors. You can't compare embeddings from Model A with embeddings from Model B. Pick one and stick with it, or re-embed everything when you change.
Embeddings inherit the biases of their training data. If the training text is sexist, racist, or otherwise skewed, the embeddings will reflect that. Any production system needs bias testing.
➡️ What's next
You understand how machines read text. Next: how they generate it.
❓ Frequently asked questions
What is NLP?
Natural Language Processing (NLP) is the field of making machines work with human language — text and speech. It covers everything from spam detection and sentiment analysis to translation, summarization, and chatbots. Modern NLP is dominated by deep learning, specifically the transformer architecture.
What is tokenization?
Tokenization is the process of splitting text into pieces that the model can process. A 'token' is usually a word, part of a word, or a punctuation mark. Modern models use 'subword tokenization' — common words are one token, but rare words get split into pieces.
What is a word embedding?
An embedding is a way to represent text as a vector of numbers — usually 100 to 4096 numbers per token. Similar words have similar vectors. 'Cat' and 'kitten' are close together in embedding space. Embeddings let machines 'understand' meaning by measuring distances between vectors.
How does 'king - man + woman = queen' work in embeddings?
It's arithmetic on the vectors. Each word is a point in a high-dimensional space. The direction from 'man' to 'woman' is roughly the same as the direction from 'king' to 'queen'. This works because the training process positioned similar concepts near each other.
What is the attention mechanism?
Attention lets a model look at all parts of the input at once, and decide which parts are relevant to each output. For the word 'bank' in 'I sat by the river bank,' attention weighs the word 'river' heavily, giving the model the context to disambiguate.
What is a transformer?
A transformer is a neural network architecture built entirely on attention. Introduced in the 2017 paper 'Attention Is All You Need,' it replaced the older RNNs and LSTMs. Every modern LLM — GPT, Claude, Gemini, Llama — is a transformer.
What is the difference between NLP and LLMs?
NLP is the field. LLMs are one type of NLP model — the ones trained on massive text data to predict the next token. LLMs are a subset of NLP, just like deep learning is a subset of ML.
Do I need to understand transformers to use AI?
No. You can use ChatGPT, Claude, and other AI products without understanding the mechanism. But understanding attention, tokens, and embeddings gives you the ability to predict when the model will fail, write better prompts, and evaluate claims about AI capability.