Neural networks.
Stacked math that learns.
A neural network is not a brain. It's a mathematical function — layers of weighted sums and nonlinear curves, stacked and trained to map inputs to outputs. That's the entire mechanism. This lesson explains what a neuron does, how backpropagation adjusts weights, why activation functions matter, and why "deep" networks beat shallow ones. No calculus. No PhD. Just the mechanism.
A neuron does four things: multiply each input by a weight, sum the results, add a bias, and pass the result through an activation function. A layer is a collection of neurons. A network is stacked layers. Training adjusts the weights and biases so the network's output matches the training data. Backpropagation is the algorithm that calculates how much each weight contributed to the error, so it can be nudged in the right direction. Repeat for thousands of examples — the network learns.
Before this lesson: complete Lesson 2 for the ML fundamentals (training, overfitting, train/test split).
🧠 The neuron — weighted sum + bias + activation
Four operations. That's the entire unit. No magic.
INPUTS WEIGHTS COMPUTATION OUTPUT ──────────────────────────────────────────────────────────────── x1 = 2.0 × w1 = 0.5 x2 = 3.0 × w2 = -1.0 → sum = (2×0.5) + (3×-1.0) x3 = 1.0 × w3 = 2.0 sum = 1.0 - 3.0 + 2.0 sum = 0.0 + bias (b = 1.5) = 1.5 → activation function → ReLU(1.5) = 1.5 → output = 1.5 A neuron is: output = activation( Σ(xi × wi) + bias ) That's it. That's the entire unit. The "intelligence" comes from stacking thousands of these and training the weights. Weights (wi) and bias (b) are the LEARNED parameters. The inputs (xi) are the DATA. The activation function is FIXED (chosen by you).
Weights
Learned numbers — one per input connection. A weight of 0 means "ignore this input." A large weight means "this input matters a lot." Training adjusts these numbers.
Bias
Another learned number added to the sum. It shifts the activation threshold — like the intercept in a line equation. Without bias, the neuron's output would always be zero when all inputs are zero.
Activation
A nonlinear function applied to the sum. Without it, stacking neurons would be pointless — the whole network would be one big linear function. ReLU, sigmoid, and tanh are the most common.
What a neuron isn't
It's not a brain cell. It doesn't "understand" or "decide." It's weighted addition plus a curve. The complexity emerges from thousands of them working together, not from any single one being smart.
A neuron is a tiny linear regression with a curve on top. If that sounds underwhelming, that's the point. Neural networks aren't magic — they're thousands of tiny functions stacked and trained. The surprise of deep learning isn't that each unit is complex. It's that stacking simple units and training them on data produces behavior that feels intelligent. Understanding this removes the mystique.
📚 Layers — the network's structure
Input, hidden, output. The three types of layer and what each does.
INPUT LAYER HIDDEN LAYER 1 HIDDEN LAYER 2 OUTPUT LAYER (3 neurons) (4 neurons) (4 neurons) (2 neurons) ─────────── ───────────── ───────────── ──────────── x1 ───────┬──→ h1 ───────┬──→ h1 ───────┬──→ y1 │ │ │ x2 ───────┼──→ h2 ───────┼──→ h2 ───────┼──→ y2 │ │ │ x3 ───────┴──→ h3 ───────┴──→ h3 ───────┘ h4 h4 Input layer: just the data — no computation Hidden layers: where the learning happens Output layer: the prediction (1 = regression, 2+ = classification) FULLY CONNECTED: every neuron in layer N connects to every neuron in layer N+1. That's the standard. For a 3-layer network with 4 neurons each: Layer 1 weights: 3 × 4 = 12 Layer 2 weights: 4 × 4 = 16 Layer 3 weights: 4 × 2 = 8 Total parameters: 36 weights + 10 biases = 46 For GPT-3: ~175 billion parameters. Same structure. Just more layers, more neurons.
Input layer
The raw data — one neuron per feature. For an image, that's one per pixel. For text, one per token embedding dimension. No computation happens here; it's just the entry point.
Hidden layers
Where the learning happens. Each layer transforms the previous layer's output into a more abstract representation. More hidden layers = more capacity to learn complex patterns = "deep" learning.
Output layer
The final prediction. One neuron for regression (predict a number). One neuron for binary classification (sigmoid output 0-1). N neurons for N-class classification (softmax output probabilities).
Fully connected
Every neuron in one layer connects to every neuron in the next. The standard structure. Specialized architectures (CNNs, RNNs, transformers) use different connection patterns — but the principle is the same.
Depth is what makes "deep" learning. A shallow network (1 hidden layer) can theoretically approximate any function — but it might need exponentially many neurons to do it. A deep network (many layers) can learn the same function with far fewer parameters because it builds hierarchical features. Edges → shapes → objects → scenes. Each layer reuses the work of the previous one. That's why deep learning beat shallow learning for images, speech, and language.
🔄 Backpropagation — how networks learn
The algorithm that adjusts weights. Two steps, repeated thousands of times.
REPEAT for each batch of training examples: ┌─────────────────────────────────────────────────┐ │ 1. FORWARD PASS │ │ Feed input through the network layer by layer│ │ Get a prediction │ │ │ │ prediction = network(input) │ │ │ │ 2. CALCULATE LOSS │ │ Compare prediction to the true label │ │ Loss = how wrong the prediction was │ │ │ │ loss = (prediction - actual)² │ │ │ │ 3. BACKWARD PASS │ │ Use calculus (chain rule) to compute │ │ how much each weight contributed to the loss │ │ │ │ gradient = ∂loss/∂weight │ │ │ │ 4. UPDATE WEIGHTS │ │ Nudge each weight in the direction that │ │ reduces the loss │ │ │ │ weight = weight - learning_rate × gradient │ │ │ └─────────────────────────────────────────────────┘ AFTER thousands of batches: The weights have converged to values that make good predictions on the training data. If the model generalizes (test accuracy is also good), training succeeded. If not (test accuracy is bad), the model overfit.
Forward pass
Data flows from input to output. Each layer applies its weights, bias, and activation. The result is a prediction. This is inference — but during training, you also remember the intermediate values for the backward pass.
Loss function
Measures how wrong the prediction was. Mean squared error for regression. Cross-entropy for classification. The entire goal of training is to minimize this number.
Backward pass
Calculus (chain rule) computes the gradient of the loss with respect to each weight. It tells you: "if you increase this weight by a tiny amount, how much does the loss change?" That's the direction to move.
Weight update
Nudge each weight in the opposite direction of its gradient. The learning rate controls the step size. Repeat thousands of times — the weights converge to values that minimize loss.
Backpropagation is just the chain rule. If you took calculus, you know it. If you didn't, here's the intuition: the loss depends on the output, which depends on the last layer, which depends on the previous layer, and so on. Backprop works backward through the chain, computing how much each weight is responsible for the final error. The "calculus" is just a systematic way of attributing blame. You don't need to derive it by hand — PyTorch and TensorFlow do it automatically.
📈 Activation functions — the nonlinearity
Without them, a neural network is just a fancy linear regression.
# ReLU (Rectified Linear Unit) — the default def relu(x): return max(0, x) Positive input passes through unchanged. Negative input becomes 0. Fast, simple, avoids vanishing gradient. Used in almost every modern network. Sigmoid — squashes to 0-1 def sigmoid(x): return 1 / (1 + math.exp(-x)) Output: 0 to 1. Used for binary classification (probability of class 1). Problem: saturates (gradient → 0 at extremes). Tanh — squashes to -1 to 1 def tanh(x): return math.tanh(x) Output: -1 to 1. Zero-centered. Better than sigmoid for hidden layers, but still saturates. Softmax — probabilities that sum to 1 def softmax(x): exp_x = [math.exp(i) for i in x] total = sum(exp_x) return [i / total for i in exp_x] Takes a vector, outputs a probability distribution. Used for multi-class classification. The outputs sum to 1 — you can read them as probabilities. THE KEY INSIGHT: Without a nonlinearity like ReLU, stacking layers does nothing. Two linear functions composed is still a linear function. Nonlinearity is what lets the network learn complex patterns.
ReLU
The default choice for hidden layers. Fast, simple, doesn't saturate for positive values. The "dead ReLU" problem (neurons stuck at 0) is rare with proper initialization. Use ReLU unless you have a reason not to.
Sigmoid
For binary classification output. Squashes to 0-1, readable as a probability. Rarely used in hidden layers now — ReLU is better. The vanishing gradient problem makes sigmoid slow in deep networks.
Tanh
Zero-centered version of sigmoid (-1 to 1). Sometimes used in hidden layers, especially in RNNs. Still saturates, so ReLU is preferred in modern CNNs and transformers.
Softmax
For multi-class output. Converts a vector of scores into probabilities that sum to 1. The highest probability is the prediction. Every classifier with more than two classes uses softmax at the output.
The universal recipe: ReLU in hidden layers, softmax or sigmoid at the output. ReLU for all hidden layers. Sigmoid for binary classification (one output neuron). Softmax for multi-class (N output neurons, one per class). Linear (no activation) for regression. That's the pattern 95% of neural networks follow. Start there, deviate only when you have a specific reason.
🏗️ Why deep networks beat shallow ones
The hierarchical feature learning that made deep learning dominant.
A shallow network (1 hidden layer) can theoretically approximate any function. So why do we use deep networks (many layers) for images, speech, and language? Because "theoretically" hides a brutal practical constraint: a shallow network might need exponentially many neurons to learn what a deep network learns with far fewer. Depth gives you efficiency through hierarchy.
Layer 1: edges
The first hidden layer of an image network learns to detect edges — vertical, horizontal, diagonal. Simple patterns. The building blocks for everything else.
Layer 2: shapes
The second layer combines edges into shapes — corners, curves, circles. Each neuron in layer 2 is looking for a specific combination of edges from layer 1.
Layer 3: objects
The third layer combines shapes into objects — eyes, wheels, letters. Each neuron in layer 3 detects a specific arrangement of shapes from layer 2.
Layer N: scenes
The final layers combine objects into scenes — a face, a car, a room. Each layer reuses the previous layer's work. That's the efficiency of depth.
Depth is a form of parameter sharing. A deep network can detect "an eye" once and reuse that detector everywhere in the image. A shallow network would need to relearn "eye detector" for every possible position. Depth lets the network compose features hierarchically — each layer builds on the previous one. That's why deep learning needs far less data than a shallow network with the same theoretical capacity. It's the same reason programmers use functions instead of copy-pasting code.
Deep networks are harder to train. More layers means more opportunities for gradients to vanish or explode during backpropagation. The 2010s breakthroughs — ReLU, batch normalization, residual connections, better initialization — were all about making deep networks trainable. Before these, adding layers made performance worse, not better. The "deep" in deep learning is only useful if you can actually train the depth.
🏆 Challenge
Three tasks. No coding required — just thinking through the mechanism.
Task 1 — Trace a neuron
A neuron has inputs [1, 2, 3], weights [0.5, -0.5, 1.0], and bias 0.5.
1. Calculate the weighted sum: (1×0.5) + (2×-0.5) + (3×1.0)
2. Add the bias
3. Apply ReLU (max(0, x))
4. What's the final output?
Then repeat with weights [-1.0, -1.0, -1.0] and the same inputs. What happens? Why is the bias important?
Task 2 — Design a network
For each problem, design the network structure:
1. Predict house price from 10 features
2. Classify an image as cat or dog (224×224 pixels)
3. Classify an email as spam, not spam, or phishing (3 classes)
4. Predict tomorrow's temperature from 7 days of history
For each one, answer:
• How many input neurons?
• How many hidden layers? How many neurons each?
• What activation function for hidden layers?
• What activation function for the output?
• What loss function?
Task 3 — Debug a network
Here are three training results. Diagnose each:
Network A: train accuracy 50%, test accuracy 50%
Network B: train accuracy 99%, test accuracy 60%
Network C: train accuracy 88%, test accuracy 86%
1. Which is underfitting?
2. Which is overfitting?
3. Which is well-fitted?
4. What would you change for each?
Hint: use the concepts from Lesson 2 — the train/test gap is the diagnostic.
⚠️ Common mistakes
The neural network bugs that catch every beginner.
Stacking linear layers without activation is mathematically identical to one linear layer. You lose all the power of depth. Always add ReLU (or another nonlinearity) after every hidden layer.
Sigmoid saturates — gradients vanish in deep networks. ReLU is the default for hidden layers for a reason. Use sigmoid only at the output for binary classification.
Weights bounce around and never converge. The loss oscillates or explodes. Fix: reduce the learning rate by 10x. Most deep learning bugs are learning-rate bugs.
Same as Lesson 2 — you must evaluate on data the model hasn't seen. Without it, you can't tell if the network learned or memorized. Always split.
Features with wildly different scales (e.g., age 0-100 vs income 0-1,000,000) make training slow and unstable. Fix: normalize all features to similar ranges (mean 0, std 1, or 0-1).
A 50-layer network on 100 examples will overfit instantly. Match the model complexity to the data size. Small data → simple model. Big data → complex model.
If the data is ordered (all class A first, then class B), the network learns the order, not the pattern. Fix: shuffle before every epoch. Every framework has a shuffle option — use it.
More epochs doesn't mean better. After a point, test accuracy starts dropping — that's overfitting. Fix: use early stopping (stop when test accuracy stops improving).
➡️ What's next
You understand the machinery. Next: how it reads text.
❓ Frequently asked questions
What is a neural network?
A neural network is a mathematical function that transforms input numbers into output numbers through layers of connected 'neurons.' Each neuron takes inputs, multiplies them by weights, adds a bias, and passes the result through an activation function. Stack enough neurons in enough layers, and the network can approximate almost any function.
What is a neuron in a neural network?
A neuron is a small computation unit. It takes several input numbers, multiplies each by a weight, sums them up, adds a bias, and passes the result through an activation function. The output goes to the next layer. A neuron is not intelligent — it's just weighted addition plus a curve.
What is backpropagation?
Backpropagation is the algorithm that trains neural networks. It works in two steps: forward pass (feed input through the network to get a prediction) and backward pass (calculate how much each weight contributed to the error, using the chain rule). Each weight is then adjusted slightly in the direction that reduces the error.
What is an activation function?
An activation function is a nonlinear function applied to each neuron's output. Without it, stacking layers would be pointless — the whole network would collapse into a single linear function. ReLU is the most common: it returns the input if positive, else 0.
What is gradient descent?
Gradient descent is the optimization algorithm that trains neural networks. It calculates the gradient (the direction of steepest error increase) and takes a small step in the opposite direction to reduce error. Repeat thousands of times, and the model converges to a minimum.
Why do we need deep networks?
Deep networks (many layers) can learn hierarchical features. The first layer might detect edges, the second combines edges into shapes, the third combines shapes into objects. Shallow networks can't do this — they'd need exponentially more neurons to learn the same thing.
What is the vanishing gradient problem?
In very deep networks, gradients can become exponentially small as they propagate backward through many layers. The early layers get almost no signal, so they don't train. This was a major obstacle to deep learning until ReLU activation and better initialization methods solved it in the 2010s.
How many layers does a neural network need?
For simple problems: 1-3 layers. For images: 10-50 layers. For LLMs: 50-100+ layers. More layers means more capacity to learn complex patterns, but also more data and compute needed to train. Start simple.