LLM Basics

Course

Neural Networks from Scratch

Take the tiny autograd engine from Zero to GPT and build a real, trainable multi-layer perceptron with it — initialization, activations, backprop, gradient checking, every major loss function, batching, and generalization, ending with a 3-class spiral capstone that puts every idea to work at once.

Lesson 1

A Single Neuron

The atomic unit of every neural network: a weighted sum and a squashing function.

Start

Lesson 2

A Layer of Neurons

Several neurons, same inputs, different weights — a vector out instead of a number.

Start

Lesson 3

Stacking into an MLP

Chain layers together and depth starts doing real work.

Start

Module: Setting Up a Network

Lesson 4

Starting From Somewhere Sensible

Before a network learns anything, its starting weights already decide whether it can.

Start

Lesson 5

Choosing a Nonlinearity

tanh isn't the only option, and the choice shapes how well gradients survive.

Start

Module: Backpropagation & Gradient Flow

Lesson 6

Backpropagation: The Chain Rule, by Hand

How gradients flow backward through a chain of operations, the algorithm that trains every neural network.

Start

Lesson 7

Vanishing and Exploding Gradients

The chain rule multiplies. Chain enough small numbers and you get zero; enough large ones and you get infinity.

Start

Lesson 8

Gradient Checking

How do you know your hand-derived backward pass is actually correct?

Start

Module: Choosing a Loss Function

Lesson 9

A Loss Function for Classification

Turning 'right or wrong' into a number the network can take a derivative of.

Start

Lesson 10

A Loss Function for Regression

Not every target is a category. Sometimes the network needs to predict a number.

Start

Lesson 11

Multi-Class Classification: Softmax and Cross-Entropy

Generalizing 'confidently correct = low loss' from one output neuron to K classes at once.

Start

Lesson 12

Training on a 2D Toy Dataset

Forward pass, loss, backward pass, update — watch a real decision boundary form.

Start

Lesson 13

Mini-Batches: Trading Off Noise and Speed

Every training loop so far looked at the whole dataset before updating. Real ones rarely can.

Start

Module: Generalization

Lesson 14

Overfitting and Regularization

More capacity can fit noise as easily as signal — unless you tax complexity.

Start

Lesson 15

Dropout and Early Stopping

Two more ways to fight overfitting: randomly silencing neurons, and just stopping in time.

Start

Lesson 16

Capstone: A Multi-Class Spiral

Softmax, batching, L2, and dropout, all at once, on a problem a straight line can't touch.

Start