Lesson 1 of 15

What Is a Language Model?

You've already used a language model, even before ChatGPT existed. Every time your phone's keyboard suggests the next word as you type, it's running a tiny language model.

At its core, a language model is just a system that predicts what word (or piece of a word) comes next, given the words that came before it.

Try finishing this sentence in your head:

"The cat sat on the ___"

You probably thought of mat, floor, or couch — not airplane or Tuesday. That's because your brain has learned, from a lifetime of reading and hearing language, which words are likely to follow others.

A large language model (LLM) does the same thing, but it learned this by reading enormous amounts of text and adjusting billions of internal numbers (called parameters) until its next-word predictions got very good.

Predict before you look

Try the default phrase before touching anything else: "the cat sat on the ___." What word do you think will get the highest bar?

Those percentages are real — computed by an actual pretrained model (Google's "tiny BERT"), not hand-picked. That's exactly the shape of what a language model outputs at every single step: not one "answer," but a full probability distribution over every possible word, with the most likely ones getting the most weight.

One honest detail: this particular model's real task is filling in a masked word using context from both directions, not just predicting what comes next left-to-right (that's the GPT-style task the rest of this course builds toward). Same core idea — a probability distribution over words — just not identical mechanics. More on that difference later.

The real computation behind those bars, reproduced here in Python against the exact same real model weights:

Python

import math
from pyodide.http import pyfetch
import array

manifest_resp = await pyfetch("/data/bert-tiny-manifest.json")
manifest = await manifest_resp.json()
bin_resp = await pyfetch("/data/bert-tiny-word-embeddings.bin")
flat = array.array("f")
flat.frombytes(await bin_resp.bytes())

hidden = manifest["hidden"]
vocab = manifest["wordpieceVocab"]
vocab_index = {w: i for i, w in enumerate(vocab)}

def word_vec(word):
    i = vocab_index.get(word, vocab_index["[UNK]"])
    return list(flat[i * hidden:(i + 1) * hidden])

def add(a, b):
    return [x + y for x, y in zip(a, b)]

def layer_norm(v, weight, bias, eps=1e-12):
    mean = sum(v) / len(v)
    var = sum((x - mean) ** 2 for x in v) / len(v)
    denom = math.sqrt(var + eps)
    return [((x - mean) / denom) * w + b for x, w, b in zip(v, weight, bias)]

def linear(v, weight, bias):
    return [sum(w * x for w, x in zip(row, v)) + b for row, b in zip(weight, bias)]

def gelu(v):
    return [0.5 * x * (1 + math.erf(x / math.sqrt(2))) for x in v]

def softmax(scores):
    m = max(scores)
    exps = [math.exp(s - m) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

# "the cat sat on the [MASK]" — embed every token
tokens = ["the", "cat", "sat", "on", "the", "[MASK]"]
hidden_states = []
for i, tok in enumerate(tokens):
    summed = add(add(word_vec(tok), manifest["positionEmbeddings"][i]), manifest["tokenTypeEmbedding"])
    hidden_states.append(layer_norm(summed, manifest["embeddingsLayerNorm"]["weight"], manifest["embeddingsLayerNorm"]["bias"]))

# run both real transformer layers
num_heads, head_dim = manifest["numHeads"], manifest["headDim"]
for layer in manifest["layers"]:
    attn = layer["attention"]
    Q = [linear(h, attn["query"]["weight"], attn["query"]["bias"]) for h in hidden_states]
    K = [linear(h, attn["key"]["weight"], attn["key"]["bias"]) for h in hidden_states]
    V = [linear(h, attn["value"]["weight"], attn["value"]["bias"]) for h in hidden_states]

concat = [[0.0] * hidden for _ in hidden_states]
    for h in range(num_heads):
        s = slice(h * head_dim, (h + 1) * head_dim)
        for i, qi in enumerate(Q):
            scores = [sum(a * b for a, b in zip(qi[s], k[s])) / math.sqrt(head_dim) for k in K]
            weights = softmax(scores)
            for j, vj in enumerate(V):
                for d in range(head_dim):
                    concat[i][h * head_dim + d] += weights[j] * vj[s][d]

after_attn = []
    for v, h in zip(concat, hidden_states):
        projected = linear(v, attn["outputDense"]["weight"], attn["outputDense"]["bias"])
        after_attn.append(layer_norm(add(projected, h), attn["outputLayerNorm"]["weight"], attn["outputLayerNorm"]["bias"]))

ffn = layer["feedForward"]
    hidden_states = []
    for h in after_attn:
        inter = gelu(linear(h, ffn["intermediateDense"]["weight"], ffn["intermediateDense"]["bias"]))
        projected = linear(inter, ffn["outputDense"]["weight"], ffn["outputDense"]["bias"])
        hidden_states.append(layer_norm(add(projected, h), ffn["outputLayerNorm"]["weight"], ffn["outputLayerNorm"]["bias"]))

# real MLM head on the [MASK] position, decoded against the full real vocab
mlm = manifest["mlmHead"]
mask_hidden = hidden_states[-1]
transformed = layer_norm(
    gelu(linear(mask_hidden, mlm["transformDense"]["weight"], mlm["transformDense"]["bias"])),
    mlm["transformLayerNorm"]["weight"], mlm["transformLayerNorm"]["bias"],
)
logits = [sum(t * e for t, e in zip(transformed, word_vec(vocab[v]))) + mlm["decoderBias"][v] for v in range(len(vocab))]
probs = softmax(logits)

ranked = sorted(
    [(vocab[v], probs[v]) for v in range(len(vocab)) if vocab[v].isalpha() and vocab[v].islower()],
    key=lambda x: -x[1],
)
for word, p in ranked[:5]:
    print(f"{word}: {p:.1%}")

The word "large" in LLM refers to two things:

The amount of text it learned from — often a large fraction of the public internet, books, and code.
The number of parameters — the internal numbers the model tunes during training. Modern LLMs have anywhere from millions to hundreds of billions of them.

Everything else in this course is really just answering one question: how does a model turn text into predictions, and how does it learn to do that well? Before we touch language at all, we need the machinery behind that second question — how a model adjusts its parameters. That starts with derivatives.

Go deeper

But what is a neural network? | Deep learning, chapter 13Blue1Brown

A visual introduction to neurons, layers, and weights using handwritten digit recognition — the same intuition this course builds from scratch in code.

What is the fundamental task a language model is trained to do?

← Syllabus Derivatives: How Change Flows →