Context Limits in Practice

Everything in this course lives inside one constraint that hasn't come up directly yet, the context window has a fixed size, measured in tokens, not words or characters. The system prompt, the few-shot examples, every prior turn of a conversation, any retrieved document you splice in, all of it shares that one budget. Fill it up, and something has to give.

At the loosest limit the whole conversation fits, nothing gets cut. Tighten it, and the middle starts disappearing first, the system prompt stays because it's pinned, the newest turn stays because it's the one being answered right now, and the older exchanges in between get dropped one at a time until what's left fits. That's a real, common truncation strategy, not this widget's invention.

This means once a turn gets pushed out, it isn't just deprioritized, it's gone. The model has no way to attend to tokens that were never actually sent to it, the same attention mechanism from Zero to GPT can only work with what's still in the window. A long enough conversation, a big enough retrieved document, or a big enough pile of few-shot examples can quietly push earlier instructions out entirely, and the failure looks exactly like the model "forgetting" something, even though nothing about the model changed, the input just got smaller.

There's a second, more mundane reason this matters. Every token still sitting in the window costs money and processing time, on top of whatever hard limit the model enforces. Trimming a prompt down to what's actually necessary isn't only about staying under a ceiling, it's usually the cheaper and faster prompt too, which is a good habit even on a task where you're nowhere close to running out of room.