Guardrails & Failure Modes
Every lesson in this course handed the model more room to act on its own, emit a tool call, decide to search again, plan its own steps. None of that room comes free. A model that can call tools can call the wrong one, call the right one with a hallucinated argument, or decide a task needs one more step when it actually needed zero more, and unlike a plain chat reply, an agent's mistake can execute before anyone reads it.
An injected instruction tries to make the agent forward emails and delete an audit log
Set a permission level for each tool below, then run the agent. The injected instruction is identical every time — what changes is whether the system, not the model, stops it.
send_email(to, subject, body)
delete_file(path)
In a real transcript against this exact injected email, the model itself refused to call either tool — but that refusal came from the model's training, not from anything architectural. Set both tools to "Auto-allowed" and imagine a weaker model, a cleverer injection, or just an off day: the permission system is what still has to hold.
AI Security's tool-permission-sandbox already made this exact point, defensively, from the angle of an attacker trying to get a tool call executed it shouldn't be allowed to make. Here it's the same sandbox and the same three levels, blocked, requires approval, auto-allowed, but the threat isn't a person crafting an injection anymore, it's the agent's own ordinary reasoning going wrong on its own, no attacker required. A hallucinated file path handed to delete_file does the same damage whether a person tricked the model into it or the model just guessed wrong.
This means the actual guardrail decision is the same one either way, which tools get to run automatically, which ones pause for a human first, and which ones never run without one, chosen ahead of time, not improvised after something's already gone wrong. Real agent systems layer a few more of these on top, a hard cap on how many tool calls a single loop can make before it's forced to stop, validation on a tool's arguments before it actually runs, a human checkpoint before anything irreversible. None of it prevents an agent from ever being wrong. All of it decides how much damage being wrong is allowed to do before a person sees it.
Why does the same tool-permission thinking from AI Security apply here, even with no attacker involved?