Friction Is the Point
Nothing Is More Dangerous Than Getting Away With It Once
The most dangerous moment in system design is not failure.
It is early success.
When a fragile system fails, attention increases.
When it succeeds once, confidence grows.
When it succeeds repeatedly, vigilance decays.
This is how accidents are incubated.
We see this pattern everywhere: aviation, security operations, financial controls, software deployment. When safeguards appear unnecessary, they are quietly bypassed—not by malicious intent, but by fatigue and efficiency.
AI accelerates this pattern.
When an AI system produces results that are usually good enough, people stop reviewing its work. Proofing turns into skimming. Skimming turns into habit. Habit turns into reflex.
The “approve” button becomes muscle memory.
This is not a failure of discipline. It is a failure of design.
In safety-critical domains, this problem is well understood. Pilots don’t crash planes because they ignore procedures—they crash when procedures become routine enough that deviations feel safe. Security guards don’t fail because they are careless—they fail because nothing unusual happens for long stretches of time.
Lack of negative feedback breeds complacency.
AI systems create the same conditions, but faster.
In software development, this pattern is already visible. Programmers grew tired of approving non-destructive changes, so they introduced automatic approvals. When that still felt slow, permissions expanded. Eventually, agents were given full access—and began deleting directories.
Nothing broke until it did.
The lesson is not “humans should be more careful.” That lesson has never worked.
The lesson is that systems must be designed to remain safe even when humans relax.
If safety depends on constant attention, it is already failing.
Effective systems do not rely on vigilance. They rely on hard boundaries, enforced pauses, and visible failure modes. In aviation, unsafe states are designed to be impossible or obvious. In security, silent failure is treated as unacceptable. In medicine, procedures exist not to guide good days, but to protect against bad ones.
AI requires the same thinking.
Success does not validate safety.
Repetition does not prove reliability.
Safety is proven only by how a system behaves when it is wrong.
If your AI system has no clear failure mode—no point where it stops, defers, or refuses—then its success is misleading you.
And if people are rewarded for speed while relying on vigilance to prevent harm, the outcome is not uncertain.
It is inevitable.