Break-Glass Access: Designing the Emergency Path Before You Need It

Why it must exist

A gateway that brokers all administrative access can also block all administrative access. Directory unreachable, MFA provider offline, certificate expired, the cluster itself down — in each case the control designed to protect production is what prevents restoring it.

Refusing to build a break-glass path does not remove that risk. It relocates it, to a 3am decision made under pressure by whoever is on call, with no controls at all.

The design goal. A path that works when everything else is broken, that nobody can use quietly, and that cannot become the normal way of working.

What the path must not depend on

Trace the dependencies honestly, then remove them:

Not the directory — a local credential, since the domain may be exactly what failed
Not a cloud MFA service — a hardware token validated locally, or a sealed physical factor
Not the production network — out-of-band management, a console server, or physical access
Not the gateway cluster — the whole point is that it may be unavailable
Not a single person — the holder will be on a flight the night you need it

The controls that keep it honest

Break-glass is not an absence of control. It is a different set, weighted toward detection and after-the-fact review rather than prevention:

Sealed and split. Credentials stored so that use is physically evident — a sealed envelope in a safe, or split across two custodians so no individual can act alone.
Loud on use. Retrieval alerts the security team, the system owner, and management simultaneously. Not an email; a page.
Recorded anyway. Where technically possible, the emergency session is still captured. An emergency is not a reason for no record.
Rotated immediately after. The credential is single-use by policy and replaced once the incident closes.
Reviewed every time. A written post-incident review covering why the normal path failed — with no exceptions, including for the CISO.

The failure mode to watch for

Break-glass procedures decay in a predictable way. It is used legitimately during a real incident. It works, and it is faster than the normal path. It gets used again for something slightly less urgent. Within a year it is the standard route for a particular team, and the alerting has been muted because it fires constantly.

The signal to watch is frequency. Break-glass should be used approximately never — a handful of times a year at most. If it is being used monthly, the normal path is failing to serve a legitimate need, and the fix is to widen the normal path, not to formalise the bypass.

Drill it

An untested emergency procedure is a document, not a capability. Exercise it at least annually:

Confirm the credential still works and the envelope is where the runbook says
Confirm the out-of-band path is reachable and its own credentials are current
Time the whole process, from decision to working session
Confirm the alerts actually fired and reached someone awake
Rotate afterwards, exactly as a real use would require

The drill routinely finds that the console server was decommissioned, the safe combination changed, or the alert address belongs to someone who left. Better to find that in an exercise.