Reliability · Iceberg · Drills
Failure Drills for Batch Cutovers
Cutover anxiety spikes when teams skip rehearsals. We script partial failures—stuck locks, poisoned partitions, delayed upstream SaaS drops—and run them in daylight hours with observers from support.
Participants document the exact comms channel, tone, and customer-facing language for each scenario, not just the kubectl incantations.
The drills surfaced surprising gaps in our students’ runbooks: missing phone trees for vendor escalation and unclear ownership when marketing re-sends a broken extract.
We include a printable timeline you can tape above incident keyboards as a gentle mnemonic.
Back to insights