Sometimes, a change cannot be undone safely. When you don't have the option to revert, redeploy or rollback, how can you handle failed deployment quickly and safely?
It later occurred to me that some changes, which may not be revertible through code, can still recoverable at other layers. For example, depending on the kind of DB & amount of data, it might be possible to make taking DB snapshots a precondition of moving forward with risky (or perhaps all) DDL operations. It wouldn't eliminate all cases of of data loss, but could minimize the recovery effort if one of the other mitigations aren't possible.
Great article, Tom!
It later occurred to me that some changes, which may not be revertible through code, can still recoverable at other layers. For example, depending on the kind of DB & amount of data, it might be possible to make taking DB snapshots a precondition of moving forward with risky (or perhaps all) DDL operations. It wouldn't eliminate all cases of of data loss, but could minimize the recovery effort if one of the other mitigations aren't possible.
Love that! Maybe snapshots could be coupled with a replay log of some kind too.