Recovery is part of the system

In February 2026, a production website was down after a partial Terraform destroy left its AWS estate in an inconsistent state.

The immediate goal sounded simple: remove what remained, rebuild the stack, and restore the site. The recovery instead crossed database protection, provider deletion semantics, network dependencies, secret lifecycles, generated credentials, container permissions, and imported application data.

There was no single root cause capable of explaining the whole outage. There was a sequence of reasonable local behaviours that had never been exercised together as one recovery path.

A destroy is an operation, not the inverse of an apply

The first obstacle was deliberate: the production database had deletion protection enabled. Terraform could not remove it until that protection changed.

After the database was removed, its final snapshot retained a reference to an option group. That prevented the option group from being deleted. Remaining security groups then blocked deletion of the VPC.

Each service was protecting or retaining something according to its own contract:

• the database resisted accidental deletion;
• the final snapshot preserved recoverability;
• the option group remained attached to that recovery artifact; and
• the VPC refused removal while dependent network objects existed.

None of those behaviours was surprising in isolation. The missing piece was an explicit teardown sequence that treated them as a dependency graph rather than assuming terraform destroy was an undo button.

The lesson is not to disable protection by default. It is to decide which protections must be deliberately crossed during recovery, who may cross them, what evidence should be retained, and which resources must be handled in which order.

Provider lifecycle semantics outlive Terraform state

The rebuild then encountered a secret scheduled for deletion. AWS Secrets Manager retains a deleted secret for a recovery window, so Terraform could not immediately create another secret under the same name.

The infrastructure definition wanted a resource that did not exist in its current state. The provider still reserved its identity.

This is a broader infrastructure-as-code problem. Provider lifecycle state does not disappear because a resource leaves Terraform state or because an operator intends to start again. Recovery procedures need to account for pending deletion, retained snapshots, asynchronous cleanup, globally or account-scoped uniqueness, and objects created outside the current run.

A clean plan is not proof of a clean provider account.

Shared integrations need one owner

The next apply failed because two Terraform modules attempted to configure the same Slack channel through AWS Chatbot while an existing environment already used it.

The modules were locally tidy: monitoring owned one notification path and the WAF module owned another. The provider imposed an account-and-channel-level constraint that cut across both module boundaries.

The permanent design response was to consolidate the Chatbot configuration under one owner and allow additional SNS topics to feed it. That change removed the competition rather than adding retries around it.

This is why module boundaries cannot be judged only by directory structure. A resource with one provider-level identity needs one lifecycle owner, even when several concerns consume it.

Generated credentials are arbitrary input

Once the infrastructure returned, WordPress could not authenticate to the database. The generated password in the instance environment did not match the value expected by RDS.

The bootstrap script used sed to replace placeholders. A generated password contained a character with meaning inside a sed replacement string, causing the placeholder itself to be inserted into the credential.

The failure was quiet. All the surrounding systems could appear correctly provisioned while the application received a corrupted value.

The permanent fix replaced shell substitution with exact string replacement in Python. The important principle is larger than the tool choice: generated credentials must be treated as arbitrary data, never as syntax for the mechanism transporting them.

Escaping a few familiar characters would only have moved the boundary. The safer operation was one whose replacement semantics did not reinterpret the value.

Infrastructure recovery is not application recovery

The site still was not restored when the compute and database were healthy. Imported uploads had filesystem ownership incompatible with the web server, and the import process had installed development URLs into production content.

Correcting file ownership addressed one layer. Updating the canonical site URL and replacing the development address across 102 stored records addressed another.

This is where an infrastructure recovery can produce a healthy stack and an unhealthy product. Compute, network, and database availability establish a platform for the application. They do not prove that media, configuration, content references, permissions, and user-visible behaviour survived.

A recovery check therefore needs both infrastructure assertions and application acceptance:

• can the service reach its database with the issued credential?
• does the public origin resolve and serve the intended site?
• do imported assets load under the production URL?
• are filesystem ownership and runtime identities correct?
• can monitoring receive signals without competing ownership?

Restore first, then remove the heroics

The production site was restored within the incident session. That outcome matters. It is not the same as saying the recovery design was complete.

At the time the incident was documented, the permanent code changes remained on a repair branch and main was untouched. The record also preserved follow-up work: a repeatable reset path, a production-safe import interface, and the eventual review and merge of the fixes.

That distinction prevents an effective incident response from becoming an inflated system claim. The operator successfully navigated the failure. The next task is to make the same knowledge executable, reviewable, and testable so the next recovery needs less improvisation.

The durable product of an incident is not the list of commands that worked once. It is a smaller set of failure modes, one owner for shared resources, bootstrap logic that treats values safely, and a recovery test that reaches the application rather than stopping at the infrastructure.