Key takeaway: Drift is not a discipline failure to be scolded away. It is an inevitable byproduct of humans fixing urgent problems, and it needs detection rather than prohibition.
How It Happens
Production is degraded. An engineer opens the cloud console, raises an instance size, adjusts a security group, and service recovers. Everyone is relieved.
That change now exists in reality but not in the Terraform configuration. The state file records the old value. Nothing is broken and nothing is flagged.
Weeks later an unrelated change runs terraform apply. Terraform compares configuration to state, sees the resource as it believes it should be, and reverts the emergency fix. The original incident recurs, and its cause is now buried under a change that appeared unrelated.
Three Distinct States
Confusion here causes real mistakes, so the distinction matters.
| Configuration | State file | Reality | |
|---|---|---|---|
| Healthy | Matches | Matches | Matches |
| Drift | Old value | Old value | New value |
| State staleness | New value | Old value | Old value |
| Manual resource | Absent | Absent | Exists |
terraform plan compares configuration against the state file, then refreshes against reality. Drift appears as a plan proposing to undo something. The danger is that in a large plan, one reverted attribute among forty changes is easy to approve without noticing.
Detecting It Deliberately
Run terraform plan -detailed-exitcode on a schedule — nightly is sufficient for most teams — against every workspace. Exit code 2 means changes are pending. On a workspace where nobody has committed anything, pending changes mean drift, and that should page or ticket rather than sit in a log.
This separates detection from deployment. Discovering drift during a scheduled check is routine. Discovering it inside an urgent deploy plan is how emergency fixes get reverted.
For resources genuinely managed elsewhere — autoscaling group capacity, tags applied by a governance tool — use ignore_changes in a lifecycle block so Terraform stops fighting the other system. Declaring the boundary explicitly is better than tolerating perpetual noise, because perpetual noise trains people to ignore plans.
Closing the Loop
Detection only helps if there is a defined response. When drift is found, the decision is binary: either the manual change was correct, in which case it gets codified and committed, or it was not, in which case it gets reverted deliberately with the reason recorded.
Restricting console write access to break-glass roles reduces frequency, but eliminating it entirely tends to backfire — engineers need a path to act during incidents, and removing it produces worse outcomes than drift. Accept the emergency path, then reconcile afterwards as a standard post-incident task.
The Bottom Line
Schedule automated drift detection and alert on unexpected pending changes. Use ignore_changes where another system legitimately owns an attribute, and make codifying emergency fixes an explicit step in incident follow-up.
