Resolving broken deleted state in terraform

Question

When terraform tries to deploy something and then times out in a state like pending or deleting the state will eventually update to successful or deleted but this never gets updated in the tf state so when I try to run something again it errors because the state doesn't match.

Error: error waiting for EC2 Transit Gateway VPC Attachment (tgw-attach-xxxxxxxxx) deletion: unexpected state 'failed', wanted target 'deleted'. last error: %!s(<nil>)

What is the correct way to handle this? Can I do something within terraform to get it to recognise the latest state in AWS? Is it a bug on tf's part?

question from:https://stackoverflow.com/questions/65884743/resolving-broken-deleted-state-in-terraform

Answer 1

tl; dr

It's probably less of a bug and more of a design choice.

You should investigate and if appropriate (e.g. the resource was created or deleted successfully and the state was not updated appropriately), you could either

• run terraform refresh, which will cause Terraform to refresh its state file against what actually exists with the cloud provider
• manually reconcile the situation by manipulating the Terraform state with the terraform state command, removing deleted resources or adding created resources

Detail

Unlike CloudFormation, Terraform's approach to 'failures' is to just drop everything and error out, leaving the operator to investigate the issue and attempt to resolve it themselves. As a result, operations which timeout are classed as failures and so the relevant resources are often not updated in Terraform's state.

Terraform does give us some recourse to handle this however. For one, we can manually manipulate Terraform's state file. We can add resources or remove resources from the state file as we like, though this should be done with caution.

We can also ask Terraform to 'refresh' its state, basically comparing the state file to reality. Implicitly this should remove resources which no longer exist, but it will not adopt resources into the state file which were provisioned outside of a successful Terraform run.

As an aside, timeouts relating to the interaction with any service provider, are a feature of the relevant Terraform Provider, in this case the AWS Provider. Only the Providers can expose configurable timeouts. For example, the AzureRM Provider does provide a means to configure timeouts, but it appears the AWS Provider does not.

Efforts are presumably made to incorporate sensible timeout values, but it's not unusual to see trivial operations take an age to complete properly.