[ecs] Terraform destroy failed because it destroy cluster before scaling down asg

[pingleig]

I upgraded our ECS module https://github.com/infrablocks/terraform-aws-ecs-cluster from 3.0.0 to 4.0.0 because it is having syntax error. After that I start to get Error: Error deleting ECS cluster: ClusterContainsContainerInstancesException: The Cluster cannot be deleted while Container Instances are active or draining hashicorp/terraform-provider-aws#4852 (comment)

I didn't dive too much into the ECS module. One possible cause is in previous version we create asg after creating ECS cluster, so asg got destroyed first and there is no exception. In the new version this behaviour is reversed and we are trying to delete ECS cluster before scaling down asg.

There are a couple ways to fix it

• write our own ECS module without relying on third party modules
• use aotutil/awscli to scale down the asg before/during terraform destroy (similar as what we do for naws patching)
• downgrade my local terraform version because I remember we locked version in workflow for some reason
  • and the reason is the ECS module we were using does not work for 0.14/5 ... 😄

[pingleig]

Well, shell out to aws cli did the trick. I just need to mark aoc's ECS service to depend on this null resource. And it will scale down the asg before deleting the ECS cluster.

# This is a hack for known issue https://github.com/hashicorp/terraform-provider-aws/issues/4852
# We always create ECS cluster with active EC2 instances, so when destroy we need to scale down
# the asg so the cluster can be destroyed.
resource "null_resource" "scale_down_asg" {
  # https://discuss.hashicorp.com/t/how-to-rewrite-null-resource-with-local-exec-provisioner-when-destroy-to-prepare-for-deprecation-after-0-12-8/4580/2
  triggers = {
    asg_name = module.ecs_cluster.autoscaling_group_name
  }

  # Only run during destroy, do nothing for apply.
  provisioner "local-exec" {
    when    = destroy
    command = <<-EOT
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name "${self.triggers.asg_name}" --min-size 0 --desired-capacity 0
EOT
  }
}

[AshayFernandes]

Well, shell out to aws cli did the trick. I just need to mark aoc's ECS service to depend on this null resource. And it will scale down the asg before deleting the ECS cluster.

# This is a hack for known issue https://github.com/hashicorp/terraform-provider-aws/issues/4852
# We always create ECS cluster with active EC2 instances, so when destroy we need to scale down
# the asg so the cluster can be destroyed.
resource "null_resource" "scale_down_asg" {
  # https://discuss.hashicorp.com/t/how-to-rewrite-null-resource-with-local-exec-provisioner-when-destroy-to-prepare-for-deprecation-after-0-12-8/4580/2
  triggers = {
    asg_name = module.ecs_cluster.autoscaling_group_name
  }

  # Only run during destroy, do nothing for apply.
  provisioner "local-exec" {
    when    = destroy
    command = <<-EOT
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name "${self.triggers.asg_name}" --min-size 0 --desired-capacity 0
EOT
  }
}

@pingleig I'm facing same issue while destroying the infrastructure using capacity provider with ECS and capacity provider threshold at 75, will this solution work in this case as well?