Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new flag --out for saving a plan file after dry-run migrations #37

Merged
merged 2 commits into from
Aug 12, 2021

Conversation

minamijoyo
Copy link
Owner

@minamijoyo minamijoyo commented Jul 28, 2021

Fixes #36

Add a new flag --out for saving a plan file after dry-run migrations.
It almost equivalents to terraform plan -out, but runs after dry-run migrations. Note that the new flag is --out, not -out. This is only because the pflag library we use follows the GNU style long options.

This is intuitively simple, but after implemented, it turned out it wasn't a trivial matter. The plan file internally contains terraform configurations used by plan. The tfmigrate uses a temporary local backend configuration not to change a remote state in plan phase. This means applying the plan file affects only a local state.

We need the following steps to apply the plan:

Given a migration file with force attribute.
(1) tfmigrate plan --out=foo.tfplan tfmigrate_test.hcl
(2) tfmigrate apply tfmigrate_test.hcl
(3) terraform apply foo.tfplan
(4) terraform state push -force terraform.tfstate
(5) terraform plan -detailed-exitcode
(6) rm terraform.tfstate foo.tfplan

Make sure to force push the local state to remote after terraform apply. The -force flag is required in (4) because the lineage of the state will be changed. As you know, this is unsafe operation. But I couldn't find a safe way to do this in the current technical limitations. You can confirm the final plan has no changes in (5).

There are some limitations:

  1. terraform show foo.tfplan rejects it as a stale plan. Option to not error when showing a stale Terraform plan hashicorp/terraform#28136
  2. terraform state push -force terraform.tfstate doesn't work in Terraform >= v0.12.25 and < v0.13 Terraform remote state push -force fails lineage check for S3 backend hashicorp/terraform#25761

@minamijoyo minamijoyo force-pushed the store-tfplan branch 3 times, most recently from 6a19d01 to 8761dc0 Compare August 8, 2021 15:28
@minamijoyo minamijoyo changed the title [WIP] Add a new flag tfmigrate plan --out=foo.tfplan Add a new flag --out for saving a plan file after dry-run migrations Aug 10, 2021
Fixes #36

Add a new flag `--out` for saving a plan file after dry-run migrations.
It almost equivalents to `terraform plan -out`, but runs after dry-run
migrations. Note that the new flag is `--out`, not `-out`. This is only
because the pflag library we use follows the GNU style long options.

This is intuitively simple, but after implemented, it turned out it
wasn't a trivial matter. The plan file internally contains terraform
configurations used by plan. The tfmigrate uses a temporary local
backend configuration not to change a remote state in plan phase.
This means applying the plan file affects only a local state.

We need the following steps to apply the plan:

(1) tfmigrate plan --out=foo.tfplan tfmigrate_test.hcl
(2) tfmigrate apply tfmigrate_test.hcl
(3) terraform apply foo.tfplan
(4) terraform state push -force terraform.tfstate
(5) terraform plan -detailed-exitcode
(6) rm terraform.tfstate foo.tfplan

Make sure to force push the local state to remote after `terraform apply`.
The `-force` flag is required in (4) because the lineage of the state
will be changed. As you know, this is unsafe operation.
But I couldn't find a safe way to do this in the current technical limitations.
You can confirm the final plan has no changes in (5).
Terraform >= v0.12.25 and < v0.13 has a bug for state push -force.
hashicorp/terraform#25761
I added a version check in acceptance tests to minimize the patch.
Ideally, the version check process should be implemented in TerraformCLI
instead of the test helper function because supported features may be
different for some Terraform versions. I'll follow up a separated PR to
add some version checks in runtime.
@minamijoyo minamijoyo merged commit bc27b42 into master Aug 12, 2021
@minamijoyo minamijoyo deleted the store-tfplan branch August 12, 2021 00:35
minamijoyo added a commit that referenced this pull request Dec 10, 2021
Closes #62

The tfmigrate plan --out=tfplan option was originally requested in #36,
and added in #37.

While testing Terraform 1.1 support, I found it no longer work with
Terraform 1.1. #62

After debugging, the tfmigrate plan --out=tfplan option was based on a
bug prior to Terraform 1.1.

Since terraform state push increments the serial of tfstate, a saved
plan file in tfmigrate plan phase should not be able to terraform apply.
However, prior to Terraform 1.1, there was no proper validation and
terraform apply allows a such case incorrectly. Starting from Terraform
1.1, it now rejects the plan as stale, which seems to be a correct
behavior.

That is, the tfmigrate plan --out=tfplan option doesn't work with
Terraform 1.1 or later. There is no way to do this.

Fortunately, Terraform 1.1 added a new `moved` block feature, so some
use-cases could be covered by the `moved` block.

So, I decided to deprecate the tfmigrate plan --out=tfplan option
without replacement and it will be removed in a future release.
minamijoyo added a commit that referenced this pull request Dec 10, 2021
Closes #62

The tfmigrate plan --out=tfplan option was originally requested in #36,
and added in #37.

While testing Terraform 1.1 support, I found it no longer work with
Terraform 1.1. #62

After debugging, the tfmigrate plan --out=tfplan option was based on a
bug prior to Terraform 1.1.

Since terraform state push increments the serial of tfstate, a saved
plan file in tfmigrate plan phase should not be able to terraform apply.
However, prior to Terraform 1.1, there was no proper validation and
terraform apply allows a such case incorrectly. Starting from Terraform
1.1, it now rejects the plan as stale, which seems to be a correct
behavior.

That is, the tfmigrate plan --out=tfplan option doesn't work with
Terraform 1.1 or later. There is no way to do this.

Fortunately, Terraform 1.1 added a new `moved` block feature, so some
use-cases could be covered by the `moved` block.

So, I decided to deprecate the tfmigrate plan --out=tfplan option
without replacement and it will be removed in a future release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request keeping temp plan output
1 participant