Add support for migration history management

[minamijoyo]

In the analogy of database migration, I believe a migration history management feature will be definitely useful.
This PR allow us to set a migration file directory and history storage in the configuration file, and apply all pending unapplied migrations in sequence.

Migration files will be applied in alphabetical order by filename. You can use a serial number for filename (e.g. 123.hcl), but I recommend you to use a timestamp as a prefix to avoid git conflicts (e.g. 20201114000000_mv_bar1.hcl)

A history file serializes and saves the History object in JSON format. It also contains format version in case of format change. An example of history file as follows:

{
    "version": 1,
    "records": {
        "20201114000000_mv_bar1.hcl": {
            "type": "state",
            "name": "bar1",
            "applied_at": "2020-11-14T23:18:38.743079+09:00"
        },
        "20201114000001_mv_bar2.hcl": {
            "type": "state",
            "name": "bar2",
            "applied_at": "2020-11-14T23:21:18.976782+09:00"
        }
    }
}

The history file is stored in a Storage. Since Terraform supports multiple cloud storages to store tfstate, so we probably need to support them to store the migration history, but the Terraform's backend implementation cannot be reused as it is, so I implemented it by myself only we need with a primitive SDK. There is also an external library that abstracts cloud storage, but I didn't use it because I will probably want full control authentication options to match the behavior with terraform.
I had another option which ​​implementing a custom terraform provider for tfmigrate, but I thought it wouldn't match the provider's resource model because migration isn't idempotent and its order is important.

This PR implemented local and s3 storage. The Storage interface is simple, so it's easy to add implementations from other cloud providers. If your cloud provider is not supported for now, as a workaround, you can use local storage and synchronize it to your cloud storage with a wrapper script.

A new configuration file has been added to specify the migration directory and history storage. The configuration file path defaults to .tfmigrate.hcl. You can change the path of configuration file with command line flag --config. An example of configuration file as follows:

tfmigrate {
  migration_dir = "./tfmigrate"
  history {
    storage "local" {
      path = ".tfmigrate_history.json"
    }
  }
}

tfmigrate {
  migration_dir = "./tfmigrate"
  history {
    storage "s3" {
      bucket = "example-tfmigrate-test"
      key    = "tfmigrate/history.json"
    }
  }
}

The plan / apply command applies all unapplied migrations if the argument is omitted, but you can plan / apply only a given migration file by the argument.

$ tfmigrate apply --help
Usage: tfmigrate apply [PATH]

Apply computes a new state and pushes it to remote state.
It will fail if terraform plan detects any diffs with the new state.

Arguments
  PATH               A path of migration file
                     Required in non-history mode. Optional in history-mode.

Options:
  --config           A path to tfmigrate config file

[hermanschaaf]

Nice, this would be awesome! What is the pull request still missing?

[minamijoyo]

@hermanschaaf Hi, thank you for your interest!

A persistence mechanism of history has been almost completed and connecting it to migrator implementations and UIs should be done before merge.

As I'm working on multiple side projects in my spare time, so the progress is a bit slow, but I believe it's a definitely helpful feature for us. Stay tuned!

[minamijoyo]

There are still some things I would like to fix, such as s3 authentication options, but the diff is too big and the minimum use-case works for now, so I'll merge it and fix them in the following PRs.

[minamijoyo]

@hermanschaaf Released in v0.2.0 🚀