Add a cmd type remote-state

[bzcnsh]

• delegate state locking and storage to an external command.
• call the command with one of 'GET', 'PUT', 'DELETE', 'LOCK', 'UNLOCK'.
• pass the content of state between terraform and the command through a file.
• pass the content of lock between terraform and the command through a file.
• sample scripts are at:
  https://github.com/bzcnsh/tf_remote_state_cmd_samples

[hashicorp-cla]

All committers have signed the CLA.

[danieldreier]

Hi @ggiallo28 -

I really appreciate you making an effort to contribute to Terraform, and to build a more flexible way to contribute state backends. Unfortunately, we’re not going to be able to accept this PR, and I want to explain why, and what your options are.

When we adopt new state backends in Terraform core, we take on significant responsibility, because problems with state backends significantly impact users. At the same time, it’s very difficult for us to maintain community-contributed state backends, because we have to maintain code and handle bug reports for integrations we lack domain knowledge in. We’ve tried, and it has been unsatisfactory for our users, open source contributors, and our engineers.

For example, we’ve found that subtle locking bugs are almost inevitable: even very good engineers often miss them, and when we accept community-contributed state backends, we end up responsible for testing and validation of distributed systems that we lack domain knowledge in. Our experience has been that we cannot guarantee the high quality that users need when we accept third party state backends, and so we have made the difficult decision to stop accepting new community-contributed state backends.

Until we have built a plugin system, we will not be accepting any new state backends that are not maintained by HashiCorp employees. When we thought about this, we came to very similar conclusions as you did. We are hoping to build a plugin system for state backends, similar to providers. Your proposal is very much in line with our thinking: backends should be external binaries, much like the provider system. There are two specific things we think are needed for a backend plugin system that this PR does not provide:

1. Some kind of prescriptive guard rails around locking and backend state consistency. Our experience has been that most state backends people write contain subtle bugs related to locking and consistency expectations. This part of the codebase is relatively hard to get right, and bugs in it have an extremely high impact on users. Mistakes in state management can destroy infrastructure and have significant production impact. I'm concerned that if we make it easier to contribute state backends without adding some kind of safeties, such as jepsen-style testing, this would expose users to more risk than they understand.
2. The overall user experience of getting started with Terraform is currently pretty straightforward because terraform init downloads providers. When 0.13 ships, we'll even be able to download 3rd party community-contributed providers. This ease of getting started, and the uniformity of the experience, is important. Any new system for pluggable state backends will need a way to declare those plugins in HCL, and download them, like we do with providers.

This is a tough call, because I really appreciate what you're trying to do here - it's so similar to what we want - but I think that adding a plugin system that doesn't meet these criteria, and that we ultimately replace with a different one, will cause more harm than good, and so I am closing this pull request.

If you would like to discuss this further, feel free to reach out to me at ddreier@hashicorp.com or to Petros Kolyvas, our Product Manager, at petros@hashicorp.com.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.