Terraform 0.15 support

[greysteil]

Dependabot's terraform support doesn't work with HCL 2.0.

In particular, our logic for parsing HCL files is here and shells out to this tool which only support HCL 1.0 and, unfortunately, looks unmaintained.

To fix this we need a HCL -> soemthing-readable-by-ruby parser to replace the one we're currently using.

[greysteil]

@hmarr would love your eyes on this if there's an easy fix. There seems to be almost no support for HCL in Ruby which is how we ended up using a Go project that converts HCL to JSON. 😢

[hmarr]

Yep, I'll sort something out 👍

[hmarr]

Upgrading to HCL 2 turns out to be surprisingly difficult. I've tried several different approaches, but none of them have worked so far. The closest I've got is using github.com/hashicorp/terraform/config as a library, but that breaks Terragrunt support. Parsing HCL 2 seems to require a full specification of the HCL's structure, and I can't find one for Terraform's HCL that includes Terragrunt. We could start maintaining one, but that would be a fair amount of work.

I'm going to park this for now as we have several other high priority issues to address, and come back to it in a bit.

[domcleal]

Parsing HCL 2 seems to require a full specification of the HCL's structure, and I can't find one for Terraform's HCL that includes Terragrunt. We could start maintaining one, but that would be a fair amount of work.

Terragrunt itself doesn't support 0.12 yet either and they're making some changes over at gruntwork-io/terragrunt#731 which may be relevant. The syntax in the current Terragrunt config file is known not to be HCL2 compatible, so they're moving it into a terragrunt.hcl file which will be HCL2 compatible.

Edit: Terragrunt 0.19 was released with these incompatible changes: migration guide

[wata727]

terraform-config-inspect might be a good replacement in the case.

[lorengordon]

@hmarr terragrunt now works with terraform 0.12. But, I think I'd suggest keeping the current parser as a fallback, if possible. That way the HCL1 syntax still works, either terraform or terragrunt.

[evenh]

Some alternatives to look into:

• https://github.com/tmccombs/hcl2json
• https://github.com/PetrusHahol/pyhcl2

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[scottybrisbane]

Bump, this is an important issue to resolve.

[johncblandii]

Can we get this worked out soon? It makes our repos noisy and renders Dependabot, a tool we really like, useless for those specific TF 0.12 repos.

[tbugfinder]

This tool is very close compare to terraform-config-inspect: https://github.com/amplify-education/python-hcl2

[jakauppila]

Until an HCL2.0 -> JSON library is available, would it be feasible to add support for Terraform 0.12 configurations that are expressed as JSON?

[sheldonhull]

Idea on possible fix: specifying an exact file to match.

• Scenario: I place all my provider versions into versions.tf
• Dependabot directory filters to only *versions.tf
• Versions submitted for bump now is only against this versions.tf without problem of parsing the remaining files.

This actual requirement to me seems like it might be the fix needed to support HCL2 without more complicated parsing requirements for an entire new HCL2 format in the remaining content in a folder.

[patrickjahns]

@sheldonhull
There is already a pull request from the community at #1388

I've taken the liberty to convert it into a github action for the time-being - hopefully this is useful for anyone else as well https://github.com/marketplace/actions/terraform-dependabot

[sheldonhull]

I tried the scenario above and it doesn't work as expects a directory, not file path.
@patrickjahns EPIC! thanks for this. I am totally checking this out for sure!
Another great case for github actions!

[asciimike]

Hey folks, I split the issue into per-quarter ecosystem updates so we were accurately tracking when the work is actually getting done, and apparently the automation didn't pick up the Q2 updates issue :(

Don't worry, both these ecosystems are on the schedule for this quarter, and I'll post the new issue when it's updated on the roadmap.

[asciimike]

Roadmap issue has been updated: github/roadmap#191

Apologies on the delay and confusion, we promise we still care about getting these ecosystems shipped!

[wagnst]

Some days ago terraform 0.15.0 was released. Will this roadmap cover that as well? Is a new issue required?

[asciimike]

Looks like yesterday: https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-15-general-availability

I think we'll have to look at it and assess how much extra effort it will be to support it, though my guess is that from where we're at (0.11 IIRC?) to 0.14 or 0.15 will be the same amount of effort (the changes in 0.12 were the thing we never implemented).

[lorengordon]

I feel like the problem is really the syntax of HCL1 vs HCL2, rather than any specific terraform versions. TF 0.11 and earlier used HCL1. TF 0.12 and later use HCL2.

EDIT: Clarifying, I just mean I would not expect there to be any problem supporting TF 0.15 any differently than TF 0.12-0.14...

[asciimike]

@lorengordon, correct. In #1176 (comment) _hmarr took a look at HCL2, and we've seen #1388 and #3063 (a few months ago) as attempts to get it working.

[jurre]

Just wanted to let y'all know that we've started working on this. If you have any (public) projects we could test against please let me know.

[lorengordon]

@jurre Here are a couple of ours you can use, where the test configs should have external modules with stale refs, and the dependabot config is already setup...

• https://github.com/plus3it/terraform-aws-tardigrade-security-group
• https://github.com/plus3it/terraform-aws-tardigrade-vpc-flow-log

[jurre]

Support for HCL2 should be available now, we're still working on support for lockfiles, so if you have those checked into version control you'll have to hold on for a bit longer.

We're also planning to add support for provider updates, something that we did not do until now.

A big thank you to @userhas404d and @jmahowald who's PR's formed the core of our implementation.

I'm going to leave this issue open until we land support for lockfiles and providers, but please open issues for other things you run into.

[lorengordon]

@jurre Confirmed it's working on those projects I linked before, thanks!

[mwarkentin]

Are there some docs on how this works / how to set it up? :)

[wagnst]

Great! now testing shows unfortunately that private registries are not supported (for modules hosted there), they don't reveal their version until authenticated. Created #3723

[asciimike]

Are there some docs on how this works / how to set it up? :)

You can just use terraform as you've always used it, we just support the new versions!

[asciimike]

Great! now testing shows unfortunately that private registries are not supported (for modules hosted there), they don't reveal their version until authenticated. Created #3723

It's on our radar (along with lockfiles and some other goodies), so stay tuned!

[mwarkentin]

@asciimike yeah I haven't used dependabot for terraform yet. Wondering if there are docs or examples over https://dependabot.com/terraform/

[asciimike]

Ah apologies, https://docs.github.com/en/code-security/supply-chain-security/enabling-and-disabling-version-updates#enabling-dependabot-version-updates is going to be the best place to start.

[ohookins]

Any intention/interest of supporting terraform-bundle files?

[jurre]

Any intention/interest of supporting terraform-bundle files?

Currently no plans to support them, and we likely won't be able to work on it the near future. We can explore it a bit, and if someone wants to contribute support for it, I'd definitely be open to discussing that.

Would you expect Dependabot to only update the bundle file, or also generate the actual bundles?

I have some questions around how we should even handle cases like this:

  google = {
    versions = ["~> 1.0", "~> 2.0"]
  }

[ohookins]

Can't comment on the multiple versions as we only use one. But my hope was that Dependabot would just update the bundle file, and then we re-run our build pipeline to regenerate the bundle.

[ohookins]

I'm looking into alternative approaches, like putting our bundle dependencies in a traditional required_providers block and then using an HCL2 parser to reform it into what terraform-bundle would read - allowing me to use Dependabot to update the versions and then trigger a build and bundle it with a little extra work. Sadly the first phase of checking for updates is failing.

updater | INFO <job_137261663> Checking if hashicorp/archive  needs updating
  proxy | 2021/05/23 11:43:07 [022] GET https://registry.terraform.io:443/v1/providers/hashicorp/archive/versions
  proxy | 2021/05/23 11:43:07 [022] 200 https://registry.terraform.io:443/v1/providers/hashicorp/archive/versions
updater | INFO <job_137261663> Latest version is 2.2.0
updater | INFO <job_137261663> Requirements to unlock update_not_possible
updater | INFO <job_137261663> Requirements update strategy 
updater | INFO <job_137261663> No update possible for hashicorp/archive 

Initially I was defining constraints using the twiddle operator but changed them all to >= which should allow it to work. Currently I have this dependency defined as:

    archive = {      
      source = "hashicorp/archive"
      version = ">= 2.1"
    }

Not sure what's going on here that is preventing the update, although I haven't dug into the logic in depth yet.

[masutaka]

Do you plan to support the terraform version?

terraform {
  required_version = "0.15.4" # <- this
  # ...
}

[asciimike]

@masutaka, we don't currently support updating the actual terraform version (nor do we support updating the package manager version for any ecosystem). While it's something we've talked about, it's not currently on our roadmap.

[masutaka]

I strongly hope dependabot will support the actual terraform version. 😄

[jurre]

Lockfile support and private registries are now available, with that, I'm going to consider this done.

Feel free to open separate feature requests for things not currently supported and we can discuss those separately.