Support for private terraform modules #3723

wagnst · 2021-05-14T22:40:14Z

Please support private terraform modules hosted in registries that need authentication:

Running the latest dependabot (which now supports HCL 2 - #1176), reveals:

terraform-google-modules/project-factory/google is public, working fine
lhsystems/serviceaccount/google is a private module hosted on Terraform Enterprise

Its called like this:

module "svc" {
  source  = "app.terraform.io/lhsystems/serviceaccount/google"
  version = "0.0.28"
}

The dependabot logs show:

updater | INFO <job_134426693> Checking if terraform-google-modules/project-factory/google  needs updating
  proxy | 2021/05/14 22:26:51 [034] GET https://registry.terraform.io:443/v1/modules/terraform-google-modules/project-factory/google/versions
  proxy | 2021/05/14 22:26:51 [034] 200 https://registry.terraform.io:443/v1/modules/terraform-google-modules/project-factory/google/versions
updater | INFO <job_134426693> Latest version is 10.3.2
updater | INFO <job_134426693> Requirements to unlock update_not_possible
updater | INFO <job_134426693> Requirements update strategy 
updater | INFO <job_134426693> No update possible for terraform-google-modules/project-factory/google 
updater | INFO <job_134426693> Checking if lhsystems/serviceaccount/google  needs updating
updater | INFO <job_134426693> Latest version is 
updater | INFO <job_134426693> No update needed for lhsystems/serviceaccount/google

Dependabot needs to authenticate against the private module registry e.g. via terraform login or via the API token (TF_API_TOKEN) which can be supplied via ENV var, some example is also described here

The text was updated successfully, but these errors were encountered:

feelepxyz · 2021-05-18T09:45:29Z

@wagnst 💯 we're working on adding support right now 👍

xlgmokha · 2021-06-09T18:21:31Z

We shipped support for Terraform private registries. You can find out more in github/docs#6808, #3790, #3821, #3811, #3790, #3756.

Please give it a try and let us know what you think. https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates#terraform-registry

wagnst · 2021-06-10T12:58:34Z

@xlgmokha @feelepxyz tried it for terraform cloud, but it doesnt work. I created a secret under my personal user (added it as github dependabot org secret) - my user is owner in TFC. I get the following:

pdater | INFO <job_156796200> Starting job processing
updater | INFO <job_156796200> Starting update job for lhsystems/base-landingzone
updater | INFO <job_156796200> Checking if lhsystems/managementgroups/azurerm 0.0.4 needs updating
  proxy | 2021/06/10 12:54:06 [028] GET https://app.terraform.io:443/.well-known/terraform.json
  proxy | 2021/06/10 12:54:06 [028] 200 https://app.terraform.io:443/.well-known/terraform.json
  proxy | 2021/06/10 12:54:06 [030] GET https://app.terraform.io:443/api/registry/v1/modules/lhsystems/managementgroups/azurerm/versions
  proxy | 2021/06/10 12:54:06 [030] 401 https://app.terraform.io:443/api/registry/v1/modules/lhsystems/managementgroups/azurerm/versions
updater | I, [2021-06-10T12:54:06.606744 #8]  INFO -- sentry: ** [Raven] Sending event 62f16504eeee44afb3a3aab9ebf4630d to Sentry
  proxy | 2021/06/10 12:54:06 [032] POST https://sentry.io:443/api/1451818/store/
  proxy | 2021/06/10 12:54:07 [032] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_156796200> Error processing lhsystems/managementgroups/azurerm (Dependabot::DependabotError)
updater | ERROR <job_156796200> Response from registry was 401
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/vendor/ruby/2.6.0/gems/dependabot-terraform-0.152.0/lib/dependabot/terraform/registry_client.rb:124:in `http_get!'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/vendor/ruby/2.6.0/gems/dependabot-terraform-0.152.0/lib/dependabot/terraform/registry_client.rb:47:in `all_module_versions'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/vendor/ruby/2.6.0/gems/dependabot-terraform-0.152.0/lib/dependabot/terraform/update_checker.rb:79:in `all_module_versions'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/vendor/ruby/2.6.0/gems/dependabot-terraform-0.152.0/lib/dependabot/terraform/update_checker.rb:70:in `latest_version_for_registry_dependency'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/vendor/ruby/2.6.0/gems/dependabot-terraform-0.152.0/lib/dependabot/terraform/update_checker.rb:16:in `latest_version'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:460:in `all_versions_ignored?'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:191:in `check_and_create_pull_request'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:80:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `block in run'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `each'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `run'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_156796200> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:28:in `run'
updater | ERROR <job_156796200> bin/update_files.rb:21:in `<main>'

My dependabot.yml:

version: 2
registries:
  terraform-lhsystems: # Define access for a private registry, https://github.com/github/docs/pull/6808
    type: terraform-registry
    url: https://app.terraform.io
    token: ${{secrets.TF_API_TOKEN_DEPENDABOT}}
updates:
  - package-ecosystem: "terraform"
    directory: "/"
    schedule:
      interval: "daily"

Where secrets.TF_API_TOKEN_DEPENDABOT is the mentioned global dependabot secret.

When i personally open up the upper pages (like https://app.terraform.io:443/api/registry/v1/modules/lhsystems/managementgroups/azurerm/versions) I get a proper response from the API.

jurre · 2021-06-10T13:01:53Z

version: 2
registries:
  terraform-lhsystems: # Define access for a private registry, https://github.com/github/docs/pull/6808
    type: terraform-registry
    url: https://app.terraform.io
    token: ${{secrets.TF_API_TOKEN_DEPENDABOT}}
updates:
  - package-ecosystem: "terraform"
    directory: "/"
    registries: "*" # or terraform-lhsystems
    schedule:
      interval: "daily"

Should fix it

wagnst · 2021-06-10T13:05:36Z

awesome @jurre thanks a lot, works with that! Was not aware of that introduced option as it wasnt part of the PR ocmment of github/docs#6808

jurre · 2021-06-10T13:09:31Z

awesome @jurre thanks a lot, works with that! Was not aware of that introduced option as it wasnt part of the PR ocmment of github/docs#6808

Yeah tbh it trips up more people, it's in the docs but it can be confusing. Happy it's working now!

norman-zon · 2023-02-22T16:33:28Z

Does this only work for Terraform registries or also for modules that are directly stored in git. For example:

module "example" {
  source = "git@github.com:myOrg/terraform-modules.git//example?ref=v0.1"
...
}

and

version: 2
registries:
  terraform-git
    type: terraform-registry
    url: https://github.com/myOrg/terraform-modules.git
    token: ${{secrets.TF_API_TOKEN_DEPENDABOT}}

FelixTheodor · 2023-05-23T11:56:45Z

I have the same questions as @norman-zon ! Did anyone figure this out yet?

norman-zon · 2023-05-23T11:59:42Z

Dependabot does not seem to support this. So I use renovate for this use-case instead.

FelixTheodor · 2023-05-23T12:01:06Z

Thanks for the quick response!

jeffwidman · 2023-05-24T21:50:50Z

For modules directly stored in git, any reason why you can't use the git_submodules provider rather than the terraform provider?

ajmal-basheer-ntt · 2023-06-15T10:26:05Z

Dependabot does not seem to support this. So I use renovate for this use-case instead.

Could you share some example of how your using renovate ?

IuryAlves · 2023-07-31T11:54:42Z

Does this only work for Terraform registries or also for modules that are directly stored in git. For example:
module "example" {
  source = "git@github.com:myOrg/terraform-modules.git//example?ref=v0.1"
...
}

I can confirm that it works for modules stored in git.

Here is the configuration for it:

version: 2
updates:
  - package-ecosystem: "terraform"
    directory: "/"
    schedule:
      interval: "weekly"
    registries:
      - git-terraform-modules
registries:
  git-terraform-modules:
    type: git
    url: https://github.com
    username: x-access-token
    password: ${{ secrets.GITHUB_PAT }}

wagnst added the T: feature-request Requests for new features label May 14, 2021

wagnst mentioned this issue May 14, 2021

Terraform 0.15 support #1176

Closed

asciimike added L: terraform Terraform packages F: private-registries 💂‍♂️ Issues about using private registries with Dependabot; may be paired with an R: label. labels May 19, 2021

xlgmokha self-assigned this Jun 9, 2021

xlgmokha closed this as completed Jun 9, 2021

HorizonNet mentioned this issue Mar 13, 2023

Terraform: fix NoMethodError due to no versions in module #6828

Closed

baynezy mentioned this issue Sep 18, 2023

Support Private Terraform Registries #2112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for private terraform modules #3723

Support for private terraform modules #3723

wagnst commented May 14, 2021 •

edited

Loading

feelepxyz commented May 18, 2021

xlgmokha commented Jun 9, 2021

wagnst commented Jun 10, 2021 •

edited

Loading

jurre commented Jun 10, 2021

wagnst commented Jun 10, 2021

jurre commented Jun 10, 2021

norman-zon commented Feb 22, 2023

FelixTheodor commented May 23, 2023

norman-zon commented May 23, 2023

FelixTheodor commented May 23, 2023

jeffwidman commented May 24, 2023

ajmal-basheer-ntt commented Jun 15, 2023

IuryAlves commented Jul 31, 2023 •

edited

Loading

Support for private terraform modules #3723

Support for private terraform modules #3723

Comments

wagnst commented May 14, 2021 • edited Loading

feelepxyz commented May 18, 2021

xlgmokha commented Jun 9, 2021

wagnst commented Jun 10, 2021 • edited Loading

jurre commented Jun 10, 2021

wagnst commented Jun 10, 2021

jurre commented Jun 10, 2021

norman-zon commented Feb 22, 2023

FelixTheodor commented May 23, 2023

norman-zon commented May 23, 2023

FelixTheodor commented May 23, 2023

jeffwidman commented May 24, 2023

ajmal-basheer-ntt commented Jun 15, 2023

IuryAlves commented Jul 31, 2023 • edited Loading

wagnst commented May 14, 2021 •

edited

Loading

wagnst commented Jun 10, 2021 •

edited

Loading

IuryAlves commented Jul 31, 2023 •

edited

Loading