terraform init -from-module does not work with -get=false

[brikis98]

Terraform Version

$ terraform --version
Terraform v0.11.7

Terraform Configuration Files

In module/main.tf:

 module "nested" {
   source = "./nested-module"
 }

In module/nested-module/main.tf:

output "foo" {
  value = "foo"
}

Expected Behavior

You can run terraform init -from-module module -get=false without error.

Actual Behavior

$ terraform init -from-module module -get=false
Copying configuration from "module"...

Error: Error loading modules: module foo: not found, may need to run 'terraform init'

Steps to Reproduce

terraform init -from-module module -get=false

Additional Context

We use -from-module in Terragrunt to download remote Terragrunt configurations and the inability to disable -get causes Terragrunt to be slower. See gruntwork-io/terragrunt#523 for context.

Note that this only happens if the path I pass to -from-module has .tf files in the root folder; if the .tf files are in a subfolder (e.g., I set -from-module to the root of some Git repo), it works fine. I'm guessing the init command is trying to do some sort of extra validation on the code, which is impractical if -get=false is set. Is there any way to disable this validation?

[apparentlymart]

Hi @brikis98!

That -from-module thing is implemented as a funny sort of "get", so I expect these two arguments together are causing some sort of conflict in the terraform init logic here.

It sounds like what you're trying to do here is use terraform init -from-module essentially just as a thin wrapper around the go-getter and registry functionality to retrieve a single codebase, which isn't really how this option was designed to behave.

We may be able to do something here in future, but I expect it would require some significant refactoring, particularly in our new codepath (in the 0.12 development branch) where it literally is just running the module installation function with some overridden paths.

To better support this sort of wrapper program, probably what we ought to do is make each of the different operations terraform init does available as a separate plumbing command, so that a wrapper program can more directly control what happens in what order while terraform init remains the "easy case" for users directly running Terraform.

Unfortunately with our current focus on the configuration language for the next release I expect we won't be able to do much here in the short term, but we'll have another look at it once the dust has settled on those changes.

[brikis98]

Hi @apparentlymart!

It sounds like what you're trying to do here is use terraform init -from-module essentially just as a thing wrapper around the go-getter and registry functionality to retrieve a single codebase, which isn't really how this option was designed to behave.

Yup, pretty much. Perhaps it would be better for Terragrunt to just call go-getter directly for the code-downloading step. My only concern was if Terraform had extra logic around go-getter (i.e., pre-processing URLs or doing some logic after download) that we'd lose. Could you point me to the relevant Terraform code?

[apparentlymart]

I think it's probably better to have Terraform do it so that it will always match the current version of Terraform; the way it is configured is pretty standard now, but the details have slowly changed over time and will probably continue to do so.

The registry installer is also within Terraform itself rather than go-getter, due to the versioning support, so registry-style paths wouldn't work.

I'm on my phone right now so can't dig in to the code, but when I am at a computer I'll dig up a reference to the newer implementation of -from-module and installation in general. Since details are changing for the next release around how the .terraform/modules dir is arranged (no more hashes for directory names) but the retrieval behavior is still essentially the same.

[brikis98]

Update: this is still broken in Terraform 0.12, but now with a different error message:

$ terraform init -get=false -get-plugins=false -backend=false -from-module=<MODULE> <FOLDER>

Copying configuration from "<MODULE>"...
- nested

There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Module not installed

  on main.tf line 1:
   1: module "nested" {

This module's local cache directory <DIR> could not be read. Run "terraform init" to install all modules required by this configuration.

[pselle]

Hi @brikis98 I'm looking at this, since you mentioned it in #21653 -- I'm concerned I'm not replicating your cases as you are, notably, where you are running terraform init.

Is this issue still present for you?

[brikis98]

The last I saw of it was what I mentioned in my comment above. Due to this issue and #21653, Terragrunt now uses go-getter directly instead of terraform init -from-module, so I'm not sure if it's still a problem.

[rifaterdemsahin]

I have a similar issue as well
#22791