Is there no support for explicitly defining access_key and secret_key in terragrunt backend configuration?

[ghost]

I'm trying to define my access_key and secret_key explicitly as per the example below in the .tfvars file. But I'm receiving this error...

[terragrunt] 2019/06/06 22:02:59 Error finding AWS credentials
Is this because Terragrunt uses the AWS library to find the credentials at the shared credentials file and environmental variables? I'm utilizing DigitalOcean spaces as an S3 compatible backend. Do I have to drink the AWS koolaid on this one?

# Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules,
# remote state, and locking: https://github.com/gruntwork-io/terragrunt
terragrunt = {
  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state {
    backend = "s3"

    config {
      skip_credentials_validation = true
      skip_metadata_api_check     = true
      access_key = "ACCESSKEY"
      secret_key = "SECRETKEY"
      endpoint = "https://sfo2.digitaloceanspaces.com"
      region = "us-east-1"
      bucket = "..."
      key = "${path_relative_to_include()}/terraform.tfstate"
    }
  }

  # Configure root level variables that all resources can inherit
  terraform {
    extra_arguments "-var-file" {
      commands = ["${get_terraform_commands_that_need_vars()}"]

      optional_var_files = [
        "${get_tfvars_dir()}/${find_in_parent_folders("account.tfvars", "ignore")}",
      ]
    }
  }
}

[yorinasub17]

Hi @kadenlnelson,

This is a use case we haven't addressed yet: using the s3 backend with something that is not AWS. What is happening is that because you are using the s3 backend, you are triggering the workflow where terragrunt tries to auto create the bucket (See: https://github.com/gruntwork-io/terragrunt#create-remote-state-and-locking-resources-automatically). In this workflow, terragrunt makes API calls to try to determine if the bucket needs to be precreated.

I think there are two issues here:

1. terragrunt is ignoring the access key and secret key configured on the remote state block. We haven't implemented this, because we generally recommend avoiding leaking credentials in the config file and recommend using other mechanisms to provide it. In theory, this should all still work if you switch to using the env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Note that terraform also uses the go SDK so if it works for terraform, it will work for terragrunt. That said...
2. terragrunt will also try to auto create dynamodb tables to enable locking, which I am assuming DO doesn't have an equivalent for. So even if you get past the authentication credentials issue, you will most likely hit an error when terragrunt tries to determine if the dynamodb tables should be created.

I think the best way to address this is to enhance terragrunt with a feature flag that disables the auto backend initialization. A PR to implement this functionality would be much appreciated! A starting point for implementation would be to update the TerragruntConfig with a flag (https://github.com/gruntwork-io/terragrunt/blob/master/config/config.go#L19), which is used to skip initialization of the s3 backend by returning false in the initializer.NeedsInitialization function call (https://github.com/gruntwork-io/terragrunt/blob/master/remote/remote_state_s3.go#L94).

Note that any implementation should be held off until #731 merges, which addresses tf12 compatibility.

[ghost]

In theory, this should all still work if you switch to using the env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Yes, I have successful S3 backend initialization with this method + with aws profiles using shared credentials file.

terragrunt will also try to auto create dynamodb tables to enable locking, which I am assuming DO doesn't have an equivalent for. So even if you get past the authentication credentials issue, you will most likely hit an error when terragrunt tries to determine if the dynamodb tables should be created.

DO doesn't have managed NoSQL, but I don't think that Terragrunt auto-creates DynamoDB tables, though I'm not 100% sure

If I have some free time I'd love to submit a PR, but if anyone else wants to crack at this, please help! 😢

[yorinasub17]

I don't think that Terragrunt auto-creates DynamoDB tables, though I'm not 100% sure

Ah it looks like it only creates the table if dynamodb_table is set in the remote state config.

So I think it should actually work as is against the DO endpoint, if you switch to using env vars. Is that not the case? Are you getting an error / is the bucket being created in AWS?

Or do you have something in your environment that prevents you from using env vars and need to hard code the creds in the config?

[ghost]

In theory, this should all still work if you switch to using the env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Yes, I have successful S3 backend initialization with this method + with aws profiles using shared credentials file.

[k3a]

Lost several hours due to this because of unhelpful error messages "SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method."

If there is no support for access_key and secret_key (and there is support in Terraform), it should ideally warn user that this is not supported and that env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY needs to be used.

I agree that it's not ideal to have secrets in the config but there may be cases for this, especially if Terraform itself supports it. But there may be cases where the credentials are already loaded in a variable from somewhere else.

Similarly locking - it is a nice thing, but may not be necessary for smaller deployments with S3-compatible remote stores so please don't ignore S3-compatible stores just because there is no locking.

[yorinasub17]

We would love to address this, but we don't use anything other than S3 or GCS internally at Gruntwork and being a small team, it's hard for us to maintain and implement all the features for the different clouds. This doesn't mean that we don't want to support this, but it does mean that this is a lower priority for us to implement and need help from the community.

FWIW, a workaround in newer versions of terragrunt would be to bypass the remote_state block and use generate blocks for your backend configuration. I don't have an example for this, but hopefully it is possible to figure it out from the docs. The only advantage of going through remote_state blocks at this point is for the auto bucket initialization features.

[voron]

Thanks @yorinasub17 for a workaround, here is my working example:

• remove remote_state block, do not use generate inside it. remote_state block activates terragrunt's feature to provision/check for bucket despite of generate.
• and in place of removed remote_state block use dedicated generate block inside your root terragrunt.hcl:

generate "s3_backend" {
  path      = "s3_backend.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
terraform {
  backend "s3" {
    bucket      = "bucket-name"
    key         = "${path_relative_to_include()}/terraform.tfstate"
    access_key = "access_key"
    secret_key = "secret_key"
    # it may be any valid aws region, it doesn't affect anything
    region = "us-west-1"
    # endpoint should match bucket's region
    endpoint = "https://fra1.digitaloceanspaces.com"
    skip_credentials_validation = true
    skip_get_ec2_platforms = true
    skip_requesting_account_id = true
    skip_metadata_api_check = true
  }
}
EOF
}

[yordis]

My fix was that I had to add disable_init = true

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    skip_metadata_api_check     = true
    skip_credentials_validation = true
    endpoint                    = "https://nyc3.digitaloceanspaces.com"
    region                      = "us-east-1"
    bucket                      = "..."
    key                         = "${path_relative_to_include()}/terraform.tfstate"
  }
  disable_init = true # here
}

Otherwise when I do the following

terragrunt init --backend-config=../../config.do.tfbackend

I would get some weird questions that wouldn't work

terragrunt init --backend-config=../../config.do.tfbackend
WARN[0000] Encryption is not enabled on the S3 remote state bucket sht-tf. Terraform state files may contain secrets, so we STRONGLY recommend enabling encryption! 
Remote state S3 bucket sht-tf does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) 

[introquest]

haven't addressed yet?

[andrewsomething]

Just wanted to share that I've found it possible to use a proper remote_state block with DigitalOcean Spaces rather than using the generate block work-around or disabling the init altogether. There are a number of AWS-specific options that can be disabled.

Here's an example that works for me:

remote_state {
    backend      = "s3"
    generate     = {
        path        = "backend.tf"
        if_exists   = "overwrite"
    }
    config = {
        endpoint            = "https://sfo3.digitaloceanspaces.com"
        key                 = "${path_relative_to_include()}/terraform.tfstate"
        bucket              = "my-state-bucket"
        region              = "us-east-1"

        # Disable AWS-specific options for Terraform
        skip_credentials_validation = true
        skip_metadata_api_check     = true

        # Disable AWS-specific options for Terragrunt
        # https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#remote_state
        skip_bucket_public_access_blocking = true
        skip_bucket_ssencryption           = true
        skip_bucket_root_access            = true
        skip_bucket_enforced_tls           = true
    }
}