RFC: Integration with Terraform Test

[novekm]

Is this related to an existing feature request or issue?

No response

What part of the Cloud Game Development Toolkit does this RFC relate to?

Modules

Summary

This proposal is for integration with Terraform Tests. Ideally, all local and pipeline tests will be transitioned from Terratest to Terraform Tests.

Use case

The reasoning behind this is to streamline testing for the entire project, especially when it comes to new module contributions. Currently there is some portion that is manual, and in addition the requirement of tests in Terratest having to be written in Go can be a barrier to entry for contributors. With Terraform Test, tests are written in HCL and examples that will already be written/modified, can be used as the tests themselves which simplifies the process and encourages the writing of tests more frequently. Terraform Test manages state on your behalf, and will create and destroy resources based on the tests that are defined by running a simple terraform test command.

Proposal

This proposal is for integration with Terraform Tests. Ideally, all local and pipeline tests will be transitioned from Terratest to Terraform Tests. Alongside a new module addition, a subsequent example is written. This example will be pointed at for the test. Additionally, this makes it easier for newer contributors as they can easily run local and pipelines tests to have confidence in the functionality in their PR, even when making smaller changes. This can also be leveraged by dependabot to ensure small version updates also don't inadvertently break functionality.

For configuration, we'd leverage a centralized CI account where we could have dependencies already setup/configured (such as domains, etc.) which would be used for pipeline tests.

For local tests, we can leverage abp-local and define a series of tests that can run locally with the abp-local binary and docker. This will accelerate local development, and result in less pipeline runs until there is a substantial level of confidence in the functionality of the PR. Examples of these types of configurations can be seen in the IAM Identity Center Terraform Module, specifically in the .config, .project-automation, examples, and tests directories.

Out of scope

Testing of Packer may need to be out of scope for now in terms of Terraform Test, however I'd like to add it in the future depending on how much of a focus it will continue to be. There was a a few Packer issues (#439, #440) identified that a dedicated test would help avoid in the future. The test would create the AMI, validate it's creation in the correct region, and then destroy the AMI for cleanup).

At the moment I believe Terraform Test is only scoped to Terraform (as you'd probably assume). However, there may be other ways we can automate testing of Packer templates, though this may require leveraging a combination local-exec and null resource resources to run packer CLI commands, which would need to be in the context of the related Terraform Module(s) (e.g. Perforce modules). Alternatively, we could make a dedicated module for testing of all packer templates available in CGDTK.

Potential challenges

Potential challenges may be timing, as we currently are in a sprint. Ideally this would be implemented after this sprint is completed, and before the next one begins. There are already manny large PRs in the works, so adding another one that could potentially impact all modules and the ability of PRs to be merged is risky to do at the moment.

Dependencies and Integrations

No response

Alternative solutions

No response

[novekm]

@henrykie @gabebatista thoughts?

[gabebatista]

I think that integrating Terraform Tests into the project is a great idea and should be a high priority. We can dedicate a sprint to create the tests and flesh out any examples we might be missing, but I think that we should wait until the large PRs (Cloud DDC and Perforce updates) we are currently reviewing are merged. Regarding the dependencies in a CI account, we might be able to mock some of the resources as stated in the docs here. A few quick questions for you:

• Do we have any tests written out in terratest today? I didn't see any in the repo.
• Will these tests be primarily used to speed up development or do you see this as something we add to our PR process (github actions perhaps)?
• While I agree that starting with just testing the actual infrastructure (terraform) is the right first step, I think we should think through how we would implement all of our tests to make sure all the frameworks align. Do you have any suggestions regarding how we would implement tests for packer and ansible?

[gabebatista]

Moving this issue to Discussions.

[novekm]

Agreed - I think the ideal time is right after this sprint is completed, but before the next one starts. We can mock the resources in the CI account that may be necessary, but I think we can also create a standard to store any value that potentially may be needed (domains, email addresses, VPC IDs, etc.) in SSM Parameter Store in the CI account.

Responses to the above questions:

Do we have any tests written out in terratest today? I didn't see any in the repo.

I may have been mixing this up with another project. Perhaps I'm thinking of a discussion where we were talking about tests and were deciding between terratest or terraform test.

Will these tests be primarily used to speed up development or do you see this as something we add to our PR process (github actions perhaps)?

Both, but slightly more related to the PR process. I see a few ways these can be used:

Speed up development

In the most simple form, tests can just reference a module, including a root module, such as the core configuration files a contributor would be writing. For example, see this directory structure for the perforce-complete sample I'm working on:

  ├── .gitignore
├── README.md
├── data.tf
├── dns.tf
├── local.tf
├── main.tf
├── outputs.tf
├── security.tf
├── ses.tf
├── tests
│   └── 01_mandatory.tftest.hcl
├── variables.tf
├── versions.tf
└── vpc.tf

Within my test (01_mandatory.tftest.hcl) I have the following:

run "unit_test" {
  command = plan
  module {
    source = "./"
  }
}

run "e2e_test" {
  command = apply
  module {
    source = "./"
  }
}

The above will run terraform plan and terraform apply (and subsequently terraform destroy) when running the terraform test command. This can be used for local development, or also by the pipeline for testing against PRs. While the terraform test is running, it will maintain state on your behalf. You can also create additional tests configured similar to the examples in the link you sent. Those are good for being able to test for intentional errors with additional validation, etc. However you can also create standard tests in.

The small downside with the standard structure is it leads to a bit of duplication, since you'd be configuring the test to create resources, when you could just point at the resources that are already defined in the configuration files. Each way of writing the tests are valuable and have their own place, but I propose at the minimum all submitted code has a mandatory test (testing basic functionality using the configuration files themselves). I think this will keep the barrier to entry for contributors low.

For more complex use cases, such as the modules themselves where we likely would want to have some additional, more robust tests, we lean into the other standard way to define tests.

For speeding up development, users always will have the option to simply run terraform test if they'd like. They can also follow the typical path of:

• write code
• run terraform plan
• troubleshoot any errors
• run terraform apply
• repeat
• test full terraform apply and terraform destroy once finished

In general, I'd recommend following the typical path until closer to "done" with whatever TF code is being written, but the option is always there.

GitHub Actions for Testing PRs

This is the core way I see terraform test being used - alongside out other tests (checkov, etc.). When submitting a PR, the final step should be to replace values with SSM Parameters (using data sources) that point to those that exist in the CI account. That way the test can run without us writing sensitive values for things that live in the CI account. The other benefit is that this will speed up tests. We should not necessarily be creating VPCs, ALBs, etc. during each test, though that is likely something we'd want depending on what the module/sample does. In general, we should lean into using data sources to reference things that already exist in the CI account (VPCs, subnets, etc.) where relevant. I believe that's how most customers would be using the modules - passing in their own existing resources, at least for the larger/more complicated resources or anything that may be in a shared services/shared networking type AWS account

While I agree that starting with just testing the actual infrastructure (terraform) is the right first step, I think we should think through how we would implement all of our tests to make sure all the frameworks align. Do you have any suggestions regarding how we would implement tests for packer and ansible?

I agree. I have some ideas for packer. We could use terraform test with local-exec to run local packer CLI commands and validate AMI creation in target AWS accounts. I haven't used Ansible before, but if it's just running locally, we could probably do the same for that

Potential challenges/things to look out for

The blessing and curse of Terraform Tests are that they manage the state on your behalf. This means that if you run into a weird issue and aren't sure what's going on, you can't inspect the state file if needed since it won't be accessible. Also, if for some reason you force a cancellation of a running test, all of the existing resources that have been deployed will be orphaned in the AWS account and will need manual cleanup. At the end of the day terraform test is just a wrapper that simplifies running multiple terraform commands, so the same logic applies for if you cancelled a terraform apply in the middle of running. Another challenge is that currently the logs in the terminal displayed running terraform test are very minimal. It just shows which test is running, and if it passes or fails. This can cause a bit of a concern for resources that take a long time to deploy, because it can look like a test is stalled. If you cancel it, the above happens - your resources are all orphaned in the AWS account. Ask me how I know this... 😅