[QT-577] Execute scenario sub-commands faster #101
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our initial implementation of flightplan decoding would always decode and validate all scenarios and their corresponding variant combinations. While this was nice to ensure that all possible combinations are valid, with large matrices this becomes quite a burden. Vault and Vault Enterprise are two examples where you can see the "generate" step (which is actually decoding, validating, and transpiling the scenario into a Terraform root module) take upwards of a minute for both
enos scenario run
andenos scenario destroy
. It also makes local development difficult because it is run every time you try and test a module.Instead of decoding all flightplan plans and selecting those that match the filter, we instead decode the matrix, create a sub-matrix with all combinations that match a given filter, then decode only those scenarios that that would match the filter. This greatly improves our time because we don't act on scenarios that are not targeted. For cases where we do match lots of scenarios, we've also implemented concurrent decoding, which allows us to parallelize the decoding process across all available CPU cores. In vault we see validation of all scenarios go from ~43 seconds on my machine to ~5. We also see the "generate" step finish almost immediately. This should shave anywhere from 1.5 -> 2 minutes from all enos scenario runs in CI.
As part of this change I also updated all of the Go modules and modernized our golangci-lint configuration. This makes our CI standards for enos-provider and enos the same. Updating the linters required a number of fixes. Most of small, the most significant were how we could parallelize the acceptance tests. That required further work on the gRPC listener automatically accepting free ports when one is not provided.
make test-acc
was cut down by 66%.Checklist