Tracking: Test environment reservation service

[mrgrain]

Overview

Our integ tests need to run in an exclusive environment. Currently we are achieving this roughly by two ways:

• use of sandbox accounts to establish strong boundaries (e.g. canaries vs test pipelines)
• designation of regions per test to separate different tests within the same sandbox

However this model doesn't scale well. We should keep the Sandbox account boundary because we won't have that many
different sandboxes and setting up accounts as a one-off operation is relatively cheap.

Instead of region designations, we should however have a service that reserves and assigns a an account and region pair to a test case. Some test cases (like cross-account deployments #31934) might reserver multiple pairs. The service can also ensure environments are regularly cleaned up from any accidentally left-over resources.

Requirements

A new service an integration test can request exclusive, pre-configured environments to execute tests in.

• MUST support exclusive reservations for pairs of account/region from a pool
• MUST guarantee that test reservations do not starve production reservations (release, canaries...) (for example via the use of different pools)
• MUST support requesting combinations for cross-account deployment use cased
• MUST provide the requesting client with required credentials (i.e. role arn) to execute tests, and other execution-specific metadata like unique prefixes, tags to use, existing resource names, etc...
• MUST be robust against clients dying while holding a reservation (i.e. not hold on to reservations forever)
• SHOULD support metrics about wait time, test run time, failure rates etc.
• SHOULD support bootstrapping of envs for a given reservation, e.g. configuring 2 accounts with cross account trust
• SHOULD support clearing of environments from accidentally left-over resources
• SHOULD support metrics on accidentally left-over resources and the reservations that caused that

[rix0rrr]

Would also be neat if it could keep statistics about wait time, test run time, failure rates etc.

And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.

[rix0rrr]

I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly (pre-existing resources names, unique prefixes, etc). Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible

[mrgrain]

And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.

I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly. Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible

Yes, but. We also need to provision for testing of bootstrapping. But I agree that for non-bootrapping test cases, pre-bootstrapped environments would be desirable.

[rix0rrr]

Sure. Maybe then we just ask for 2 accounts and do the --trust bootstrapping as part of the test, that works too. It's probably better!

Requests for the other hand-provisioned crap still stands though 😉 .

The example I always go to is certificates and Hosted Zone IDs. I guess that doesn't apply to CLI tests yet... but (a) we could use the same coordinator for automating construct library tests and (b) I'm sure there will be something manual at some point.

[iliapolo]

It would great if this service could also be used in local development. This will make writing and developing complex tests easier, which means we will do more of them.

Perhaps we can create a dedicated pool for local dev usage.

[iliapolo]

Maybe then we just ask for 2 accounts and do the --trust bootstrapping as part of the test, that works too. It's probably better!

Yes, I like that better.