[QT-504] Add scenario sampling

[ryancragun]

This feature fully replaces the current manual, brittle method of integrating new test scenarios and variants into product CI/CD pipelines. Instead, Enos scenario authors will be able to automate the process by defining a named subset of possible scenario variants to be run. This feature will allow us to:

• Quickly and easily add new Enos scenarios and variants to the CI/CD pipeline to test new features, configurations, and integrations, with the confidence that our new random sampling configs are ensuring test coverage across all combinations
• Gain greater assurance of product quality by testing a deeper, broader range of elements and configurations over a longer time horizon
• Add tracking of long-term metrics for Enos scenarios, allowing us to identify trends and patterns beyond individual tests [Note: This is a future feature not included in the scope of the sampling feature itself, but is enabled by this work]

Background

Enos scenarios support multi-variant matrices which commonly include parameters like architecture, Linux distro, storage backend, expected version, expected edition, and many more configurations. These matrices allow us to test across every possible combination of these variants, which is part of what makes Enos such a powerful tool for testing.

Of course, as our matrices grow, so does the total number of possible combinations — exponentially. We’re currently in the tens of thousands range across all scenarios in the Vault repos; we’ll easily grow to the hundreds of thousands range once we begin adding new scenarios and variants with the help of this work. Running all scenarios and variant combinations would be prohibitively expensive at any given point in the CI pipeline. Instead, we'll implement an acceptance sampling strategy by algorithmically selecting scenario variants at each multiple stages of the pipeline, with the assumption that throughout the overall pipeline, we’ll achieve a high confidence level of test scenario coverage.

On PRs, release testing, and other points of the pipeline, we generate a variety of artifacts during the build process. These artifacts represent different architectures, package types, and product versions and editions. Our sample selection needs to be structured in a way that allows us to:
Select the scenario variants that are supported for each artifact (for example, for an RPM package for Vault Enterprise built for x86_64, we need to ensure that we are only running scenarios that use that package type, product edition, and architecture).
Distribute the remaining matrix variants across all of the artifacts in a structured way so that we optimize for the highest possible test coverage across variants.

Current State

In its current prototype form, which exists as a convoluted set of configurations in Github Actions, this process is manual, brittle, and not scalable. We do implement semi-random selection, but the distribution of scenario variants (to match artifact types and to attempt to distribute the remaining matrix variants evenly) is manually handcrafted, hardcoded, and difficult to update or change if we want to integrate a new scenario or variant into the pipeline.

The result is that new scenarios and variants are rarely added. Therefore our test coverage isn’t expanding as rapidly and regularly as we’d like.

Proposal

We’ll replace the current brittle configurations in Github Actions with a robust Enos feature that allows us to define and generate named sample groups of scenarios.

Terminology

• Acceptance sampling
  A plan of quality assurance that utilizes statistical sampling to determine whether or not to accept or reject a product.

• Sample element
  In this context, an element refers to one possible configuration of a scenario (in other words, one of the possible matrix combinations for a scenario along with any additional sample attributes)

• Sample frame
  A list of all possible elements which can be sampled. This includes all the possible scenario matrix combinations for all the defined subsets.

• Sample subset
  A subset of the possible matrix combinations for a certain scenario, defined with a matrix or filter. The sample frame is constructed of all included sample subsets.

• Sampling method
  The method of choosing which elements from the sample frame to select.

• Purposive stratified sampling
  The strategy our algorithm will use for randomly selecting scenarios. Stratified sampling refers to weighting each selection based on the size of its subset relative to the total population. For example, if we had three subsets with 100, 30, and 10 elements each, the algorithm would randomly select the total specified number of elements using the same proportions. The largest number of elements would come from the first subset, a smaller number from the next subset, and finally, the fewest from the last and smallest subset. However, in its purest probabilistic form, this strategy could potentially result in a very small subset getting extremely little or no coverage if paired with a much larger subset. Thus, we add in an opinionated “purposive” element to our algorithm, that prioritizes representation across all scenarios, then weights the randomly selected elements by subset size as mentioned above.

• Sample observation
  The process of choosing one-or-more elements from the sample frame via a given sampling method. In our case, this means using our algorithm to select the specified number of elements from all of the defined scenario subsets.

We'll extend the Enos domain-specific language (DSL) to support defining named sample groups. Within these sample groups we'll allow users to configure subsets with different scenario filters, matrices, and attributes that describe compatibility.

We'll use the Enos CLI to interact with the Enos server to observe a given sample. The CLI will also be responsible for shaping the sample frame, i.e. including or excluding subsets from the sample. It will also be used to set boundary conditions on the sample size, i.e. minimum number of elements, maximum number of elements, or a percentage of total elements in the frame.

To ensure that we get coverage over all scenarios, we'll implement a purposive stratified sampling algorithm as our method. Depending on our sample size limitations, we'll favor breadth across all samples before dividing the subsets by size and sampling based on overall proportions.

We'll also support injecting additional metadata into samples and subsets that can be distributed to each sample element during observation. This will allow us to dynamically configure the Enos variables for a sample and pass any other additional data through to our execution environment.

When taking an observation, the Enos CLI we'll have the option of human or machine readable output. The machine readable output will be used to generate a Github Actions matrix to execute scenarios on a per-workflow basis. We'll also generate a random seed for the random number generator. This will allow us to deterministically recreate a sample should we wish to run it locally or retry a failure.

DSL

The following is an example of a full Enos configuration file that could outline a complex test scenario with associated samples. It has examples of simple name and filter based sample subsets, and also more complex samples that can define multiple subsets with different filter types, matrices, and attribute inheritance from the sample to the subsets.

module "upgrade" {
  source = "./modules/upgrade"
}

scenario "upgrade" {
  matrix {
    backend = ["raft", "consul"]
    seal    = ["shamir", "awskms", "cloudhsm"]
  }

  step "upgrade" {
    module = module.upgrade
  }

  variables {
    backend = matrix.raft
    seal    = matrix.seal
  }
}

sample "simple" {
  subset "raft_cloudhsm" {
    scenario_filter = "upgrade backend:raft seal:cloudhsm"
  }
}

globals {
  upgrade_attrs = {
    initial_version = "1.13.2"
  }
}

sample "complex" {
  attributes = {
    aws-region        = ["us-west-1", "us-west-2"]
    continue-on-error = false
  }

  subset "upgrade_consul" {
    scenario_name = "upgrade"

    attributes = {
      continue-on-error = true
    }

    matrix {
      backend = ["consul"]
    }
  }

  subset "upgrade_raft" {
    scenario_name = "replication"
    attributes    = global.upgrade_attrs

    matrix {
      arch    = ["amd64", "arm64"]
      backend = ["raft"]
    }
  }
}

CLI

The following is an example of the human readable output of a sample observation. We must be able to select which sample we wish to observe, have the ability to limit the frame to one-or-more subsets, and to choose our configure size limits.

enos scenario sample observe --chdir all --min 1 --max 5

SAMPLE  SUBSET           	SCENARIO FILTER             	ATTRIBUTES
all 	smoke            	smoke arch:amd64 distro:sles	  aws-region=us-east-1
all 	smoke            	smoke arch:arm64 distro:amz 	  aws-region=us-west-1
all 	smoke_allow_fail  smoke arch:s390x distro:amz     aws-region=us-west-1
all 	upgrade          	upgrade arch:amd64 distro:rhel  aws-region=us-east-1
all 	upgrade          	upgrade arch:arm64 distro:sles	  aws-region=us-west-1

There will also be a machine readable format that includes the sample elements. Each element will contain references to the sample, subset, scenario and elements.

enos scenario sample observe all --min 1 --max 1 --format json | jq

{
  "decode": {},
  "observation": {
    "elements": [
      {
        "sample": {
          "id": {
            "name": "all"
          }
        },
        "subset": {
          "id": {
            "name": "upgrade"
          }
        },
        "scenario": {
          "id": {
            "name": "upgrade",
            "variants": {
              "elements": [
                {
                  "key": "arch",
                  "value": "arm64"
                },
                {
                  "key": "distro",
                  "value": "rhel"
                }
              ]
            },
            "uid": "89d9a70dfb24d8f3602a6523bcf7c46fefde2cd907f6e42b09bca7bef3713f5b",
            "filter": "upgrade arch:arm64 distro:rhel"
          }
        },
        "attributes": {
          "aws-region": "us-west-1",
          "continue-on-error": false
        }
      }
    ],
    "filter": {
      "sample": {
        "id": {
          "name": "all"
        }
      },
      "maxElements": 1,
      "minElements": 1,
      "percentage": -1,
      "seed": "1692986528586626000"
    }
  }
}

Github Actions

The machine readable json output can be used to generate a list of sample elements that we can use to create a matrix of parallel scenario runs. A simplified workflow would look like this:

---
name: enos

on:
  workflow_call:
  inputs:
  sample-name:
    required: true
    type: string

jobs:
  metadata:
    outputs:
      matrix: ${{ steps.metadata.outputs.matrix }}
    steps:
      - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9
      - id: metadata
        run: |
          echo "matrix=$(enos scenario sample observe ${{ inputs.sample-name }} --min 1 --max 2 --format json)" >> "$GITHUB_OUTPUT"

  run:
    needs: metadata
    strategy:
      matrix: ${{ fromJson(needs.metadata.outputs.matrix) }}
    steps:
      - run: enos scenario run ${{ matrix.scenario.id.filter }}
    env:
      ENOS_VAR_initial_version: matrix.attributes.initial-version
      ENOS_VAR_aws_region: matrix.attributes.aws-region

Checklist

• The commit message includes an explanation of the changes
• Manual validation of the changes have been performed (if possible)
• New or modified code has requisite test coverage (if possible)
• I have performed a self-review of the changes
• I have made necessary changes and/or pull requests for documentation
• I have written useful comments in the code

[rebwill]

Somewhere (probably before this description, like in the README or docs or something) we need to define all the vocabulary used in this feature: sample, subset, frame, observation, etc. They are the statistically accurate terms but are likely not very meaningful/obvious to anyone who does not have a statistics background.

[rebwill]

Brainstorm for a more layperson-friendly version of this text:

"Take an observation of the scenario sample. This will print a list of all the possible scenario filters included in the subsets for this sample (also known as the sample frame). The observation can be limited to a particular subset by using the --subset [insert the actual arg/flag for this] flag. The possible scenarios can be limited using: a) subset filters or b) --include or --exclude flags. The number of randomly selected scenarios to display can be limited using the min (minimum number of scenarios to display), max (maximum number of scenarios to display), and pct (percentage of the overall possible scenarios according to the subset filters) flags. If the max is set to a negative number, it will set no default upper bound*. If a pct is set, it will create an upper bound of the percentage of the total sample frame. If both a max and a pct are set, whichever is lower will be used as the upper bound. By default, the min is set to 1 unless otherwise specified. If a min is set that is higher than the actual number of scenarios in the frame, an error will be returned. The seed can be used** to control the random number source to create deterministic observations. If the seed number is negative, a seed will be automatically generated for you."

*Question: what do we mean by "default" upper bound here? Is there a non-default option that would be set? If not can we just say "will not set an upper bound"?
**Question: can we say something more specific than "the seed can be used"? As a new user I'm not totally clear on what this actually means in terms of what I can/should set for a seed.
*** Note: I kind of hate having all of this in one dense paragraph, are we able to add line breaks to this?

[ryancragun]

Re *: Yes, the default upper bound is the entire frame. If you want to set an upper bound you have to specify a max or pct.
Re: **: We pass the seed number to our random number entropy source. If you don't provide it then you'll get a "random" but predictable seed of the Unix timestamp of the current time. You'd never do that for anything secure but it's fine for our purposes. If you provide the seed then it will use the same randomness entropy and thus yield a deterministic output.
Re: ***: Line breaks should be fine.

[rebwill]

What does half-opened interval [0,n) mean here?

[ryancragun]

Interval [0,n) where n == 3 = [0,1,2]. So here we're assuming we're starting the interval at zero and ending with whatever you pass in. So it's half opened and you're setting the closed bound.

[ryancragun]

Notes:

• We don't currently validate that all subsets have a valid frame.
• Maybe validate should have a mode to do that?

[ryancragun]

Interval [0,n) where n == 3 = [0,1,2]. So here we're assuming we're starting the interval at zero and ending with whatever you pass in. So it's half opened and you're setting the closed bound.

[rebwill]

Stoked for sampling 🚀