Adversarial tests for provenance-only workflow

[ianlewis]

Depends on #28

Adversarial tests:

• Adversarial tests with an invalid input file name

[ianlewis]

@laurentsimon Do we really need adversarial tests? The seem to be inherently flaky and only really seem be testing the sha check that we do when uploading/downloading the artifact between jobs.

We already check most of the important stuff like modifying the binary in our existing e2e tests

[laurentsimon]

I think we want the checks for places where we upload/download to/from the artifact registry, in order to validate that tampering is detected. This is really important since the re-usable workflow is different from the Go re-usable workflows. So I'd say yes for the first item.

The other twos are less important. The release one is flaky and does not add much security anyway. The second item on the list is simple enough that we could add it.

Wdut?

[ianlewis]

For the generic provenance we don't build the artifacts or upload/download them so there's not really anything to actually test for the first.

I can maybe take a stab at the second and/or third, though I actually think they are pretty much all flaky and depend a lot on timing. Several of the ones for the Go workflow have just not been creating issues for other reasons. See slsa-framework/example-package#71

[laurentsimon]

For the generic provenance we don't build the artifacts or upload/download them so there's not really anything to actually test for the first.

ah right, I forgot it all happens in a single VM. Good point.

I suppose item 2 is the only one we may verify then?

[ianlewis]

Yeah, if we can get it to work.

[laurentsimon]

this one should be easy, since it does not involved timing

[ianlewis]

SG

[ianlewis]

I may not even need to write the one for invalid path either since the generic workflow doesn't have an input config file.

I'm assuming the corresponding test for the Go workflow is this one.
https://github.com/slsa-framework/example-package/blob/main/.github/workflows/e2e.go.schedule.main.adversarial-invalidpath.yml

[laurentsimon]

You're correct. Maybe you can write one with invalid subject formatting instead?

[ianlewis]

You're correct. Maybe you can write one with invalid subject formatting instead?

@laurentsimon added tests here:

• https://github.com/slsa-framework/example-package/blob/2cb10f485dc36dd99476f3e3b7eec99dfd796da2/.github/workflows/e2e.generic.schedule.main.adversarial-invalidsubjects.slsa3.yml
• https://github.com/slsa-framework/example-package/blob/main/.github/workflows/e2e.generic.schedule.main.adversarial-invalidpath.slsa3.yml

I think we can close this now.