Introduce CI for AWS - part 1 #2274
Conversation
| name: aws | ||
| if: | | ||
| github.event_name == 'workflow_dispatch' | ||
| needs: [podvm, image, prep_install] |
There was a problem hiding this comment.
Does AWS only work with the packer build, or also mkosi?
There was a problem hiding this comment.
oh, I forgot to say on the description...I couldn't make it work with mkosi and it's in my list to debug. The workflow supports both packer and mkosi image though.
There was a problem hiding this comment.
AWS works perfectly fine with mkosi. The default images made available (and part of instructions in confidentialcontainers.org) starting 0.11.0 are mkosi based.
Once you have created the mkosi raw image, use the raw-to-ami.sh script to upload and create AMI or you can use uplosi.
There was a problem hiding this comment.
hmmm I chained these changes with the existing workflow that builds the mkosi-based image, but all AWS e2e tests failed. Maybe the problem was/is somewhere else. I will revisit that topic soon.
| export TEST_PODVM_IMAGE="${{ env.PODVM_QCOW2 }}" | ||
| export TEST_E2E_TIMEOUT="90m" | ||
|
|
||
| make test-e2e |
There was a problem hiding this comment.
Can we get any debug logs in case things go wrong?
There was a problem hiding this comment.
Good idea, I overlooked it completely. In a next PR hopefully
|
Hi @stevenhorsman !
IMO, we can and must separate into scripts. Share common code and avoid the nasty pull_request_target limitation. I can give it a try on a following PR. |
Created a callable workflow for running the AWS e2e tests. This initial implementation has support for testing mkosi or packer based images, being default the later. The cluster_type has only support to "onprem" cluster, and the workflow will create a kcli-based kubeadm one. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The new created e2e_aws is called by the e2e_run_all, so AWS e2e tests will run on nightly. At this point it won't be triggered by pull request. It's testing the packer based podvm images. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Tagging with "Name" all the AWS resources created to help on tracking and removal of them all, mainly when running on CI. In order to tag images I had to bump github.com/aws/aws-sdk-go-v2/service/ec2 which cascated to updating other modules. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
So to generate unique names to avoid clashing published images. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
So that on VPC teardown (if enabled) the created AMI will be deleted along with its corresponding EBS snapshot. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Delete the key from bucket that contains the raw disk image. Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
|
Updated with fixes to workflow lint errors. |
There was a problem hiding this comment.
Hello Wainer, overall it looks quite good, I'd suggest tagging the resources using the workflow-id to allow an extra cleanup job that would iterate over resources and deleted the ones where workflow finished. To further simplify that, if possible, I'd like to add a ci=true tag to every resource that is generated. I guess currently that is not supported by peer-pods, right? I think it'd be a useful feature even there (for peer-pods VMs so people can add custom tags).
One thing I noticed is that the job seems to attempt to remove even VPCs it had not created. Would be nice to double-check and only cleanup when we created them... (not 100% sure about that)
Looking forward for the part 2 ;-)
|
Hi @ldoktor !
In part 2 that I should send soon I added an extra step that runs "always" to do such as cleanup in case the e2e finish abruptly. I just need to ensure it deletes only the resources created a that given workflow execution. I never tried myself, but it should be possible to configure peer-pods to tag the podvms: https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/install/overlays/aws/kustomization.yaml#L35
It should delete only the VPCs created (and related resources) created by the e2e test run.
\o/ |
This is just part 1 of a series of commits to run the e2e tests nightly for AWS too.
The way it's now, tests are executed but some fails. In particular, the TestAwsCreatePeerPodWithLargeImage fails in such as bad fashion that the job gets cancelled. The good news is that the simple pod tests at least pass. Here is an execution on my fork: https://github.com/wainersm/cc-cloud-api-adaptor/actions/runs/13122445400
Below is a list of things I still have to work on to make it acceptable running alongside our CI (at this point, the job will skip because I won't configure the AWS credentials on this repo yet). Nevertheless, I'd like to have this part merged because there are other aws-unrelated changes I plan to submit to the workflows and I want to avoid keep rebasing & resolving conflicts in my fork.
What's next: