[ENH] - Use GitHub secrets instead of Vault

[smokestacklightnin]

Reference Issues or PRs

Resolves #2835

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Any other comments?

[marcelovilla]

Thanks @smokestacklightnin, this looks good!

We'll wait for @dcmcand to come back so we can add the secrets and test the workflows. I think the provider workflow won't run from a fork and the other three (aws, azure and gcp) will need to be triggered manually.

[smokestacklightnin]

The AWS Deployment test failure shows a puzzling error (I included line numbers because it is a very long file):

222419   │ 2025-01-07T21:19:12.3277347Z 2025-01-07 21:19:12     ERROR  206 deployment_fixtures: Please specify `region` in the nebari-config.yaml or if initializing the nebari-config, set t
         │ he region via the `--region` flag or via the AWS_DEFAULT_REGION environment variable.
222420   │ 2025-01-07T21:19:12.3278318Z Traceback (most recent call last):
222421   │ 2025-01-07T21:19:12.3281269Z   File "/home/runner/work/nebari/nebari/tests/tests_integration/deployment_fixtures.py", line 204, in deploy
222422   │ 2025-01-07T21:19:12.3282066Z     _cleanup_nebari(config)
222423   │ 2025-01-07T21:19:12.3282927Z   File "/home/runner/work/nebari/nebari/tests/tests_integration/deployment_fixtures.py", line 101, in _cleanup_nebari
222424   │ 2025-01-07T21:19:12.3283757Z     aws_cleanup(config)
222425   │ 2025-01-07T21:19:12.3284696Z   File "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/_nebari/provider/cloud/amazon_web_services.py", line 1028, in aws_cleanup
222426   │ 2025-01-07T21:19:12.3285734Z     aws_delete_s3_bucket(s3_bucket_name, region)
222427   │ 2025-01-07T21:19:12.3286955Z   File "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/_nebari/provider/cloud/amazon_web_services.py", line 781, in aws_delete_s3_bucket
222428   │ 2025-01-07T21:19:12.3288162Z     aws_delete_s3_objects(bucket_name, endpoint, region)
222429   │ 2025-01-07T21:19:12.3289497Z   File "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/_nebari/provider/cloud/amazon_web_services.py", line 719, in aws_delete_s3_objects
222430   │ 2025-01-07T21:19:12.3290707Z     session = aws_session(region=region)
222431   │ 2025-01-07T21:19:12.3291441Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^
222432   │ 2025-01-07T21:19:12.3292615Z   File "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/_nebari/provider/cloud/amazon_web_services.py", line 36, in aws_session
222433   │ 2025-01-07T21:19:12.3293739Z     raise ValueError(
222434   │ 2025-01-07T21:19:12.3294860Z ValueError: Please specify `region` in the nebari-config.yaml or if initializing the nebari-config, set the region via the `--region` flag or via the AWS_DEFAULT_REGION environment variable.

I'm not sure yet why the region isn't being recognized when the region is specified in at least one place.

CC: @marcelovilla

[marcelovilla]

@smokestacklightnin I ran the tests again with a less verbose Terraform log level, and found the following issue:

ValueError: Cloudflare required environment variable=CLOUDFLARE_TOKEN not defined
2025-01-16 09:20:54     ERROR  187 deployment_fixtures: Deploy Failed, Exception: Cloudflare required environment variable=CLOUDFLARE_TOKEN not defined

We need to make sure that secret is exposed as an environment variable too. I can make that change

[marcelovilla]

@smokestacklightnin I exposed CLOUDFLARE_TOKEN as an environment variable, and the AWS and GCP tests are now running fine. See:

• https://github.com/nebari-dev/nebari/actions/runs/12806334462 (AWS)
• https://github.com/nebari-dev/nebari/actions/runs/12808306322 (GCP)

I'm still waiting until we fix #2893 to run the Azure one. Once that is done, I think we're ready to merge either of the PRs. If we're merging this, please merge my branch into it as it has the latest changes.

[marcelovilla]

Azure tests are passing: https://github.com/nebari-dev/nebari/actions/runs/12813193533

[marcelovilla]

@smokestacklightnin can you include this commit in this PR? While everything was already working fine, I noticed we had a duplicate env variable and decided to expose some of the env vars at the step level instead of the workflow level. Tested from my PR and everything looks good.

[smokestacklightnin]

@smokestacklightnin can you include this commit in this PR? While everything was already working fine, I noticed we had a duplicate env variable and decided to expose some of the env vars at the step level instead of the workflow level. Tested from my PR and everything looks good.

@marcelovilla Done

[marcelovilla]

Thanks @smokestacklightnin 🚀