Stream logs to separate buckets

[dms1981]

A reference to the issue / Description of it

#7607

How does this PR fix the problem?

Now that we have separate buckets for separate destinations, this PR streams logs to the relevant buckets

How has this been tested?

Tested with local plan, through CI pipeline, and previously in sandbox

Deployment Plan / Instructions

Deploy through CI

Checklist (check x in [ ] of list items)

• I have performed a self-review of my own code
• All checks have passed
• I have made corresponding changes to the documentation
• Plan and discussed how it should be deployed to PROD (If needed)

Additional comments (if any)

{Please write here}

[github-actions]

Trivy Scan Success

Show Output

```hcl

---

Trivy will check the following folders:
terraform/environments/bootstrap/member-bootstrap
terraform/environments/core-network-services
terraform/environments/core-vpc

---

Running Trivy in terraform/environments/bootstrap/member-bootstrap
2024-08-23T11:19:47Z INFO [db] Need to update DB
2024-08-23T11:19:47Z INFO [db] Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2"
2024-08-23T11:19:49Z INFO [vuln] Vulnerability scanning is enabled
2024-08-23T11:19:49Z INFO [misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:49Z INFO Need to update the built-in policies
2024-08-23T11:19:49Z INFO Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-08-23T11:19:49Z INFO [secret] Secret scanning is enabled
2024-08-23T11:19:49Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:49Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:53Z INFO Number of language-specific files num=0
2024-08-23T11:19:53Z INFO Detected config files num=5
trivy_exitcode=0

---

Running Trivy in terraform/environments/core-network-services
2024-08-23T11:19:53Z INFO [vuln] Vulnerability scanning is enabled
2024-08-23T11:19:53Z INFO [misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:53Z INFO [secret] Secret scanning is enabled
2024-08-23T11:19:53Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:53Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:56Z INFO Number of language-specific files num=1
2024-08-23T11:19:56Z INFO [gomod] Detecting vulnerabilities...
2024-08-23T11:19:56Z INFO Detected config files num=8
trivy_exitcode=0

---

Running Trivy in terraform/environments/core-vpc
2024-08-23T11:19:57Z INFO [vuln] Vulnerability scanning is enabled
2024-08-23T11:19:57Z INFO [misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:57Z INFO [secret] Secret scanning is enabled
2024-08-23T11:19:57Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:57Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:58Z INFO Number of language-specific files num=0
2024-08-23T11:19:58Z INFO Detected config files num=4
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/bootstrap/member-bootstrap
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running Checkov in terraform/environments/bootstrap/member-bootstrap
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-08-23 11:20:00,733 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-cross-account-access?ref=6819b090bce6d3068d55c7c7b9b3fd18c9dca648:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:00,734 [MainThread  ] [WARNI]  Failed to download module github.com/terraform-aws-modules/terraform-aws-iam//modules/iam-assumable-roles?ref=de95e21a3bc51cd3a44b3b95a4c2f61000649ebb:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:00,734 [MainThread  ] [WARNI]  Failed to download module github.com/terraform-aws-modules/terraform-aws-iam//modules/iam-assumable-role?ref=de95e21a3bc51cd3a44b3b95a4c2f61000649ebb:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:00,734 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-github-oidc-role?ref=62b8a16c73d8e4422cd81923e46948e8f4b5cf48:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:00,734 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-github-oidc-provider?ref=82f546bd5f002674138a2ccdade7d7618c6758b3:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:00,734 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 148, Failed checks: 0, Skipped checks: 51


checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/core-network-services
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-08-23 11:20:03,985 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:03,986 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/terraform-aws-observability-platform-tenant?ref=fbbe5c8282786bcc0a00c969fe598e14f12eea9b:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 492, Failed checks: 0, Skipped checks: 186


checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/core-vpc
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-08-23 11:20:08,491 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:08,491 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/terraform-aws-observability-platform-tenant?ref=fbbe5c8282786bcc0a00c969fe598e14f12eea9b:None (for external modules, the --download-external-modules flag is required)
2024-08-23 11:20:08,491 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-member-vpc?ref=3094604aee5121f9794d60d3f27d21d6209e59d4:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 504, Failed checks: 0, Skipped checks: 159


checkov_exitcode=0

CTFLint Scan Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.5.0)
tflint will check the following folders:
terraform/environments/bootstrap/member-bootstrap
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running tflint in terraform/environments/bootstrap/member-bootstrap
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/core-network-services
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/core-vpc
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output

*****************************

Trivy will check the following folders:
terraform/environments/bootstrap/member-bootstrap
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running Trivy in terraform/environments/bootstrap/member-bootstrap
2024-08-23T11:19:47Z	INFO	[db] Need to update DB
2024-08-23T11:19:47Z	INFO	[db] Downloading DB...	repository="ghcr.io/aquasecurity/trivy-db:2"
2024-08-23T11:19:49Z	INFO	[vuln] Vulnerability scanning is enabled
2024-08-23T11:19:49Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:49Z	INFO	Need to update the built-in policies
2024-08-23T11:19:49Z	INFO	Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-08-23T11:19:49Z	INFO	[secret] Secret scanning is enabled
2024-08-23T11:19:49Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:49Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:53Z	INFO	Number of language-specific files	num=0
2024-08-23T11:19:53Z	INFO	Detected config files	num=5
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/core-network-services
2024-08-23T11:19:53Z	INFO	[vuln] Vulnerability scanning is enabled
2024-08-23T11:19:53Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:53Z	INFO	[secret] Secret scanning is enabled
2024-08-23T11:19:53Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:53Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:56Z	INFO	Number of language-specific files	num=1
2024-08-23T11:19:56Z	INFO	[gomod] Detecting vulnerabilities...
2024-08-23T11:19:56Z	INFO	Detected config files	num=8
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/core-vpc
2024-08-23T11:19:57Z	INFO	[vuln] Vulnerability scanning is enabled
2024-08-23T11:19:57Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-08-23T11:19:57Z	INFO	[secret] Secret scanning is enabled
2024-08-23T11:19:57Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T11:19:57Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T11:19:58Z	INFO	Number of language-specific files	num=0
2024-08-23T11:19:58Z	INFO	Detected config files	num=4
trivy_exitcode=0

[mikereiddigital]

lg2m

[github-actions]

Trivy Scan Success

Show Output

```hcl

---

Trivy will check the following folders:
terraform/environments/core-network-services
terraform/environments/core-vpc

---

Running Trivy in terraform/environments/core-network-services
2024-08-23T12:02:34Z INFO [db] Need to update DB
2024-08-23T12:02:34Z INFO [db] Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2"
2024-08-23T12:02:36Z INFO [vuln] Vulnerability scanning is enabled
2024-08-23T12:02:36Z INFO [misconfig] Misconfiguration scanning is enabled
2024-08-23T12:02:36Z INFO Need to update the built-in policies
2024-08-23T12:02:36Z INFO Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-08-23T12:02:36Z INFO [secret] Secret scanning is enabled
2024-08-23T12:02:36Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T12:02:36Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T12:02:40Z INFO Number of language-specific files num=1
2024-08-23T12:02:40Z INFO [gomod] Detecting vulnerabilities...
2024-08-23T12:02:40Z INFO Detected config files num=8
trivy_exitcode=0

---

Running Trivy in terraform/environments/core-vpc
2024-08-23T12:02:40Z INFO [vuln] Vulnerability scanning is enabled
2024-08-23T12:02:40Z INFO [misconfig] Misconfiguration scanning is enabled
2024-08-23T12:02:40Z INFO [secret] Secret scanning is enabled
2024-08-23T12:02:40Z INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T12:02:40Z INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T12:02:41Z INFO Number of language-specific files num=0
2024-08-23T12:02:41Z INFO Detected config files num=4
trivy_exitcode=0

</details> #### `Checkov Scan` Success
<details><summary>Show Output</summary>

```hcl

*****************************

Checkov will check the following folders:
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running Checkov in terraform/environments/core-network-services
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-08-23 12:02:44,419 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
2024-08-23 12:02:44,419 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/terraform-aws-observability-platform-tenant?ref=fbbe5c8282786bcc0a00c969fe598e14f12eea9b:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 492, Failed checks: 0, Skipped checks: 186


checkov_exitcode=0

*****************************

Running Checkov in terraform/environments/core-vpc
Excluding the following checks: CKV_GIT_1,CKV_AWS_126,CKV2_AWS_38,CKV2_AWS_39
2024-08-23 12:02:49,505 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-pagerduty-integration?ref=0179859e6fafc567843cd55c0b05d325d5012dc4:None (for external modules, the --download-external-modules flag is required)
2024-08-23 12:02:49,505 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/terraform-aws-observability-platform-tenant?ref=fbbe5c8282786bcc0a00c969fe598e14f12eea9b:None (for external modules, the --download-external-modules flag is required)
2024-08-23 12:02:49,505 [MainThread  ] [WARNI]  Failed to download module github.com/ministryofjustice/modernisation-platform-terraform-member-vpc?ref=3094604aee5121f9794d60d3f27d21d6209e59d4:None (for external modules, the --download-external-modules flag is required)
terraform scan results:

Passed checks: 504, Failed checks: 0, Skipped checks: 159


checkov_exitcode=0

CTFLint Scan Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing "terraform" plugin...
Installed "terraform" (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.5.0)
tflint will check the following folders:
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running tflint in terraform/environments/core-network-services
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/core-vpc
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Trivy Scan Success

Show Output

*****************************

Trivy will check the following folders:
terraform/environments/core-network-services
terraform/environments/core-vpc

*****************************

Running Trivy in terraform/environments/core-network-services
2024-08-23T12:02:34Z	INFO	[db] Need to update DB
2024-08-23T12:02:34Z	INFO	[db] Downloading DB...	repository="ghcr.io/aquasecurity/trivy-db:2"
2024-08-23T12:02:36Z	INFO	[vuln] Vulnerability scanning is enabled
2024-08-23T12:02:36Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-08-23T12:02:36Z	INFO	Need to update the built-in policies
2024-08-23T12:02:36Z	INFO	Downloading the built-in policies...
74.86 KiB / 74.86 KiB [-----------------------------------------------------------] 100.00% ? p/s 0s2024-08-23T12:02:36Z	INFO	[secret] Secret scanning is enabled
2024-08-23T12:02:36Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T12:02:36Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T12:02:40Z	INFO	Number of language-specific files	num=1
2024-08-23T12:02:40Z	INFO	[gomod] Detecting vulnerabilities...
2024-08-23T12:02:40Z	INFO	Detected config files	num=8
trivy_exitcode=0

*****************************

Running Trivy in terraform/environments/core-vpc
2024-08-23T12:02:40Z	INFO	[vuln] Vulnerability scanning is enabled
2024-08-23T12:02:40Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-08-23T12:02:40Z	INFO	[secret] Secret scanning is enabled
2024-08-23T12:02:40Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-08-23T12:02:40Z	INFO	[secret] Please see also https://aquasecurity.github.io/trivy/v0.54/docs/scanner/secret#recommendation for faster secret detection
2024-08-23T12:02:41Z	INFO	Number of language-specific files	num=0
2024-08-23T12:02:41Z	INFO	Detected config files	num=4
trivy_exitcode=0