operator: Add support for custom S3 CA #6198

periklis · 2022-05-19T12:37:10Z

What this PR does / why we need it:
The following PR adds support to define a custom CA for object storage type S3. Custom CAs are useful for third-party S3-compatible solutions (e.g. Minio, Rook, OpenShift Data Foundation).

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

Documentation added
Tests updated
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

grafanabot · 2022-05-19T12:42:15Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-19T12:43:11Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-19T12:48:03Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-19T13:21:03Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-24T12:16:58Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental

Just tested on a cluster with ODF, seems to work fine. 👍

operator/api/v1beta1/lokistack_types.go

operator/internal/handlers/internal/storage/ca_configmap.go

operator/internal/manifests/storage/options.go

operator/internal/manifests/storage/configure.go

grafanabot · 2022-05-24T17:26:03Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-30T11:12:23Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental

Nearly there, I'd say. I'll also give it a test run again.

operator/api/v1beta1/lokistack_types.go

operator/internal/manifests/storage/options.go

operator/internal/manifests/object_storage.go

grafanabot · 2022-05-30T17:48:00Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-30T18:06:16Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

operator/internal/manifests/object_storage_test.go

operator/internal/handlers/internal/gateway/tenant_secrets.go

grafanabot · 2022-05-31T11:40:03Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-01T11:44:30Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-01T11:53:25Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

sasagarw

Small comments/errors text correction

operator/internal/handlers/internal/gateway/tenant_secrets.go

operator/internal/handlers/internal/gateway/tenant_secrets_test.go

operator/internal/handlers/lokistack_create_or_update.go

Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

…t.go Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

grafanabot · 2022-06-01T12:40:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-01T12:41:04Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-01T12:41:06Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

xperimental

Tested using ODF 👍

periklis added the sig/operator label May 19, 2022

periklis requested a review from xperimental May 19, 2022 12:37

periklis requested a review from a team as a code owner May 19, 2022 12:37

periklis self-assigned this May 19, 2022

pull-request-size bot added the size/XL label May 19, 2022

periklis force-pushed the aws-s3-service-ca branch from e097502 to d81525c Compare May 19, 2022 13:16

periklis added 13 commits May 24, 2022 14:11

Add object storage ca configmap name field

a1f18c8

Mount CA configmap into loki components

e7c5eed

Pin ca file name to service-ca.crt

9c5ac89

Fix configmap pull in handler

199f26e

Fix error checks for CA configmap pull

1d28c9d

Fix error handling

6611619

Use name instead of confimap

9122ea8

Move gateway secret extract func to gateway package

badfc48

Add CA configmap validation

ebb70b4

Add changelog entry

c7ab65d

Fix linter

f6e9d24

Fix merge conflicts

8458418

Reorder changelog

149149d

periklis force-pushed the aws-s3-service-ca branch from d81525c to 149149d Compare May 24, 2022 12:12

xperimental requested changes May 24, 2022

View reviewed changes

Fix review comments

0becf3f

Move CAName to TLS config spec

95a379d

pull-request-size bot removed the size/XL label May 30, 2022

xperimental reviewed May 30, 2022

View reviewed changes

periklis added 2 commits May 30, 2022 19:20

Fix LokiStack reference in docs

b3ae52f

Move storage configure functions to storage package

fc3908c

Fix LokiStack reference in docs

1fd6bd5

xperimental reviewed May 31, 2022

View reviewed changes

operator/internal/manifests/object_storage_test.go Show resolved Hide resolved

operator/internal/handlers/internal/gateway/tenant_secrets.go Outdated Show resolved Hide resolved

periklis added 2 commits May 31, 2022 13:27

Fix type docs

31cd9d2

Refactor storage/gateway secret map calls

8932518

Move object storage tests to package storage

94e41fd

Fix tenant/storage secret key checks for emptyness

e31736b

periklis enabled auto-merge (squash) June 1, 2022 11:56

sasagarw reviewed Jun 1, 2022

View reviewed changes

operator/internal/handlers/internal/gateway/tenant_secrets.go Outdated Show resolved Hide resolved

operator/internal/handlers/internal/gateway/tenant_secrets_test.go Outdated Show resolved Hide resolved

operator/internal/handlers/lokistack_create_or_update.go Outdated Show resolved Hide resolved

periklis and others added 3 commits June 1, 2022 14:33

Update operator/internal/handlers/internal/gateway/tenant_secrets.go

b2d1970

Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

Update operator/internal/handlers/internal/gateway/tenant_secrets_tes…

5a9cca4

…t.go Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

Update operator/internal/handlers/lokistack_create_or_update.go

8b3bfa4

Co-authored-by: Sashank Agarwal <sashankagarwal97@gmail.com>

xperimental approved these changes Jun 1, 2022

View reviewed changes

periklis merged commit d91a64f into grafana:main Jun 1, 2022

periklis mentioned this pull request Jun 8, 2022

operator: Storage schema update #5952

Merged

3 tasks

mathieu-pousse pushed a commit to mathieu-pousse/loki that referenced this pull request Jun 13, 2022

operator: Add support for custom S3 CA (grafana#6198)

3334bab

This was referenced Jun 28, 2022

dannykopping/remove cache stats dannykopping/loki#13

Closed

dannykopping/remove cache stats dannykopping/loki#14

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: Add support for custom S3 CA #6198

operator: Add support for custom S3 CA #6198

periklis commented May 19, 2022 •

edited

Loading

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 24, 2022

xperimental left a comment

grafanabot commented May 24, 2022

grafanabot commented May 30, 2022

xperimental left a comment

grafanabot commented May 30, 2022

grafanabot commented May 30, 2022

grafanabot commented May 31, 2022

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

sasagarw left a comment

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

xperimental left a comment

operator: Add support for custom S3 CA #6198

operator: Add support for custom S3 CA #6198

Conversation

periklis commented May 19, 2022 • edited Loading

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 19, 2022

grafanabot commented May 24, 2022

xperimental left a comment

Choose a reason for hiding this comment

grafanabot commented May 24, 2022

grafanabot commented May 30, 2022

xperimental left a comment

Choose a reason for hiding this comment

grafanabot commented May 30, 2022

grafanabot commented May 30, 2022

grafanabot commented May 31, 2022

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

sasagarw left a comment

Choose a reason for hiding this comment

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

grafanabot commented Jun 1, 2022

xperimental left a comment

Choose a reason for hiding this comment

periklis commented May 19, 2022 •

edited

Loading