Initial version of OCI registry as storage #2375

0xavi0 · 2024-04-26T09:06:25Z

Implements an initial proposal to use OCI registries as storage.

NOTE: The feature is only available when the environment variable EXPERIMENTAL_OCI_STORAGE is set to true.

Adds the following new options to the gitrepo yaml:

// when ociRegistry is defined fleet will use oci registry as storage
ociRegistry:
    // url is the OCI registry url.
    url: "docker.io/your-user-here"
    // secret name where the credentials for the OCI registry are.
    // expects a generic secret with username and password keys set.
    authSecretName: oci-secret
    // basicHTTP allows fleet to uses basic http connections to communicate
    // with the registry (defaults to false)
    basicHTTP: false
    // insecureSkipTLS allows connections to the OCI registry
    // without certs (defaults to false)
    insecureSkipTLS: false

New parameters are passed to fleet apply to propagate the values above.

If at least the url value is set, fleet apply will do the following:

Creates a manifest from the resources found in the bundle
Creates and pushes an OCI manifest
Clears the bundle's resource list (contents are in the OCI registry)
Adds the contentID value (this is a new value) in the bundles spec
Creates a secret to pass the OCI registry values to the fleet controller and agents. This secret contains all the values so the controller and agents are able to reach the OCI registry.

The upstream fleet controller then reconciles the just created bundle (remember, with no resources and the content's id set).

When targeting, it also clones the secret containing all the OCI values, just changing the namespace to match the bundledeployment's namespace. (This way the agents will be able to obtain the OCI values) It will also flag the bundledeployment to state that the content is stored in the OCI registry instead to a content resource.

When deploying, the agents will check if the bundledeployment is flagged to has its contents in an OCI registry. If so, it gathers the OCI values it needs to access the registry from the bundledeployment secret and pulls the OCI manifest which is converted back into a fleet manifest.

The agent deploys the manifest just like a regular manifest that was stored as a contents resource.

All secrets created have the owner set to the bundle or bundledeployment so they are properly garbage collected when their owners are deleted.

To be defined / todo / other considerations:

OCI registry garbage collection? (do we need this?)
Sign the OCI manifest when pushing the OCI manifest and check for the signature when pulling and before deploying to avoid back doors. At the moment the code checks that the id annotation is found in the OCI manifest and that it matches the expected content id. We could, maybe, use annotations to add a signature field.
This implementation does not fallback to normal bundle + resource list + content resource if there is any issue connecting to the OCI registry. That's something we could do as bundle keeps the info to know if it is stored in a remote OCI registry or not.

This has been tested with:

zot with basic HTTP
zot with HTTPS connections and insecure (self signed certs)
zot with secure HTTPS (through ngrok)
dockerhub (secure HTTPS)

Refers to: #2114

weyfonk

Thanks for this effort, leaving a few comments :)

e2e/assets/single-cluster/test-oci-simple.yaml

internal/cmd/cli/apply.go

e2e/single-cluster/oci_registry_test.go

internal/cmd/controller/gitops/reconciler/gitjob_controller.go

internal/cmd/controller/reconciler/bundle_controller.go

internal/cmd/agent/deployer/deployer.go

internal/ociutils/ociutils.go

internal/cmd/cli/apply.go

internal/cmd/controller/gitops/reconciler/gitjob_controller.go

manno · 2024-05-28T11:02:43Z

respect EXPERIMENTAL_OCI_STORAGE env var

internal/ociutils/ociutils.go

manno · 2024-06-13T08:25:32Z

internal/cmd/controller/reconciler/bundle_controller.go

+	if !ok {
+		return "", fmt.Errorf("expected data [url] not found in secret: %s", bundle.Spec.ContentsID)
+	}
+	return fmt.Sprintf("oci://%s/%s:latest", string(url), bundle.Spec.ContentsID), nil


We do this a lot, but it might not work with IPv6 addresses? Is there a url module we can use?

I'll investigate...

internal/cmd/cli/apply/apply.go

manno · 2024-06-13T08:30:40Z

internal/cmd/controller/agentmanagement/controllers/resources/data.go

@@ -34,6 +34,11 @@ func ApplyBootstrapResources(systemNamespace, systemRegistrationNamespace string
 					APIGroups: []string{fleet.SchemeGroupVersion.Group},
 					Resources: []string{fleet.BundleDeploymentResourceNamePlural + "/status"},
 				},
+				{


We grant this permission, even if ociutils.ExperimentalOCIIsEnabled is false?

good point.
This approach is passing the env variable to fleet-controller and gitjob (upstream) only.
Do we have a mechanism to pass env variables to the agent?
How can we be sure that the env variable is passed when registration is initiated by the agent?
We could have the case of an upstream supporting OCI and downstream that doesn't... which means the bundledeployments it accesses may not be valid...
Good topic for discussion. :)

It might work if we put it in the agents config.
However, these permissions are limited to the cluster namespace.

.github/scripts/deploy-fleet.sh

dev/setup-fleet

e2e/single-cluster/oci_registry_test.go

Implements an initial proposal to use OCI registries as storage. Adds the following new options to the `gitrepo` yaml: ```yaml // when ociRegistry is defined fleet will use oci registry as storage ociRegistry: // url is the OCI registry url. url: "docker.io/your-user-here" // secret name where the credentials for the OCI registry are. // expects a generic secret with username and password keys set. authSecretName: oci-secret // basicHTTP allows fleet to uses basic http connections to communicate // with the registry (defaults to false) basicHTTP: false // insecureSkipTLS allows connections to the OCI registry // without certs (defaults to false) insecureSkipTLS: false ``` New parameters are passed to `fleet apply` to propagate the values above. If at least the `url` value is set, `fleet apply` will do the following: * Creates a manifest from the resources found in the `bundle` * Creates and push an OCI manifest * Clears the `bundle`'s resource list (contents are in the OCI registry) * Adds the `contentID` value (this is a new value) in the `bundle`s spec * Creates a secret to pass the OCI registry values to the `fleet` controller and `agents`. This secret contains all the values so the controller and agents are able to reach the OCI registry. The upstream `fleet` controller then reconciles the just created `bundle` (remember, with no resources and the content's id set). When targetting, it also clones the secret containing all the OCI values, just changing the namespace to match the `bundledeployment`'s namespace. (This way the agents will be able to obtain the OCI values) It will also flag the `bundledeployment` to state that the content is stored in the OCI registry instead to a `content` resource. When deploying, the agents will check if the `bundledeployment` is flagged to has its contents in an OCI registry. If so, it gathers the OCI values it needs to access the registry from the `bundledeployment` secret and pulls the OCI manifest which is converted back into a `fleet` manifest. The agent deploys the manifest just like a regular manifest that was stored as a `contents` resource. All secrets created have the owner set to the `bundle` or `bundledeployment` so they are properly garbage collected when their owners are deleted. This to be defined / todo: * OCI registry garbage collection? (do we need this?) * Sign the OCI manifest when pushing the OCI manifest and check for the signature when pulling and before deploying to avoid back doors. At the moment the code checks that the `id` annotation is found in the OCI manifest and that it matches the expected content id. We could, maybe, use annotations to add a signature field. This has been tested with: * `zot` with basic HTTP * `zot` with HTTPS connections and insecure (self signed certs) * `zot` with secure HTTPS (through ngrok) * `dockerhub` (secure HTTPS) Refers to: rancher#2114 Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>

In order to use the feature the `EXPERIMENTAL_OCI_STORAGE` environment variable should be `true`. Adds tests for testing without the experimental flag. Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>

manno · 2024-06-27T09:20:45Z

internal/ociwrapper/ociwrapper.go

+}
+
+func newOCIRepository(id string, opts OCIOpts) (*remote.Repository, error) {
+	repo, err := remote.NewRepository(filepath.Join(opts.URL, id))


I find this problematic. We call it a "URL", but it's a simplified URL at best, as it is missing the protocol part, e.g. "oci://".
Later on we have to modify this string to turn it into a URL.
ORAS calls this a "reference": https://oras.land/docs/concepts/reference/

manno · 2024-06-27T13:52:27Z

I can't force push to this branch, conflicts are resolved in #2561

0xavi0 added this to the v2.9.0 milestone Apr 26, 2024

0xavi0 self-assigned this Apr 26, 2024

0xavi0 requested a review from a team as a code owner April 26, 2024 09:06

0xavi0 force-pushed the 2114-oci-registry-initial branch 5 times, most recently from ae8f452 to fdc4b3c Compare April 30, 2024 07:25

weyfonk reviewed May 6, 2024

View reviewed changes

0xavi0 force-pushed the 2114-oci-registry-initial branch from fdc4b3c to 87e640f Compare May 8, 2024 14:21

0xavi0 requested review from manno and weyfonk May 8, 2024 14:33

weyfonk reviewed May 17, 2024

View reviewed changes

internal/cmd/cli/apply.go Outdated Show resolved Hide resolved

internal/cmd/controller/gitops/reconciler/gitjob_controller.go Outdated Show resolved Hide resolved

0xavi0 force-pushed the 2114-oci-registry-initial branch from 87e640f to 883e55b Compare May 17, 2024 14:31

0xavi0 requested a review from weyfonk May 17, 2024 14:37

0xavi0 force-pushed the 2114-oci-registry-initial branch 3 times, most recently from 949df75 to 6ed631f Compare May 24, 2024 14:02

0xavi0 mentioned this pull request May 28, 2024

[SURE-7844] Use OCI repo to store bundle resources #2114

Closed

4 tasks

manno mentioned this pull request Jun 4, 2024

feat: use vals for secret decryption #2408

Closed

0xavi0 force-pushed the 2114-oci-registry-initial branch 2 times, most recently from b2f720f to 0e701d0 Compare June 7, 2024 08:05

0xavi0 mentioned this pull request Jun 7, 2024

Adds extraEnv to gitjob and fleet-controller deployments #2493

Merged

0xavi0 force-pushed the 2114-oci-registry-initial branch 4 times, most recently from 1bdb104 to 84d77f3 Compare June 13, 2024 07:23

manno reviewed Jun 13, 2024

View reviewed changes

weyfonk reviewed Jun 18, 2024

View reviewed changes

0xavi0 force-pushed the 2114-oci-registry-initial branch from 84d77f3 to 8029c2c Compare June 19, 2024 15:36

0xavi0 requested review from manno and weyfonk June 20, 2024 06:03

0xavi0 added 2 commits June 21, 2024 15:13

Enables OCI storage only when experimental flag is set

36d793c

In order to use the feature the `EXPERIMENTAL_OCI_STORAGE` environment variable should be `true`. Adds tests for testing without the experimental flag. Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>

0xavi0 force-pushed the 2114-oci-registry-initial branch from 8029c2c to 36d793c Compare June 21, 2024 13:20

manno reviewed Jun 27, 2024

View reviewed changes

manno mentioned this pull request Jun 27, 2024

Bundle content storage in OCI #2561

Merged

manno closed this Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial version of OCI registry as storage #2375

Initial version of OCI registry as storage #2375

0xavi0 commented Apr 26, 2024 •

edited

Loading

weyfonk left a comment

manno commented May 28, 2024 •

edited

Loading

manno Jun 13, 2024

0xavi0 Jun 19, 2024

manno Jun 13, 2024

0xavi0 Jun 19, 2024

manno Jun 27, 2024 •

edited

Loading

manno Jun 27, 2024

manno commented Jun 27, 2024

Initial version of OCI registry as storage #2375

Initial version of OCI registry as storage #2375

Conversation

0xavi0 commented Apr 26, 2024 • edited Loading

weyfonk left a comment

Choose a reason for hiding this comment

manno commented May 28, 2024 • edited Loading

manno Jun 13, 2024

Choose a reason for hiding this comment

0xavi0 Jun 19, 2024

Choose a reason for hiding this comment

manno Jun 13, 2024

Choose a reason for hiding this comment

0xavi0 Jun 19, 2024

Choose a reason for hiding this comment

manno Jun 27, 2024 • edited Loading

Choose a reason for hiding this comment

manno Jun 27, 2024

Choose a reason for hiding this comment

manno commented Jun 27, 2024

0xavi0 commented Apr 26, 2024 •

edited

Loading

manno commented May 28, 2024 •

edited

Loading

manno Jun 27, 2024 •

edited

Loading