Upgrade procedure for elemental-operator

[davidcassany]

We need to clarify the procedure. No big deal, should be around helm common practices. However there are few considerations:

• How the rancher admin know there is an update available?
• How is the latest version set for OCI registries, is that always latest? semver sort? alpabetical sort? latest changes sort? Should we actually use latest tag? using same image references for upgrades tends to be confusing. I'd prevent the use of latest tag in case helm is capable to sort with some criteria the available tags.
• Is there any way to handle version constraints (for instance upgrading to X requires current version to be higher or equal than Y)
• Is there a way in elemental operator to make explicit the oldest elemental operator client (elemental-register) that is compatible with? How those cases should be handled?

Last but not least, we should discuss the essential tests for an upgrade acceptance criteria on each release:

• New nodes can still be bootstrapped after upgrading with helm upgrade

[davidcassany]

@agracey @rancher/elemental any input and thoughts about this topic would be appreciated.

[frelon]

How the rancher admin know there is an update available?

Maybe we could add another step to the current automatic upgrade procedure where a new update has to be approved before being applied. Then that approval could go through the UI and maybe trigger a notification in rancher.

How is the latest version set for OCI registries, is that always latest? semver sort? alpabetical sort? latest changes sort? Should we actually use latest tag? using same image references for upgrades tends to be confusing. I'd prevent the use of latest tag in case helm is capable to sort with some criteria the available tags.

Yep, I'd also argue against using latest as a tag. I think semver is a great option and then use the same version when generating the helm-chart in order to not have any question of what operator version is used in a chart.

Is there any way to handle version constraints (for instance upgrading to X requires current version to be higher or equal than Y)

I have only seen this for kubernetes-versions which is not very useful for our case.

Is there a way in elemental operator to make explicit the oldest elemental operator client (elemental-register) that is compatible with? How those cases should be handled?

It would be quite easy to add this to the current operator version-negotiation, but we would need to manually bump this and keep track of the latest version supported..

[davidcassany]

How the rancher admin know there is an update available?

Maybe we could add another step to the current automatic upgrade procedure where a new update has to be approved before being applied. Then that approval could go through the UI and maybe trigger a notification in rancher.

I am not following you here. I mean upgrades on elemental-operator, not on downstream clusters. I am just wondering how are we going to notify users if there is a security fix on elemental-operator and hence a new helm chart version is released. I'd say this is or should be a Rancher plugin feature, not something to be managed by Elemental.

How is the latest version set for OCI registries, is that always latest? semver sort? alpabetical sort? latest changes sort? Should we actually use latest tag? using same image references for upgrades tends to be confusing. I'd prevent the use of latest tag in case helm is capable to sort with some criteria the available tags.

Yep, I'd also argue against using latest as a tag. I think semver is a great option and then use the same version when generating the helm-chart in order to not have any question of what operator version is used in a chart.

In fact I saw in helm docs that any current chart hosted in OCI registries must be tagged with a semver version. Hence there is no latest and the tag matches the chart version. So I understand it just pulls the highest one available if no version provided. I think there is nothing to care about on our side on that regard.

Is there a way in elemental operator to make explicit the oldest elemental operator client (elemental-register) that is compatible with? How those cases should be handled?

It would be quite easy to add this to the current operator version-negotiation, but we would need to manually bump this and keep track of the latest version supported..

Indeed I guess there is no need for that at the moment. We should just focus on being backward compatible for now.

In fact I'd say beyond setting a proper testing matrix elemental-operator upgrades I'd say we are done. Probably we could reuse the same tests for elemental-teal upgrades (rancher/elemental#651) and also upgrade and test elemental-operator in that context.

[fgiudici]

I agree with the proposals in the comments, which I try to recap (correct me if I'm wrong):

* How the rancher admin know there is an update available?

This is something mainly related to the Rancher plugin functionality (helm chart upgrade available). Maybe just check if the operator has to provided something more to have the notification pop-up in rancher, especially for security updates.

* How is the latest version set for OCI registries, is that always `latest`? semver sort? alpabetical sort? latest changes sort? Should we actually use `latest` tag? using same image references for upgrades tends to be confusing. I'd prevent the use of `latest` tag in case helm is capable to _sort_ with some criteria the available tags.

No "latest", "semver" is the way to go.

* Is there any way to handle version constraints (for instance upgrading to X requires current version to be higher or equal than Y)

We will not put any constraints on versions (at least for now).

* Is there a way in elemental operator to make explicit the oldest elemental operator client (elemental-register) that is compatible with? How those cases should be handled?

We can have a support matrix for this, but for now let's just try to keep compatibility. (I would propose: let's ensure 1 stable version backward compatibility. Older clients would be supported best effort but not tested nor guaranteed).

Last but not least, we should discuss the essential tests for an upgrade acceptance criteria on each release:

* New nodes can still be bootstrapped after upgrading with `helm upgrade`

We need a dedicated card for the tests to link here.

[kkaempf]

See #376 - we'll rely on Rancher to track (and upgrade) operator versions

[ldevulder]

we'll rely on Rancher to track (and upgrade) operator versions

And how the upgrade with the new version of the operator will be validated?

[kkaempf]

Rancher marketplace relies on Helm.
We should have a CI test that installs from Stable and upgrades to Staging via Helm.
If time/resources permit, we could also upgrade from Stable to Dev but that's of lower priority imho.