Refactoring Docker Image Versioning and Build System

[soenkeliebau]

The current purely tag based system for tracking product and image versions does not allow us to update older product images with newer base images, without also upgrading the product version.

Goal

We implement a system that supports upgrading base images for older product images. This system should be easy to use without much repetitive manual work.

New Branch System

In the new branching system, we will maintain a release branch per platform release, i.e. a branch release-22.09, release-22.11.
On these branches we will tag platform versions, i.e. 22.11.0-rc1, 22.11.0, 22.11.1.

As we now tag a whole platform release instead of individual products, the build system/GitHub actions should build all images from these tags.

Next Steps

• Update all images on the main branch #236
• Implement new Branch System for Docker Images #235

Roadmap

As this issue turned out to be more complex than anticipated, we decided to divide it into multiple stages.

Keep our next Release current under the new Scheme

• Update all images on the main branch #236
• Implement new Branch System for Docker Images #235

LTS Strategy and Upgrading older Images

Do we want to upgrade base images for all previous release, all the time, indefinitely? This is to be decided. Upgrading older product images is not strictly necessary for the next release, so we decoupled this.

• Devise an LTS strategy for our Docker Images #237
• Upgrade Docker images for older Versions according to LTS strategy #238
• Automate manual Steps #239

Lastly, manual steps should be automated as best we can.

---

Old Ticket Content

Description

As part of our release tracking epic we have an item to update the ubi8 base image version in a new release cycle.
As this affects all supported product images this can potentially require quite a bit of manual effort and we think is worth creating an automation for.

Since releasing an image is done automatically upon pushing a tag, all this automation needs to do is retrieve existing tags from the repository, filter and mutate these tags to get a list of new tags that represent the updated images and push these tags.

It probably makes sense to write a python script for this instead of a bash script, as we'll need to parse semver at least in the filtering step.

Update

I have thought on this a bit more and am afraid that it is more complex than I initially wrote in this ticket / I forgot to mention a part of the issue.
When I write "push tags" we of course need to also identify the correct commit to push these tags to, and that cannot be HEAD because then we would include all current changes to the image in older versions as well.

For example we have kafka3.3.0-stackable0.7.0 as current image and also kafka3.3.0-stackable0.4.2 as an older version, if we now tag HEAD as kafka3.3.0-stackable0.4.3 then that image will suddenly include all changes that were made between 0.4.x and 0.7.x as well and potentially break compatibility.

So we'll nead release branches for image versions as well and then cherrypick the commit that contains the base image update to those branches.

This is done when

• An action has been commited to the repository that allows re-releasing all images

Example

Step 1 - Get tags

To retrieve the tags something like this migth be done, which filters on a regex that pretty much just requires a stackable version to be present.

➜  docker-images git:(main) ✗ git tag -l | grep -E ".*-stackable[0-9]+\.[0-9]+\.[0-9]+"
airflow-stackable0.2.0
antora-stackable0.1.0
hadoop3.3.3-stackable0.1.0
hbase-stackable0.4.0
hbase2.4.11-stackable0.7.0
hbase2.4.12-stackable0.1.0
hbase2.4.12-stackable0.2.0
hbase2.4.6-stackable0.7.0
hbase2.4.8-stackable0.7.0
hbase2.4.9-stackable0.7.0
hive2.3.9-stackable0.5.0
hive3.1.3-stackable0.1.0
java-base-stackable0.2.1
kafka-stackable0.3.0
kafka3.2.0-stackable0.1.0
nifi1.15.1-stackable0.1.0
nifi1.15.2-stackable0.1.0
nifi1.15.3-stackable0.1.0
nifi1.16.0-stackable0.1.0
nifi1.16.1-stackable0.1.0
nifi1.16.2-stackable0.1.0
nifi1.16.3-stackable0.1.0
opa0.41.0-stackable0.1.0
pyspark-k8s3.3.0-stackable0.1.0
pyspark-k8s3.3.0-stackable0.2.0
spark-k8s-stackable0.2.0
spark-k8s-stackable0.3.0
spark-k8s3.3.0-stackable0.1.0
spark-k8s3.3.0-stackable0.2.0
superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.0
superset-stackable2.0.1
superset-stackable2.1.0
superset1.5.1-stackable0.1.0
superset1.5.1-stackable0.2.0
testing-tools0.1.0-stackable0.1.0
tools0.2.0-stackable0.3.0
trino387-stackable0.1.0
trino395-stackable0.1.0
trino396-stackable0.1.0
zookeeper-stackable0.4.0
zookeeper-stackable0.7.1
➜  docker-images git:(main) ✗

Step 2 - Filter tags

Taking the superset images as an example we now need to remove images for which a subsequent version already exists.

superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.0
superset-stackable2.0.1
superset-stackable2.1.0

In this case that would be 'superset-2.0.0' which is superceded by 'superset-2.0.1'.
So that would leave us with this list:

superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.1
superset-stackable2.1.0

Step 3 - Create new tags

Now we increase the patch level for every tag:

superset-stackable0.1.1
superset-stackable0.2.1
superset-stackable0.3.1
superset-stackable1.0.1
superset-stackable2.0.2
superset-stackable2.1.1

Step 4 - Push tags

Push all tags that were just created.

[nightkr]

I started discussing this with @soenkeliebau last week (before he ran off on vacation) and ultimately it seems like this is a slightly bigger issue than we anticipated at first, since our current repository structure isn't set up to accomodate maintaining multiple image branches.

Maybe it would make sense to split this into one issue for just doing patch releases for the current release of each product, and a separate one for figuring out a sensible LTS-compatible branching strategy?

[soenkeliebau]

@fhennig and I will refine this together tomorrow

[fhennig]

Notes from a refinement meeting with Sönke and Teo

Some Notes

• Release branches sound like a good idea
• Not all versions are tagged correctly at the moment, this needs fixing

Branching strategy

• Make a branch for every platform release (i.e. 22.09, 22.11)
  • new versions of products can be added to older releases by adding them to the conf.py in the platform release branch
  • ubi8 images for older platforms can also be updated to new versions in that branch

Steps

Currently, this ticket is part of the epic to update versions for the upcoming release. We need to decide what to do now, and what to do later.

For this release 22.11

• update base image
• introduce versioning for the java base image
  • basically treat different java versions as different products, and version them with our platform version instead
• make a branch for the 22.11 release
• tags will be 22.11.0, 22.11.1, 22.11.0-rc1 etc.
  • the build action should be modified to build all product images from that tag

In the Future

• make branches for previous platform release 22.09 as a "spike" to see if what we came up with works
  • update ubi8 images etc.
• come up with an LTS strategy

More thoughts

• if we would version operators with the platform release as well, this would simplify image selection
• nicer caching

[fhennig]

@lfrancke Sönke, Teo and I did a refinement, and I adapted the issue content above accordingly. This ticket turned out larger than expected, and so now it became an epic. Two of the tasks are immediately relevant to the next release, which ahve been refined and put into the "Waiting for acceptance column" the other tickets still need refining, but are also not immediately relevant for the next release.

I'm tagging you to move the tickets around the board as you see fit 😄 (especially this epic, I'm not sure where to put it)

[lfrancke]

Thank you, will do tomorrow!

But this is also "done" with refinement? It's still marked as "In Progress"

[fhennig]

it is done for now, but since some of the tasks this epic contains are still not refined, the epic might need re-refining at some point as well. I think what I'm saying is, that maybe the "Epic" type doesn't fit this workflow very well. I think we could move it into "Track" or back to "Ideas" I think.

[sbernauer]

Hi ho, you had a meeting yesterday, what was the outcome? Is the Issue description decided and up to date?

[lfrancke]

No, this will require more time.
We might bump this but haven't made a decision yet, sorry.

[sbernauer]

I sadly couldn't participate in the meeting and have concerns after speaking with @Maleware about this. Happy to have a follow-up meeting :)
Honestly i would not wait further on this with the product image selection and get that out of the door if there are no objections

[adwk67]

Closing this as replaced by #260 (which covers LTS and the upgrading of older images issues only). New release-branches are covered by stackabletech/issues#309.