Propose pre-release (RC) as part of our process

[alenkacz]

What this PR does / why we need it:
Just a proposal to make RCs part of our release process

[kensipe]

I agree we should make some improvements... but this doesn't make sense to me

the tests that need to confirm a release should be done on a snapshot and confirmed prior to a release. The value of an RC isn't clear to me if we capture the testing of a snapshot prior to release.

[alenkacz]

I agree that we need to have better test coverage but we'll never have 100% coverage and that same coverage for all operators. As we're a community project I see a big advantage on giving a community a heads up and a build that they can run and test that everything they have still work. We could do the same for our operators/scenarios we know we don't have yet covered by automated tests while improving the coverage in general.

But I am open to any counter-proposals. In the past RCs were a great way how to reduce post-release bugs as the bugs were caught while testing the RC.

[ANeumann82]

I do agree with @alenkacz. I think RCs would add value:
a) More familarity with release process
b) Community can use the RCs to do their own testing. Might not be the biggest use case right now, but community won't start testing with RCs unless we have them. I think starting this process early allows the community to get used to having them.
c) Our TC tests for C* and Kafka use releases for testing. It would be possible to change that, but we would have to put the artefacts for snapshots somewhere.
b) Having RCs is a bigger mental stop for last-minute changes: It feels more like a serious "release" point than a snapshot.

I do agree that we need to do more comprehensive testing on snapshots though.

[zmalik]

RC should be made available to the community, but we cannot make sure they test them. The only point of making available RC is testing from the user's perspective and in user's infra (kubernetes distro etc). In this case, operator developers, testing it in GKE or D2iQ distros of Kubernetes.

Right now we have the same team also developing operators, this might feel that this has to converge but I think it should be separate.

makes sure it's manually tested at least in the parts where we lack automatic tests

So this part can be only ensured partially. As can be quite broad. I know we will test it as we are the operator developers also. But as the community grows this might dilute. What about just a pre-release 2 days before the release process and making it available to the community?

And the operator developers team, try to provide feedback via public channels around the RC. As for now we can make compulsory to get feedback from kafka/cassandra/spark on a specific kubernetes distro.

[meichstedt]

To me, the value of an RC is that users can upgrade and report issues based on a defined artifact. Operator developers (ourselves!) should verify that e.g. KUDO Kafka can be installed and operated on 0.11 before we release that KUDO version (or, we release the version with a guide that explains what needs to be adjusted in order to make it work).

Re snapshots vs RCs: I personally see the value of an RC as a matter of requesting input from users on that particular version. So, sure, people can use snapshots. But which one, and when? Has the functionality already been merged? And, would we want issues to be created against snapshots? Now here's a thin line: if a snapshot's artifacts are provided in a meaningful way, and it's communicated that a certain snapshot shall be used for integration testing, then that's as good as an RC to me.

Concluding, I think this is a step in the right direction but I'd rephrase a bit. Something like

RM prepares a pre-release 3 days before the release process is started and requests feedback from users.

3 days, to grant

• operator developers a timeframe of 24 hours to test an operator with the RC and report issues,
• a 24 hour timeframe for us to fix any reported issues and cut a new RC that can be tested,
• and another 24 hour timeframe for operator developers to confirm.

[kensipe]

After some continued thoughts around this... If the majority believes this is a valuable change I feel it is missing important aspects of the process. For instance... if we are adding a number of new features in version X and we want to put out an X-rc1. Do we pull together release notes and release highlights for the RC? or for GA releases? If we highlight features on RCs, do we repeat them for the GA?

[ANeumann82]

Good questions Ken!

As we should treat RCs as (almost) releases, I think we should prepare Release Notes and Highlights already, I'm not sure if they should be in the RC release page.

On the RC release page, there should be at least:

• Changelog since last release
• Maybe Highlights in One-Sentence Description?

On The GA release page there should be:

• Changelog since RC (Or since last GA? I think since previous release makes more sense)
• Full Highlights with examples, etc.

[ANeumann82]

I think we've now done pre-releases a couple of times... Maybe we can just update the document based on what we currently do?

[alenkacz]

@ANeumann82 And is that what this PR proposes or not? I still kind of feel it is... ? 🤔

[ANeumann82]

@ANeumann82 And is that what this PR proposes or not? I still kind of feel it is... ? 🤔

@alenkacz It is. I mostly meant the issues @kensipe mentioned. I have added two notes that clarify the process that we seem to be following right now, I think that should be enough.