[RFC] Quarterly Releases

[driazati]

Cross-posted to discuss here: https://discuss.tvm.apache.org/t/rfc-quarterly-releases/12629

cc @areusch @leandron @Mousius

[areusch]

cc @apache/tvm-committers

[Mousius]

Thanks for formalising this RFC @driazati, this is a great improvement to how releases are currently done in TVM. I've highlighted some concerns but the sooner we start doing this the sooner we'll be able to figure out how to minimize the friction of releasing 😸

[Mousius]

Just clarifying, I assume that even though the features themselves are independent of the release we'd eventually want to pick a versioning scheme that reflects the changes made?

[driazati]

I'm imagining it as 0.9 -> 0.10 -> 0.11, etc until we decide to 1.0, I added some info about this in the reference section

[Mousius]

Regarding "As of this RFC it has been five months since the last release", we could've released far earlier but there was some discomfort around evolving the relay.build API multiple times - with these regular releases, are we suggesting that we can evolve the API over a number of releases?

[driazati]

We certainly can change APIs over releases, though maintaining as much backwards compatibility and consistency as we can is good. @areusch and I discussed this specific issue and we were thinking we'd wait until that is resolved to start doing this RFC, I added another part to the reference about this issue specifically

[Mousius]

Doesn't this RFC just provide a schedule for following the release process? The release candidate vote thread is part of that process and ensures we don't mistakenly ship a broken release, why do we want to abolish it as part of this rather than incorporate it into the two week review window?

[tqchen]

Thanks @driazati for proposing the change and bringing up the goal of making release streamlined (which we all share).

Just to clarify this particular point(of release voting) as it is something that is related to the policy. Overall release policy is something that apache handles (extra) carefully. Release voting thread is mandatory per policy and it is not something that we can change project-wide.

The rationale is that the release voting thread brings broad community awareness and overall license requirements. A release needs at least three PMC vote to be shipped. We can however, keep the schedule streamlined and improve the overall efficiency driven by a release manager.

The original release policy guide is a good reference pt -- about the GPG key needs and where to upload, posts.

[driazati]

Ah, I wasn't aware of the rules on the vote thread. I think it is fine to keep in that case but move it to the end of the process, so the release manager does most of the work and testing but sends it to the PMC for approval before publishing anything

[tqchen]

no need to explicit send to PMC as PMC members normally vote on the same public voting thread

[Mousius]

Interestingly, the Kafka document states that if noone is interested they just don't release - that seems a fair way to decide to skip the release and come back in 3 months rather than stalling it which could create the scenario we have now.

[tqchen]

I agree that skip the release would be a good lightweight option for now. We can also do out of schedule release if necessary if there is a strong need to do so. But the goal is to keep a regular cadence where things happens most of the time

[driazati]

good notes, I added a small section about this to the reference. Practically it makes sense as well, since there are always people in the loop deciding whether to proceed or not

[u99127]

A project need releases so that end users have a way of consuming the work done since the last release including bug fixes, no releases means that as a community we are saying that end users don't need to consume the work done in the last 3 months including fixes for bugs and it lessens the value of releases and we leave it to people checking out random git SHAs and working from that. Not very sustainable for end users.

For the TVM community to be more end user focussed releases need to happen on a cadence rather than it being dependent on someone's interest. If there is no volunteer , maybe the PMC "volunteers" someone ?

[tqchen]

would be useful to clarify what is the "official release". The official release is the source code tarball posted to the TVM webpage download section(with the corresponding GPG keys that can be verified) and the apache svn repo.

The binary packages, and source code tag on github are "convenient by product" that is not part of the official release. I know it sounds a bit strange. ASF have a quite pedantic approach towards release and licensing. The main retionale is that for people who want to be extra "safe" on licensing implications, they want to go and download from the webpage -- where the source code comes clean without binary dependencies and things can be built from scratch. This pedantic view, however, is also why many organizations trust apache software (e.g. they can be applied without review)

Of course for most people the ability to quickly use and try things out is more important. So it is mainly a terminology concern, but does not mean these "convenient byproducts" are not important.

[driazati]

This list has both uploading the source to GitHub and Apache as a task for the release, do we just need to clear on GitHub that the officially released code can be found on the Apache site?

[tqchen]

we can do that in release note, but attaching the signature and sha is also fine in github.

[tqchen]

Let us add a dependency license check to quickly verify the new dependencies added since last release, and ensure license files are inplace and they are compatible with ASF policy

[tqchen]

A vote period(at least three days) would be needed on the release candidate. I would suggest keep the original schedule, feedback period(one week) and vote period(one week).

The feedbacks are based on the rc tarball(with GPG keys and sha512) to be verifiable to avoid possible errors during the release shipping(e.g. mistakenly packge in something or ignores something into the source tarball).

Changing a rc during voting period is quite cumbersome. As the original vote thread needs to be abandoned. A new rc tarball needs to be generated with an updated link, then open a new vote thread for the new rc. The feedback period would help to reduce possible problems here. A longer period would also help give general visibility to the community.

[leandron]

Perhaps we should also have a deputy, who's aware about what is going on, and in the absence of the release manger can take over.

[driazati]

agreed that's a good idea but I don't think we need to explicitly codify that, if there is a problem with the release manager's time they should communicate that to the community and figure something out from there

[hogepodge]

To help mitigate with this, several projects stagger release managers, with an outgoing release manager and an incoming shadow release manager. This spreads out the responsibility and gives an opportunity to formally train someone on release management duties.

[u99127]

@driazati - this is a really good starting point for releases and I'm very glad this is coming together.

I had a ponder last night.

A few additional points to consider for an enhancement to this RFC or make it clearer here in this RFC itself.

• Lifecycle of a release , what happens to a release x.y.0 after the release point ? The community has 2 choices, use frugal release manager resources in making sure the next (x.y.0)+1 release happens on schedule and on time . Or continues to improve the quality of x.y.0 by fixing bugs and making sure that x.y.1 / x.y.2 keep happening - but then after a year we'll have too many releases and given where we are in the lifecycle of TVM, this may not be necessary.
• At what point do we declare an old release something that the community will not support further ? i.e. x.y.0 is valid for 6 months or a year after which there will be no releases from the branch and the community will not take any bug fixes into the branch ?
• Are release managers appointed for the life cycle of a 0.x release - i.e is it the responsibility to keep 0.x until its end of life ?

[hogepodge]

This looks like semver, and will likely be interpreted as semver, but is not. I would strongly encourage considering officially adopting and adhering to semver to help manage API expectations for groups trying to use TVM in production. Time-based releases don't always fit perfectly with semantic versioning, but careful consideration of API changes and additions should make adherence possible. https://semver.org

[hogepodge]

Another alternative is adopting a date-based version number such as year.quarter.minor, which will meet the needs for resolving dependencies but does not imply the strict API compatibility requirements across releases that semver has.

[areusch]

i agree we are not semver here. however, we are also not 1.0, so it might be worth reconsidering our versioning scheme once we do hit 1.0. I'd like to see some more infrastructure and processes developed (e.g. feature flags and a more codified set of expectations for large projects like metascheduler which are currently developed in-tree and merged in an effort-specific way) before we can commit to semver.

we could consider e.g. 0.202206.0, though it's a bit hard to parse. i don't think we should propose to rev the major version past 1.0 at this time, though.

[driazati]

agree semver is probably too much for us right now and we should revisit at 1.0. Increasing version numbers seem simplest to think about which is what this RFC proposes and anecdotally date releases aren't as common for Python packages (though they are listed in PEP 440)

[areusch]

thanks @driazati ! left some comments

[areusch]

i agree we are not semver here. however, we are also not 1.0, so it might be worth reconsidering our versioning scheme once we do hit 1.0. I'd like to see some more infrastructure and processes developed (e.g. feature flags and a more codified set of expectations for large projects like metascheduler which are currently developed in-tree and merged in an effort-specific way) before we can commit to semver.

we could consider e.g. 0.202206.0, though it's a bit hard to parse. i don't think we should propose to rev the major version past 1.0 at this time, though.

[areusch]

shall we add a note about contributors testing the release on real hardware?

[driazati]

added a short note about it but I don't want to expand the scope of this RFC too much, specific release testing instructions should be in https://tvm.apache.org/docs/contribute/release_process.html somewhere

[areusch]

@u99127 these are great points! i have some thoughts:

Lifecycle of a release , what happens to a release x.y.0 after the release point ? The community has 2 choices, use frugal release manager resources in making sure the next (x.y.0)+1 release happens on schedule and on time . Or continues to improve the quality of x.y.0 by fixing bugs and making sure that x.y.1 / x.y.2 keep happening - but then after a year we'll have too many releases and given where we are in the lifecycle of TVM, this may not be necessary.

It seems like there may be a difference between a 1.0 release and a 0.x release in terms of support. Perhaps a 0.x release is patch-supported for a short time (e.g. until the next release cycle starts)?

At what point do we declare an old release something that the community will not support further ? i.e. x.y.0 is valid for 6 months or a year after which there will be no releases from the branch and the community will not take any bug fixes into the branch ?

yeah this is a great question. it seems like right now our policy is to "support" the previous release and no further back. also something i'd expect to grow or perhaps we'd arrive at an LTS-style support policy depending on development cadence.

Are release managers appointed for the life cycle of a 0.x release - i.e is it the responsibility to keep 0.x until its end of life ?

it could be good to state that a release manager is expected to shepherd questions (though the entire set of committers and community is expected to try and answer without the need for shepherding) about the release on the forum. i'd propose we start with a shorter support period so it is not overly daunting to be a release manager.

[driazati]

I agree with @areusch that a lot changes pre and post 1.0. This RFC's main focus is getting code out the door often, so with that I think it makes the most sense to only support the current version and only publish a new release mid-cycle for critical bug fixes and security issues.

Ostensibly the release manager should be responsible for the release they published (and in the best case this would mean a release needs no patches and thus requires no extra work), but this could get passed off or reassigned as we move along based on who's available.

[areusch]

We discussed this RFC briefly this morning at the Community Meeting. Here are notes:

• @Mousius notes that because we're 0.x version numbers, we don't need to worry too much about breaking APIs
• @driazati notes that many of our APIs are stable and we should still think about this, but that we shouldn't spend a ton of time on those issues and should get releases out the door.

[areusch]

I think the main outstanding thing is the question of support here. I'd love for a few more folks to weigh in, tagging a few folks who may have ideas: @tqchen @jroesch @kparzysz-quic @u99127 @Mousius @leandron @comaniac @zhiics @Hzfengsy @ZihengJiang @yzhliu @masahi @icemelon

broadly, the question is: once we release version 0.X.0, should we discover a bug, we should likely fix it on that branch and on main and release 0.X.Y. how long shall we commit to doing that? If a bug is discovered on 0.X-1.0, will we fix it?

my vote is the following (for now, we can revisit at 1.0):

• we agree we will fix major bugs on 0.X.Y where 0.X.Y is the latest release.
• when we make a new minor version release e.g. 0.(X+1).0, we will stop maintaining releases starting with 0.X. at that time.
• committers/pmcs should exercise their judgement when determining if something warrants a fix.
• the same policy should apply on the forum--if someone asks a question on an old minor release version, it should be acceptable to ask them to reproduce it on the latest release.

[Mousius]

my vote is the following (for now, we can revisit at 1.0):

• we agree we will fix major bugs on 0.X.Y where 0.X.Y is the latest release.
• when we make a new minor version release e.g. 0.(X+1).0, we will stop maintaining releases starting with 0.X. at that time.
• committers/pmcs should exercise their judgement when determining if something warrants a fix.
• the same policy should apply on the forum--if someone asks a question on an old minor release version, it should be acceptable to ask them to reproduce it on the latest release.

Just checking for my understanding, this would be:

• Release 0.9.0 next
• We find a bug, release 0.9.1
• 0.9.1 is now the only supported release
• 0.10.0 comes out
• 0.10.0 is now the only supported release

Who does the patch releases? I'm assuming the nominated release person from 0.9.0 ?

[areusch]

@Mousius that's my suggestion. i don't know who should do it--perhaps at least the nominated person is responsible for finding someone to do it and ensuring it's done?

[driazati]

thanks for summarizing the meeting! I agree with @areusch's points on support and have added some text about it

[Mousius]

@Mousius that's my suggestion. i don't know who should do it--perhaps at least the nominated person is responsible for finding someone to do it and ensuring it's done?

I'd suggest we make it clear it may be required when taking the role, but equally if they find someone to swap with then that's ok 😸

[u99127]

I think the main outstanding thing is the question of support here. I'd love for a few more folks to weigh in, tagging a few folks who may have ideas: @tqchen @jroesch @kparzysz-quic @u99127 @Mousius @leandron @comaniac @zhiics @Hzfengsy @ZihengJiang @yzhliu @masahi @icemelon

broadly, the question is: once we release version 0.X.0, should we discover a bug, we should likely fix it on that branch and on main and release 0.X.Y. how long shall we commit to doing that? If a bug is discovered on 0.X-1.0, will we fix it?

my vote is the following (for now, we can revisit at 1.0):

• we agree we will fix major bugs on 0.X.Y where 0.X.Y is the latest release.

Ack. I'd suggest moving bug classification in another RFC.

• when we make a new minor version release e.g. 0.(X+1).0, we will stop maintaining releases starting with 0.X. at that time.

When we make a new minor version release 0.(x+1).0 , we do a 0.X.(Y+1) and close the branch, otherwise there might be bug fixes since 0.x.y that never see a release or the light of day.

I'd suggest that minor adjustment to the text to make it explicit about what we do even though it may be obvious :)

• committers/pmcs should exercise their judgement when determining if something warrants a fix.

s/fix/backport

• the same policy should apply on the forum--if someone asks a question on an old minor release version, it should be acceptable to ask them to reproduce it on the latest release.

Hmm, I'd suggest that we say all bugs / questions should be reproduced with trunk / main and the latest release rather than an unsupported release.

We probably need to update the "Releases tab" on gh to make this work.

This is looking like good small steps forward. Very good to see this coming together.

Ramana

[areusch]

ok cool, so i think we have these actionables:

• note that when we do a minor release, we check the previous minor release branch and also do a revision release if there are unreleased changes there
• make it clear that the release manager is responsible to ensure that bugs raised with the release are addressed via patch/revision release, but that such tasks are largely expected to be delegated to members of the community.

[u99127]

I think the main outstanding thing is the question of support here. I'd love for a few more folks to weigh in, tagging a few folks who may have ideas: @tqchen @jroesch @kparzysz-quic @u99127 @Mousius @leandron @comaniac @zhiics @Hzfengsy @ZihengJiang @yzhliu @masahi @icemelon

broadly, the question is: once we release version 0.X.0, should we discover a bug, we should likely fix it on that branch and on main and release 0.X.Y. how long shall we commit to doing that? If a bug is discovered on 0.X-1.0, will we fix it?

my vote is the following (for now, we can revisit at 1.0):

• we agree we will fix major bugs on 0.X.Y where 0.X.Y is the latest release.

Agreed.

• when we make a new minor version release e.g. 0.(X+1).0, we will stop maintaining releases starting with 0.X. at that time.

Agreed.

• committers/pmcs should exercise their judgement when determining if something warrants a fix.
• the same policy should apply on the forum--if someone asks a question on an old minor release version, it should be acceptable to ask them to reproduce it on the latest release.

I agree and I'd be stronger that the bug needs to be reproducible on trunk / main and the latest release.

How do we get this moving forward ?

R

[Mousius]

Thanks for the updates @driazati, I think we've gotten a good amount of input here from folks and we can move forwards with this for a July release.

Will leave this open until end of week unless someone has more comments 😸

cc @hogepodge @leandron @areusch @tqchen @u99127

[areusch]

Great. let's move this to a PMC voting thread. I'll make one.

[kparzysz-quic]

What is the test suite? Is it anything beyond what the Jenkins tests already do?

[driazati]

Not currently and this RFC intentionally doesn't specify it, we can iterate on release-specific tests more as we go through the process

[csullivan]

Developers are heavily encouraged to use feature flags ... to enable in-progress features. It is important both for users to be able to access new or experimental behavior but also that they can get a stable interface, so these kinds of changes should be hidden behind opt-in flags.

Thanks @driazati, I think this specifically is a very positive move for TVM to make. I have encountered perception among developers that designs must be near-perfect before making it into TVM.

I'm happy to see explicit language around experimental feature flags which explicitly encourage iterative design. I wonder about your and others opinion on whether we need a separate RFC to formalize the process around introducing experimental changes and their path to graduation into stable releases?

Nonetheless this is great first step, thank you for outlining it.

[areusch]

Thanks all for the comments, the vote has passed and we'll merge this RFC.