[Retrospective] Release Version 2.10.0 and 2.11.0

[gaiksaya]

Related release issue?

#3743

How to use this issue?

Please add comments to this issue, they can be small or large in scope. Honest feedback is important to improve our processes, suggestions are also welcomed but not required.

What will happen to this issue post release?

There will be a discussion(s) about how the release went and how the next release can be improved. Then this ticket will be updated with the notes of that discussion along side action items.

[gaiksaya]

Below components ignored or removed the flaky tests in this release. We need to make sure they are fixed and added back in next upcoming release:

• Alerting
• Alerting Dashboards
• Security Analytics
• Security Analytics Dashboards
• Dashboard reports
• Observability Dashboards
• CustomImportsMaps
• Dashboards Core

[gaiksaya]

OpenSearch-build repository needs to honor code cutoff date and enable branching.
On high level atleast have 1.x, 2.x and main (3.x) branches to test code and then backport to relevant branches so that a release is not hindered due to infra level changes.

[CEHENKLE]

I think we need to solidify our communication strategy -- Where should we look for updates? How can we make sure we can tell at a glance who needs to do what? the thread in the release channel got very long, and I found it hard to keep a bead on how we were doing on the exit criteria.

But overall, I liked the window model. I'll be curious to see what other folks think.

[sandervandegeijn]

I'm writing this with a ton of appreciation for all the hard work that went into this release and all the other releases. As a university we are extremely happy with the project.

I did find a few bugs in the new UI that would have limited the usability of the new layout. This happens more often, when new features are added it can take a few releases to iron out the bugs. That's understandable, but it's a shame if these bugs/inconsistencies end up in a release. It's not always about corner cases.

Thinking out loud here; maybe it could help to issue a testing period for a new release with communication to the community to get people involved. Reporting bugs should be very straight forward in this period. In the case of the UI errors it really helped that there was a direct line of communication with the developers and I could throw in more stuff I found in a simple manner. The experimental feature option is a nice one to facilitate this in part.

There are also so many new features being added to opensearch. Very very cool but sometimes that seems to have impact on the focus on quality and consistency. Maybe a small shift in allocated time to fix bugs and really look for inconsistencies could help.

[hdhalter]

What went well?

• The exit criteria "Documentation has been fully reviewed and signed off by the documentation community" works well for the documentation team. Instead of trying to reach an arbitrary date where there's potential to bypass quality measures, we write the feature documentation until we feel we are ready to release quality content.
• There was better collaboration late in the cycle. Since we have definitive exit criteria, there was a feeling of 'we are in this together' until we were all ready to release.
• Developers submitting PRs for backend documentation. This exponentially increased the time we get to merging a PR because the technical review process was much faster or non-existent.

What didn't go well?

• The entrance criteria for starting a release candidate states, "Documentation draft PRs are up and in tech review for all component changes." This is not possible given the current process. Currently, developers are adding features up until the beginning of the RC window and adding documentation issues in the last days leading up to it. This leaves no time for the documentation team to research the issue and get a PR up for review.
• Developers are not submitting documentation PRs for backend content, which is now a requirement. This slows down the documentation process, because we need to do the information gathering process and there's a lot of back and forth during the technical reviews.
• Not all features being released are included in the roadmap. This leads to lots of last-minute requests to the documentation team. We are left to scramble to get them documented late in the cycle, putting the features and quality at risk.
• SDMs communicate that a feature will move to the next release train, so we remove that item from our backlog to prioritize on other things. But then later we find out that the developer merged it anyway. Again, we scramble to document it late in the cycle. We need to have a less manual and trackable way of updating the status for a feature so everyone is on the same page.
• Similarly, sometimes a feature gets dropped and we are not told. We continue working on the documentation and trying to get approvals when there is no urgency. This puts other tasks at risk. Having a consistent, trackable system in place would help alleviate this problem.
• Demo meetings are taking place without documentation representation.
• Go/no go meetings are only announced in Slack; sometimes key people don't catch it on time and miss or are late to the call.
• During the last 2 release calls, there was no check for whether the documentation had been released. The status had to be volunteered by the documentation release manager.

Suggestions/requirements for next release:

• Introduce a "feature freeze" date ahead of the RC candidate, so documentation can meet the entrance criteria. Alternatively, we enforce the entrance criteria "Documentation draft PRs are up and in tech review for all component changes".
• Developers write doc PRs for all back-end content needed. Note that when the developer creates the PR and puts it up for review, this meets the criteria for 'in tech review'. Therefore, if this is happening consistently (developers create PRs), it puts us in a better position to meet the entrance criteria.
• Developers write doc issues for front-end content needed.
• Better communication of the status of a feature, especially if it's dropped. Make sure all parties are aware. For example, add a comment to the dev and corresponding doc issue.

[bbarani]

Closing the issue as we have documented the action items.