Migrate to GitHub issue from Jira [LUCENE-10557] #11593
Comments
|
Michael McCandless (@mikemccand) (migrated from JIRA) +1 Do we know whether we could (relatively easily) migrate our Jira issues over to GitHub issues? |
|
Michael McCandless (@mikemccand) (migrated from JIRA) Hmm at least ~4 years ago, migrating was not possible/easy. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) INFRA-16128 was the issue when RocketMQ migrated: https://issues.apache.org/jira/browse/INFRA-16128 |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Among the tasks I mentioned in the issue description, the most important (and most difficult?) one could be "Get a consensus about the migration". I look for explicit feedback (+1 or -1) on this topic from developers. I'd definitely cast +1; not because I don't like Jira (it works well for me so far), but because the most development is already going on GitHub. Consolidation of conversation would be good for improving experiences I think. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Thank you @mikemccand for joining and giving the pointer! |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) INFRA team says Vote or PMC agreement are needed for the decision in the RocketMQ's issue (it makes sense to me).
|
|
Shad Storhaug (migrated from JIRA) The Lucene.NET project switched to GitHub issues 3 years ago, and we were able to get far more contributor participation than on JIRA, but that may just be mostly a .NET thing. We held a vote and all participants were unanimous on the GitHub Issues migration. It was pretty straightforward for INFRA to do. We made a script to migrate the open issues to GitHub via API, but there were a couple of issues we had that you may need to find a workaround for:
There is a walkthrough of the process here: https://gist.github.com/jonmagic/5282384165e0f86ef105#start-an-issue-import%22 We didn't disable JIRA support, but we didn't really have to because we couldn't get users to participate on it, anyway. But that is something you will also need to take into consideration. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Shad Storhaug thanks for giving valuable information! We are having a very long discussion on the migration in the mail list. :) https://lists.apache.org/thread/78wj0vll73sct065m5jjm4z8gqb5yffk |
|
Michael McCandless (@mikemccand) (migrated from JIRA) Oh that's great news; thanks for sharing Shad Storhaug! |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) We are having a long discuss thread on the dev list and many issues are posed. Here is a short summary (with my brief thoughts/opinions).
Aside from those concerns, there seems no disagreement with GitHub is superior to Jira in terms of overall UX design, and most new developers like it. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Issue stats on 28/May/2022. # of total issues: 10543 Breakdown of unresolved issues
By Affects Version (overlapped)
By Created Date
By Component
|
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Operational policies in other projects on GitHub (randomly selected). Kubernetes (https://github.com/kubernetes/kubernetes)
Elasticsearch (https://github.com/elastic/elasticsearch)
OpenSearch (https://github.com/opensearch-project/OpenSearch)
Apache Solr Operator (https://github.com/apache/solr-operator)
Apache Airflow (https://github.com/apache/airflow)
Apache Lucene.NET (https://github.com/apache/lucenenet)
Spring Framework (https://github.com/spring-projects/spring-framework)
Gradle (https://github.com/gradle/gradle)
|
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Migration plan (an early draft)
|
|
Jan Høydahl (@janhoy) (migrated from JIRA) Thanks for thorough research! I propose to skip bulk migration of Jira issues and instead bulk comment on all open JIRAs, prompting the reporter or assignee to take whatever action they see fit. Some will choose to migrate, others will finalize the feature in JIRA, quite some will probably be closed as won't do, and some will just remain open/stale/don't care. No history will be lost, we can still refer and link to historic Jira issues, as long as ASF keeps Jira alive. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Hi @janhoy, thanks for your suggestion. "Migrate no issues and make a fresh start" is definitely an option, on the other hand, many issues that are worth revisiting may remain Jira forever - I feel a bit sorry for them. If unresolved issues are on GitHub (even just the title and description) from the start, possible contributors will be able to browse/search them. However, I think we can discuss it later. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) One feature I couldn't find out in GitHub issue that is naturally equipped in Jira is the parent-child issue hierarchy. We can have an umbrella issue and tie any number of sub-issues to it with Jira but cannot with GitHub issue; I imagine "GitHub Project" is used for such issue grouping purpose (haven't used it) though, it'd be too much for us I think. |
|
Dawid Weiss (@dweiss) (migrated from JIRA) I don't think this is a problem. You just create a description with a bullet list and reference related issues - they do show up in mentions, I think this is sufficient. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Sure - it's not a big problem I think. Except for some minor functional gaps like this, I don't see any technical blocker for migration. |
|
Jan Høydahl (@janhoy) (migrated from JIRA) The "Project" grouping could for sure be used for larger pieces of work which we know will span multiple PRs. See Airflow using it for their AIP improvement proposals https://github.com/apache/airflow/projects/12 |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Thanks, it looks like a plain Kanban Board. I know it, I'm not sure Lucene people like it though. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) Thanks @mocobeta for driving this discussion/vote! I will now remove the trailing
|
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) For version control, there are two considerations.
We have two options: Milestone or Label. One important difference between them is that an issue can have only one milestone but multiple labels. The other difference would be that while Milestone is special metadata, labels are just flexible text tags for searching. I'm personally fine with Milestone - we don't release a bug fix or improvement in multiple versions anyway. We don't have two CHANGES entries for one issue; if we resolve an issue in "10.0.0" and "9.3.0" the CHANGES entry appears only in Lucene 9.3.0's section.
35% of unresolved issues have this field. Maybe we could have issue labels such as "affectsVersion:9.3.0". I have never used this metadata field and I myself have no problem with omitting this in GitHub. Is there anyone who has thoughts on it? –
|
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) > I've added a few bullet points that script could/should handle under LUCENE-10557, hope you don't mind. If you place these script(s) in the open then perhaps indeed we could try to collaborate and see what can be done. Thanks for your suggestions, Dawid. I'd move the conversation to this issue from the mail list. I think we'll be able to handle the requirements (cross-issue links, and so on) in some ways. I started work on #11658 and added the link to the sandbox repository where the migration scripts (early draft) were pushed. For what it's worth, #1079 will be migrated something like this. Although the formatting and look-and-feel could be improved a bit, it would not be drastically changed in essentials. We cannot simulate Jira issues on GitHub. e.g.; it is not allowed to tweak the issue reporter and timestamp (very basic metadata to me), so they have to be embedded in the issue description as free texts. I'll continue to work on though - does this really meet your expectations, Mike McCandless and other folks who argue to preserve all issue history in GItHub? |
|
Dawid Weiss (@dweiss) (migrated from JIRA) I've verified that searches for old issue numbers seem to work: I'm more familiar with the "hierarchical" tags like "affects/xyz" or "type/bug" but I can live with the comma version. Good to have some of the metadata transferred as well, even as a plain text content in the issue description. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) As for User ID alignment, it'd be great if we can map the reporter/assignee/commenter to correct GitHub accounts. I just wanted to note that there is a trivial but very practical concern for me - we have to "mention" the accounts in the issue description/comment to create a hyperlink (we can't create resources on behalf of the original authors). I think we don't want to receive a huge volume of notifications from old issues. There could be a tip or workaround, otherwise we will not be able to create real links but just have markups like |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I browsed through several JSON dumps of Jira issues. These are some observations.
On GitHub side, there are no difficulties in dealing with the APIs.
As for the cross-link conversion and account mapping script:
Yes it should be possible with a set of small scripts. Maybe one problem is that it'd be difficult to detect conversion errors/omissions and we can't correct them ourselves if we notice migration errors later (it seems we are not allowed to have the github token of the ASF repository). |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I was trying to figure out how to upload attachments (patches, images, etc.) to Github issue with API for hours. {}There is no way to upload files to GitHub with REST APIs{}; it is only allowed via the Web Interface. If you want to programatically port attachment files in Jira to GitHub, you have to commit the files to the repository.
|
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Maybe GitHub's API call rate limit would be another consideration.
We can't really "bulk" import to GitHub. Every issue and comment has to be posted one by one and between the API calls, at least one-second sleep is required. I encountered this rate limit many times - actually it seem that the rate limit is strictly monitored.
|
|
Michael McCandless (@mikemccand) (migrated from JIRA) Finally catching up over here! Thank you @mocobeta for tackling this! I agree testing what is realistic/possible will enable us to make an informed decision. I really hope we are not stuck asking all future developers to fallback to Jira and use two search engines. To make Jira effectively read-only post-migration, Robert suggested we could use Jira's workflow controls to make a "degraded" workflow that does not allow any writes to the issues (creating new issues, adding comments, changing milestones, etc.). We can add that to the (draft) migration steps. For committers, https://id.apache.org has the mapping of apache userid to GitHub id, though I'm not sure if that is publicly queryable. And as @msokolov pointed out on the dev list thread, the GitHub Apache org might also have it. Did you see/start from the Lucene.Net migration tool? This is what Shad Storhaug pointed to (up above). Those few migrated issues look like a great start!
Wow that is indeed disappointing. I wonder whether GitHub's issue search also search attachments? Does Jira's search? |
|
Uwe Schindler (@uschindler) (migrated from JIRA) Maybe we can ask them to manually disable notifications during the import. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I changed the notification mails to noreply@ so that we can silence them. Could you check the notification in private@ again please, @uschindler? Thanks.
I think it's against GitHub's terms of policy to have multiple free accounts. I'm not sure it is possible though if we have a paid organization account that is not tied to a person, we could ask infra if we use it for the migration?
Hmm, then I'll revert the change.
In the recent migration test where all issues are migrated, we verified that notifications are not sent when executing migration scripts (importing issues and updating issues/comments), thanks to Houston and Dawid. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) We can't use an arbitrary github account for migration because importing/creating issues with GitHub API requires not only the access token but also admin access to the repo - it is not allowed to have for developers. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I found this issue is an excellent sample for testing - this includes:
So I would add a numbered list and a fake table in this comment to make this more convenient for testing. Please ignore this comment.
|
|
Uwe Schindler (@uschindler) (migrated from JIRA) I know from the past that INFRA had some video conferences with Github representatives, so ASF is not just "some arbitrary customer". I think there was a lot of discussions going on. The LUCENE.NET import was long before they had close contact to Github. I would really prefer to keep all orginal contributors, the change of names to some private account is a real blocker to me. When we can't modify the comment/issue creator mail address to use the official ASF one of the person or use some generic bot account, I would vote now -1 to the migration. P.S.: Spring used a generic user for the import "spring-projects-issues": https://github.com/spring-projects/spring-framework/issues/created_by/spring-projects-issues |
|
Uwe Schindler (@uschindler) (migrated from JIRA) Spring also have a cool redirector in their webserver. It only redirects if you don't have some special param: https://jira.spring.io/browse/SPR-17639?redirect=false And they also added a comment at end of all their issues (also by the bot). |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA)
Yes, I like it. It should be an organization account - maybe we can ask infra if we have one? |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I think it looks like we have too many topics to deal with in one issue? We can break up them into sub-jira tasks though, I created a few github issues in the lucene-jira-archive repo. Notifications were sent to issues@ this time. Looks fine. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) I'm sorry for the noise - Jira's special emojis should be converted to corresponding Unicode emojis. This is a test post to make sure the mapping is correct. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) Overall, I think we have a relatively safe migration strategy on GitHub side (many bug fixes and tasks still remain to be done). On Jira side, I would do just two things.
It may not fulfill everyone's request/hope though, I already have too much on my plate and can't really expect others' hands on that. Please feel free to work on further improvements on Jira side, if you think it should be done for the move to GitHub (making Jira read-only, redirecting to GitHub from Jira, and so on). |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) [TEST] This was moved to GitHub issue: https://github.com/mocobeta/migration-test-3/issues/196. |
|
Michael McCandless (@mikemccand) (migrated from JIRA)
Oooh that looks promising!! |
|
Michael McCandless (@mikemccand) (migrated from JIRA) OK I am able to administer our Jira instance. There are some wrinkles – apparently because some of our workflows are shared across two projects (Lucene and Solr), the workflows themselves are read-only! So we cannot change them unless we work the workflows. But there is much discussion about this problem, e.g.: https://community.atlassian.com/t5/Jira-questions/Fastest-way-to-make-JIRA-read-only/qaq-p/1261492 I'll try to find the simplest way that works for us. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) Hi @mocobeta – I added you as a Jira Administrator so you can poke around if you want to. But I'll still try to figure out how to make Jira read-only. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) @mikemccand thank you, I'm now able to edit the configuration. However, I am struggling with figuring out how to tweak the issue creation panel. I just wanted to set a placeholder or default value to the "Description" field... |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) it seems "Project admin" is not really allowed to do meaningful things, almost all components are shared between projects, and only Jira administrators can change them. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) OK hmm we will likely need Infra's help for this then. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) I realized that the Infra team makes Jira projects read-only sometimes (e.g. when they retire / expire to Attic): https://issues.apache.org/jira/browse/INFRA-23440?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel I'll send an email to Infra to see if they have some clean/simple way to do this at the start of our migration process. |
|
Michael McCandless (@mikemccand) (migrated from JIRA) Though, one small wrinkle we have is that we will need to append one final comment to all Jiras (what @mocobeta has been iterating on recently!) after the GitHub issue assignment is known ... so we can't make it completely read-only until after the migration. Tricky. There will be a window where users may update some Jiras mid-migration. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) The migration plan takes into account some moratoriums (three days, I think).
I don't think it is a big problem. We'll be able to easily detect issues/comments that are created during migration and add follow-up comments to encourage the authors migrate the comments manually.
Again - I don't think we have had any discussion about making Jira read-only in the mail list. I don't have strong opinion on that, but I am against it if we are going to do so without giving others time to consider / chance to express opinions on this. |
|
Michael McCandless (@mikemccand) (migrated from JIRA)
OK – I'll start a [DISCUSS] and then [VOTE] thread to reach consensus on this. |
|
ASF subversion and git services (migrated from JIRA) Commit 8938e6a in lucene's branch refs/heads/main from Tomoko Uchida LUCENE-10557: Add GitHub issue templates (#1024) |
|
ASF subversion and git services (migrated from JIRA) Commit 781edf4 in lucene's branch refs/heads/main from Tomoko Uchida LUCENE-10557: Refine issue label texts (#1036) |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) [TEST] This was moved to GitHub issue: #10626. |
|
Tomoko Uchida (@mocobeta) (migrated from JIRA) The work has already been carried over to https://github.com/apache/lucene-jira-archive. |
A few (not the majority) Apache projects already use the GitHub issue instead of Jira. For example,
Airflow: https://github.com/apache/airflow/issues
BookKeeper: https://github.com/apache/bookkeeper/issues
So I think it'd be technically possible that we move to GitHub issue. I have little knowledge of how to proceed with it, I'd like to discuss whether we should migrate to it, and if so, how to smoothly handle the migration.
The major tasks would be:
Conclusion for now: We don't migrate any issues. Only new issues should be opened on GitHub.Migrated from LUCENE-10557 by Tomoko Uchida (@mocobeta), resolved Aug 18 2022
Parent: #11579
Attachments: image-2022-06-29-13-36-57-365.png, screenshot-1.png, Screen Shot 2022-06-29 at 11.02.35 AM.png
Linked issues:
Pull requests: #988, #1024, #1036
The text was updated successfully, but these errors were encountered: