🚀 Switchover from Trac to GitHub

[mkoeppe]

0. Announce switchover schedule on sage-devel.
1. Jan 30, 13:00 UTC: Make Trac read-only (this needs help by a Trac admin):
   • Take Trac offline,
   • Reconfigure Trac ticket system, Trac wiki to be read-only
   • Make sure Trac git repo is fully synced to https://github.com/sagemath/sagetrac-mirror.
   • Make Trac git repo read-only
   • Bring Trac online.
   • Announce read-only availability of Trac on sage-devel
   • this needs to be completed by Jan 30, 23:00 UTC the latest
2. Archive sagemath/sagetrac-mirror (= set it to readonly).
3. Final update of GitHub usernames.
4. Re-run write-closing-commits.sh (done in 83bc2e2, includes 9.8.rc0)
5. Use fresh worktree worktree-prod (instead of clearing the trac-to-github disk cache (rm -rf trac_cache) and the migration archive (git clean -fX archive))
6. Run migrate.py to create the migration archive in archive/. about 6.5 hours runtime, needs running, read-only Trac server
7. Set migrate.cfg.sagetracmigrationarchive [wiki] migrate: yes
8. Check any remaining pull requests for the migration script, merge master into production. Clear the migration archive again (git clean -fX archive).
9. Re-run migrate.py. about 30 minutes.
10. (Optional:) put a mirrored clone of sagemath/sage into the migration archive (but remove all branches except master, develop!) at archive/repositories/sagemath/sage.git. In the bare repo, run rm -Rf refs/pull && git gc to get rid of the pull request refs from sagemath/sage; we do not transfer these PRs in this step. (Alternatively, we can push to the repo later.)
11. Init the bare repository archive/repositories/sagemath/sage.wiki.git. (Importing a populated wiki repository does not seem to work – wiki at https://34.105.185.241/sagemath/sage-20230130181123/wiki was empty on import. So we push the wiki later.)
12. Edit the repo description of sagemath/sage as preparation for renaming/archiving it.
13. (Optional:) Import into our GHE server for checking for necessary additions to map-ghe.csv.in and general inspection. about 20 hours runtime.
14. Rename sagemath/sage to sagemath/sage-archive-2023-02-01.
15. Archive sagemath/sage-archive-2023-02-01.
16. Feb 1, 9am Pacific Time: Request access to Enterprise Cloud Importer. (this needs help by our GitHub contacts)
17. Upload the migration archive archive.tar.gz; also upload map-github.csv (edited from map-ghe.csv.in -- see script rsync-archive-to-ghe-and-import.sh).
18. Set org Member permissions to minimal. Invite users to sagemath GitHub org using API (get-a-user, create-org-invite), post heads-up on sage-devel, create org README as a landing page for invitees
19. Feb 3, 0:00 UTC: Fix up the map using GraphQL (ECI does not allow me to go back to change the map when it thinks it is "READY") or restart the import from scratch (~1h).
20. Import ("perform") – as a new repo sagemath/sage (this should break the redirect sagemath/sage -> sagemath/sage-archive-2023-02-01).
    4. (Optional:) Create a single-branch fork of sagemath/sage-temp called sagemath/sagetrac-mirror and push all branches (or all branches of open tickets) from sagemath/sagetrac-archive to it. Open PRs from sagemath/sagetrac-mirror to sagemath/sage-temp for all open tickets with attached branches.
21. Archive the repo so that we don't catch premature PRs
22. Announce preview on sage-devel.
23. Rename sagemath/sage to sagemath/sage-migration-preview-2023-02-02
24. Fix remaining attachment issues (Fix up handling of repository_files / attachments for some types #176)
25. Feb 4, 22:00 UTC Regenerate archive again, import as sagemath/sage-prod
26. Rename faulty import sagemath/sage to sagemath/sage-migration-preview-2023-02-04
27. rename sagemath/sage-prod to sagemath/sage
28. Disable Actions until Disable pull-request github workflows #157 is done (6526184)
    17. [ ] Rename the new repo sagemath/sage-temp to sagemath/sage. (Yes, this will work and will break the redirect sagemath/sage -> sagemath/sage-archive-2023-02-01.)
    18. [ ] (Optional:) Transfer the existing (few) issues from sagemath/sage-archive-2023-02-01 to sagemath/sage.
29. fast-forward develop to 9.8.rc1,
30. enable Actions,
31. check that Projects are disabled
32. disable Discussions
33. planned: by Feb 6, 08:00 UTC Announce on sage-devel that sagemath/sage is now open for Issues and PRs.
34. Clone https://github.com/sagemath/sage-archive-2023-02-01.wiki, add converted wiki. merged-wiki
35. Push to https://github.com/sagemath/sage.wiki
36. Display a link from each Trac ticket to the converted GitHub Issue (this needs help by a Trac admin)

[tobiasdiez]

I'm a bit out of the loop about how exactly open tickets with existing branches are now converted (issue or pr?), but in either way the migration guide should document how to keep working on an existing trac ticket/branch.

[mkoeppe]

Per discussion in #57, trying to have the migration script create PRs is probably not worth the effort.

#68 asks to document the procedure to start a PR from a migrated issue with an existing branch.

[kwankyu]

Where is the procedure to transfer trac wiki pages?

Like the practice of using the script, perhaps we should migrate the wiki first and then tickets, for safety.

[mkoeppe]

The procedure depends on the decision made in #83

[kwankyu]

Convert all tickets to Issues in a new repo sagemath/sage-temp via a migration archive.

Do we use the migration archive (the json files in /archive) for this? As I understand it, we use the migration archive only for GitHub Enterprise Server.

[mkoeppe]

Yes, we will use the migration archive for the final import too.

[kwankyu]

Sorry about asking perhaps a settled matter.

According to the documentation like https://docs.github.com/en/enterprise-server@3.7/admin/user-management/migrating-data-to-and-from-your-enterprise, I understand that migration archive is for migration to GitHub Enterprise Server, but not to Github.com. Does Github.com provide an API for migration archive? Did we test it? Or do we have a special agreement with Github.com?

[mkoeppe]

We have a contact at GitHub who will take care of the import of the migration archive for us.

[kwankyu]

OK! Thanks.

[kwankyu]

If we are on the schedule as planned here, an announcement of the migration day and the schedule should be made on sage-devel. Perhaps a notice of what an individual developer should do, if any, may be included.

[mkoeppe]

Yes, I'm waiting for a Trac admin to confirm that they are able to do step 1 by Jan 30. @dimpase

[dimpase]

yes, sure, I'll do this in some way. Let's say I'll start at 13:00 GMT on 30 Jan, and hope to be done by 23:00 GMT (maybe earlier).

I wish we had @embray helping though - he's an expert in trac...

[mkoeppe]

Great, thank you!

I'll send out an announcement on sage-devel.

[tobiasdiez]

Can you please also rename the master branch to main as one of the last steps. Theoretically, this could also be done afterwards but this would only result in unnecessary overhead.

[kwankyu]

Why? I am sympathetic with the argument in https://medium.datadriveninvestor.com/why-githubs-change-from-master-to-main-is-not-the-solution-a3ac38cc48dd

[mkoeppe]

Perhaps master should actually be renamed to stable or release

[tobiasdiez]

I think "main / develop" is the common terminology for the gitflow workflow. I would say we can use the github migration to move to the modern trunk-based workflow and only have one main branch. As a temporary solution, we could have main (the current develop branch) and release (the current master branch); and at some point the release branch is just deleted?

Personally, I also don't feel attacked by the word "master" but I can understand that I'm in a privileged position and there are people who feel different about this word. It's a simple thing to use "main" instead, so why not do it? Of course, it doesn't "solve" the problem but it's a step in the right direction.

[mkoeppe]

This whole thing will need to be discussed on sage-devel and with our release manager.
I don't think there's any reason to make this change at the same time as the migration.

[tobiasdiez]

I agree that the move from gitflow to trunk-based needs further discussion, but the rename of master to main was included from the beginning in the github migration proposal and can still be found in https://github.com/sagemath/trac-to-github/blob/master/docs/Migration-Trac-to-Github.md#release-managers-workflow.

[dimpase]

Let us not complicate the move to github by the branch renaming. Once it's done, renaming should be very easy.

[tobiasdiez]

Renaming the branches can be easily be done in the github ui, yes. But every dev needs then to update their local config (and their fork?) to reflect these changes. Thus it would be easier to simple rename the branches before everyone setups their local repo to track the github repo.

Moreover, the rename of master to main should not really influence anyone (except for the release manager maybe), right?

[dimpase]

it would be easier to simple rename the branches before everyone setups their local repo

at least in my practice I seldom use the master branch of SageMath.

Anyway, the renaming of the master branch need not be done now, it can be done after the migration is done, but before the new setup goes live.

[dimpase]

@mkoeppe - am I right that the migration process does not use git.sagemath.org, and so it can be just be kept offline ?

[mkoeppe]

am I right that the migration process does not use git.sagemath.org, and so it can be just be kept offline ?

Yes, that's right.

[dimpase]

I expect to be able to make it read-only, but, just in case, good to know.

[dimpase]

I don't see how we can make this 101% watertight without actually changing the code of Trac. (Or doing something clever with the Apache server, which serves trac's pages - but this is not what I can see quickly).

GitHub logins are disabled now, there even no button to click on for this any more.

[dimpase]

FWIW, I am still logged in, and I can edit my and other people's comments.

can you open tickets, modify tickets descriptions, change their status? Or it's only comments?

[mkoeppe]

Haven't tried - does it matter?

[mkoeppe]

https://github.com/edgewall/trac/blob/trunk/trac/web/session.py#L389
reveals that there are commands to list and delete sessions.

[dimpase]

As far as I am concerned, it's ready for importing. I have made a snapshot...

[mkoeppe]

trac-admin session list etc. - https://trac.edgewall.org/wiki/TracAdmin

[dimpase]

https://github.com/edgewall/trac/blob/trunk/trac/web/session.py#L389 reveals that there are commands to list and delete sessions.

yes, and that's what I was doing in the last 5 hours - running these commands.

$ for uu in `trac-admin /srv/trac/sage_trac session list authenticated | cut -d " " -f1`; do trac-admin /srv/trac/sage_trac session delete $uu; done

But they still didn't do enough to kick you or David out! And now there is exactly one such a session present

[mkoeppe]

OK. Then we just declare victory.

[dimpase]

Running trac-admin breaks trac's serving pages, so server needs a mild manual cleanup and a reboot after it was used.
This is due to our home-made sage_trac plugin, which insists on cloning the gitolite-admin repo with ssh keys on each invocation of trac-admin (sic!). This cloning, done by a side user, on a running instance is knocking things down.

Perhaps this breakage of trac-admin only happened after trac update in Sept, to v1.2. I don't know. The sooner it's retired, the better...

[mkoeppe]

I have started the migration script.

[kwankyu]

Congrats!

Goodbye Trac, we are leaving!

[dimpase]

I was told where to look at Trac source for these auth_cookie (which live in a SQL database, as an SQL table). It's in Trac's web/auth.py module. If I understand its logic right, there is a check for an auth cookie for the user in the database, but no check for user to be a valid user.

[tobiasdiez]

Stripping all cookies via the apache config (so that they never reach trac) may also work and should be relatively simple to implement.

[dimpase]

Is trac needed now? I'd like to experiment with fixing this last bit, beginning from getting rid of hopefully not needed sage_trac plugin.

[mkoeppe]

No, I've scraped all necessary data from Trac.

[mkoeppe]

@tobiasdiez @dimpase @fchapoton Do you think "Display a link from each Trac ticket to the converted GitHub Issue" is easy to do?

[kwankyu]

After the final check and before opening up the repo for issues and PRs, would the repo be accessible by general public to check?

[mkoeppe]

Good idea, I have added steps for this; I'll try to use repo locking during period.

[dimpase]

@tobiasdiez @dimpase @fchapoton Do you think "Display a link from each Trac ticket to the converted GitHub Issue" is easy to do?

it should be doable. On my side, not before the next week, though.

[tobiasdiez]

@tobiasdiez @dimpase @fchapoton Do you think "Display a link from each Trac ticket to the converted GitHub Issue" is easy to do?

it should be doable. On my side, not before the next week, though.

If I remember correctly from my short interaction, there is a simple config to add batches. Thus, the simplest would be to add a badge like https://img.shields.io/badge/Github-Go%20to%20the%20migrated%20issue-orange?style=for-the-badge&logo=github with a link to https://github.com/sagemath/sage/issues/<ticket> via the config.

[kwankyu]

PR authors cannot apply labels yet: sagemath/sage#34960

[kwankyu]

Issue links do not work in wiki: https://github.com/sagemath/sage/wiki/ReleaseTours-sage-9.8

[dimpase]

Issue links do not work in wiki: https://github.com/sagemath/sage/wiki/ReleaseTours-sage-9.8

yep, it's cause we need to do a mass rewrtiting of #12345 to [#12345](https://github.com/sagemath/sage/issues/12345) - which will render as #12345

Unfortunately GitHub wikis don't do autoexpansion the way one can do in comments/issues, cf https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls#issues-and-pull-requests

[kwankyu]

We could do the mass rewriting by the migration script if people had not started editing GitHub wiki pages. Now we should do it perhaps (semi)-manually.

[mkoeppe]

I've re-run the wiki import using @kwankyu's updated script from #183 and merged with the wiki.