Reorganization of OpenSHA GitHub project structure

[kevinmilner]

First, tagging all recently-active contributors and potentially interested/impacted parties for comments. Please tag anyone else who may need to see this as well!
@field-usgs @pmpowers-usgs @nvanderelst @mtpage @mbarall @chrisbc @voj @thingbaijam @alevale19 @wsavran @scallag

Current state of OpenSHA projects

We moved OpenSHA from SVN to GitHub summer 2017. At that time, we split the single SVN project up into a number of sub-projects:

Name	Depends On	Description
commons	-	Base commons library
core	commons	Core OpenSHA library with calculators and models
ucerf3	core	UCERF3 model code and data
apps	ucerf3	GUI applications
dev	apps	Development sandbox for shared prototyping
cybershake	dev	CyberShake interface code and calculators

The thought was that these projects would be allow users to only get the pieces of code that they want, and for an external project (nshmp-haz) to potentially only use some common definitions without using the whole codebase. That latter goal was never realized, and the current structure is overly complicated. Most users of the code download all repositories through ucerf3 or even dev, and no ones uses jars from upstream projects in external projects.

There are also a couple new downstream projects that were not part of the initial plan:

Name	Depends On	Description
oaf	apps	Operational Aftershock Forecasting codes and apps
nshm-nz-opensha	ucerf3	GNS-Science codes for their UCERF3-like NZ model

Proposed Project Structure

I propose simplifying the project structure to the following:

Name	Depends On	Description
opensha	-	Primary OpenSHA repository, contains everything currently in opensha-commons, -core, -ucerf3, and -apps
opensha-dev	opensha	Relatively unchanged, development repository where we're loose with write access to external developers. Very preliminary or temporary code goes here
opensha-cybershake	opensha-dev	Continue to keep CyberShake stuff out of the main repository (for now)
opensha-oaf	opensha	Relatively unchanged, just now depends on main opensha repository

This will accomplish a few goals:

1. Most users will just check out opensha, or possibly also opensha-dev if they want to collaborate on test code.
2. Issues created by users are currently mostly in the opensha-apps project, not the repository where the code at issue actually resides. Now the bug report dialog will send people to the consolidated repository for bug reports
3. Easier for new users to get involved, won't have to create pull requests across multiple repos. Right now, API improvements in core/commons often require corresponding changes in other repositories that can be complicated to manage.
4. Fault System Solution code no longer in a 'UCERF3' repository, as it is now being applied well beyond UCERF3.
5. Development sandbox still exists with more lax write-restrictions (e.g., to help other users with demo code)

If we ever realize the goal of a real common library for things like Location, Surface, etc, we can create a higher level repository at that time (e.g., 'opensha-lib'). Those APIs would probably just be included as a jar file in 'opensha' as the classes/interfaces therein would be, by design, very rarely modified.

Proposed Other Changes (at the same time)

This change also provides a good opportunity to do a few other refactoring tasks. Here is what I propose:

Switch to Java 11

Should be an easy switch, and will allow Peter to write wrappers for nshmp-haz models

(Maybe?) Switch to default Gradle directory structure

Currently we have all of our regular source code in 'src' and test source code in 'test':

└── src
    └── org
        └── opensha ...
└── test
    └── org
        └── opensha ...

Gradle would instead structure that as:

└── src
    └── main
        ├── java
        │   └── org
        │       └── opensha ...
    └── test
        ├── java
        │   └── org
        │       └── opensha ...

That structure is more meant for projects that blend multiple languages, which is more common these days but I don't think we'll go there. The benefit of going the recommended route is that it makes things simpler for downstream projects (e.g., Kiwis), and less complicated gradle configuration files. I'm conflicted here on this one, and welcome feedback.

Get rid of 'scratch' in the main opensha repository

Finally get rid of the 'scratch' package in the main opensha repository. This involves moving UCERF3 to new packages (probably org.opensha.sha.earthquake.faultSysSolution.UCERF3), as well as any other code that snuck into apps/core/commons from our old scratch package.

Update gradle wrapper

Self explanatory

Migration plan

After we converge on a plan, everyone will need to merge in any changes they're currently working on. Then I will create a new repository and merge in the old projects following this guide:

https://saintgimp.org/2013/01/22/merging-two-git-repositories-into-one-repository-without-losing-file-history/

That will allow us to maintain the history of each file, and I have successfully prototyped this with small test repositories. It gets a little complicated for files that exist in both repositories, so I would first create a new branch in each project that contains only the source code unique to each project. I would then merge those code-only branches into a new repository ('opensha'). Once this is done, I will update the README's of each old repository to point them to the consolidated one, and archive them through GitHub (which makes them read-only).

Ideally this would all happen in the next week or so, and I welcome feedback on the plan. Please let me know if you have any outstanding commits/merges that need to be sorted out before this can take place. Thanks for reading!

[chrisbc]

@kevinmilner @voj and I are raising a couple more PRs that we'd like to merge upstream.

Kiwi feedback:

• this looks very positive as these simplifications will greatly streamline our dev workflow. Yay!
• we support the gradle layout standardisation, and you might want to look at resources also?
• NZSHM is moving into a pretty critical phase over the next few weeks, but once the aforementioned PR's are cleared we should be in good shape for a while.

Finally, Oakley & I would like to help if you think it'd be useful. Perhaps we could set up a virtual-workshop day for coding-testing-building etc, to help get through the bulk refactoring?? I'm sure we kiwis could migrate to California time just for a day :)

[pmpowers-usgs]

Great plan!

• Like it or not the Gradle/Maven layout is so well established it makes little sense to do otherwise; resources are stored at the same level as languages; we've ended up with a few repositories now that include a dash of Python.
• Java11 should be painless and there are a lot of JDKs for users to chose from

[kevinmilner]

Thanks for the feedback, everyone! It looks like there is consensus in switching to the default directory structure, and that the convention goes well beyond gradle, so we'll do that as well.

I did a test merge of commons and core which was successful. You can see the result of that test here. Unfortunately, when I renamed the directories to conform with the standard directory structure, all of the history that had been carefully copied over from the original repositories was detatched. Git can attempt to find old history if you use the git log --follow command, but it won't show up on GitHub.

I then tried a fancier method that uses the (somewhat dangerous) git filter-branch tool, and it went very well. That test repository is here, and I include all of the steps to create it in the README. In the latter repo, I also set up gradle and the project can be built and imported into eclipse. Gradle 7 has some differences, so I had to change some of the build tasks ('*compile' -> '*implementation'). The gradle configuration here still has the build-commons.gradle file, which will be used by dev/cybershake/oaf.

I'm leaning toward going forward with the fancy preserve-history move as it seems to work well.

I'll wait until at least Wednesday to give anyone else who hasn't chimed in a chance to. @chrisbc and @voj, let's discuss on our call tomorrow.

[kevinmilner]

The migration is (provisionally) complete, and I am now archiving this repository. See the new consolidated repository here: https://github.com/opensha/opensha