MPI variants

[djsutherland]

We should document the right way to do MPI variants now.

conda-forge/lammps-feedstock#1

[jakirkham]

I think @minrk has thought about this more than anyone else at this point. Maybe he has some advice?

Also @msarahan has been reworking blas in defaults. So expect he probably also has some thoughts on how to proceed (AFAICT the problems are quite similar).

Also @mcg1969, I, and others have been talking about this problem in the context of blas for a long time (though it has come up in other cases like vc and MPI features). He's done a lot of work on conda's solver and so has better insight than most into what sort of things work or not.

If anyone pinged or those watching know others that should be included in this conversation, please feel free to reach out. Trying to avoid pinging too many people myself. :)

[mcg1969]

It turns out that simply instantiating multiple variants isn't a particular problem. Just make sure they are all built to the same model and version number.

The real challenge is somehow establishing and maintaining the preference. For that, we came up with an approach for the Anaconda R / MRO preferences that I think works well and exploits track_features in a novel and surprisingly safe manner.

My time is short right now but if there is interest I can document the idea further.

[mcg1969]

(To be clear: do not assume that just giving one variant a higher build number, or even version number, will guarantee it is selected by default. We learned that the hard way)

[jakirkham]

Thanks for sharing, @mcg1969.

Is there somewhere people should look for details on the track features portion/preference selection portion?

Note: We may need to adjust the upload script to allow hash changes in the upload.

[mcg1969]

We designed by whiteboard and didn't document our efforts I'm afraid. But the result of our effort was implemented in our R package set (see our R channel).

Let me sketch it out. The key is the creation of a set of mutually exclusive "mutex" metapackages: the same package name, different build strings. There is one per variant. The packages have no dependencies. Each of the variants that you build includes its appropriate metapackage as a dependency.

To enable a strong "default" preference, the non-default packages all include a track_package value. This ensures that conda's "feature minimization" protocol will prefer the default package. It's not good enough to just give the default metapackage a lower build or version number, because that will generally not be given sufficiently high priority. The feature minimization, on the other hand, has a very high priority.

Now, in order to handle the manual selection of a non-default package, you can create a set of selector metapackages. Each of these packages has a different name, and include the corresponding mutex metapackage as a dependency. That way, if you install a selector metapackage, constraint conflicts will prevent it from being replaced; and this forces that variant to be pinned.

Now, selector metapackages may not be necessary in this case, if there are a natural set of packages that can serve as selectors. For example, perhaps the different MPI implementations can serve as selectors themselves. If so, just make sure they have a dependency on their corresponding mutex package.

[croth1]

I included Min's docs in MPI in our documentation here: http://conda-forge.org/docs/maintainer/knowledge_base.html#message-passing-interface-mpi. Can somebody double-check and check whether things are missing or need to be more clear?