Using the MSYS2 tools to build conda packages #112
Comments
|
Re having MSYS2 as a package and use it in the builds: +10000 Re patched CPython:
This shouldn't break using pip with wheels. And also not break any of the "how to compile python with native libs on windows"-guides, which users turn to when they have to build stuff for themself (e.g. the patched CPython should still be able to build and afterwards work with a pandas version which is compile the "normal way"). My guess (which is based on compiling a few python native packages and fix a few errors on the way, but no deep knowledge on how compiler and ABI work) is that it will take more time to make it work (e.g. patch out WIN32 #ifdefs which then assume a certain compiler or things like mpl which when run on windows sets up the VS compilers) than what you gain by being able to use a compiler unconstrained by the python version. |
|
Two things, in brief as I'm just heading out:
|
|
Per point 2, @patricksnape, see this comment from @mingwandroid. |
|
Per point 1, @patricksnape, we could investigate using this Though I don't know if we will want to do that or if it will be that bad. Right now we build things like |
|
I expect to add all of the build tools as a matter of priority. So far only those needed for |
|
If you would like to try PRs to those feedstocks, that would be great. I think it will be @ocefpaf and I working with you for the most part as we are maintainers on these sorts of things. Definitely agree those are high priority. That was our mentality here (at conda-forge) too when addressing Mac and Linux. |
Could you please clarify on exactly what idea you are thinking about here? Even if it is hazy, it would be useful to get some insight.
That is certainly the easiest place to start. If you haven't had a chance, I'd really recommend taking a look at our proposed recipe for CPython 3.5 ( conda-forge/staged-recipes#370 ). This is something that @jjhelmus has been working on for a bit and he just recently got Windows support with it. Though I believe it still needs a couple of things. @pelson has worked pretty substantially on this effort too most of which occurred a previous PR. It would be interesting to see what you think would need to happen to this recipe to move it in that direction. This is actually a great time to explore this as we are still working on that recipe.
Sure. I'm betting there will be a bit of patching on the first few passes, which is fine. Normally we have been pretty eager to see patches move upstream. Though given you are a core developer of MSYS2 and there may be a fair bit of patching required, we might need to be more lax just so that we can keep this effort moving at a good pace. Also we may need to come up with a better system for tracking these patches. |
|
The idea was to do a solution for DLLs and Python specifically where it looks at the C-Runtime already loaded by the process and stubs through to the functions there. The patching I refer to is specifically to patch the differences between MSYS2 upstream and Conda. You can see what I mean if you study https://github.com/mingwandroid/conda-recipes/tree/MSYS2-pkg-conversion/msys2/m2-msys2-filesystem After unpacking the MSYS2 package, the patches are applied which allows us to make such modifications as we need to. It means we don't need MSYS2 upstream to put switches in for Conda (nor should they). |
I seriously consider an MSYS-based Python ecosystem as a possible future alternative to VC-based builds. My hope (or, rather, expectation) is that recipes for MSYS will reuse much of the existing Linux scripts. In fact, there should be only one |
Yeah, I imagine that this transition is not happening over night, but I am seriously hoping we can phase out any direct dependence on VS in the long run. Things like CPython will still need this to behave well with wheels so we can't entirely eliminate it, but we should be able to disentangle compiler features from Python version which has been a huge strain on the community.
Yes, this is exactly my hope. Ideally this will lower the threshold for getting productive packagers that are comfortable on *NIX systems trying things on Windows too. It should also significantly lower the maintenance burden with Windows packages. |
|
Sorry to get a bit confused here, but, IIUC: MSYS is a port of the *nix command line tools (a shell and common that gives us the regular make and all that, which is nice. But it also essentially requires using MInGW for the compiler, yes? So this is really a question about using the MInGW compiler, yes? So I have Back in the day, you could use MinGW to build extensions compatible with However, my understanding is that MInGW never got "properly" ported and However, as of recently, the numpy folks have sponsored work on MinGW (or And does this use the MS runtime that Python is already built against? -- As for:
not sure what the "this" is -- but providing a new build of CPython with a But I take it that this is about using MSYS/ MingGW for extensions and libs -CHB On Tue, Apr 19, 2016 at 9:52 AM, jakirkham notifications@github.com wrote:
Christopher Barker, Ph.D. Emergency Response Division |
|
On Tue, Apr 19, 2016 at 11:38 AM, Ullrich Koethe notifications@github.com
https://mail.python.org/pipermail/python-dev/2010-August/102909.html It goes on a for a bit, but the upshot is that making a MInGW build is If someone does want to do that -- build a conda-distro based on fully
-CHB Christopher Barker, Ph.D. Emergency Response Division |
mingw.org and mingw-w64 are two different endeavors. There was a disagreement about whether you could use MSDN for reference, roughly speaking. mingw-w64 has had 64-bit from the start. MSYS2 is a software distro for Windows that's build on mingw-w64 and a fork of Cygwin (which is built on top of mingw-w64 these days and also provides a 64-bit variant). In Conda, our intention is to use it to provide a consistent POSIX build environment that interoperates with native tools, a native GNU toolchain and some GNU libraries without duplicating effort or further splitting the already small GNU-based Open Source on Windows developer community. |
Can someone explain the gains from such a move? This is a deviation from upstream, so this means that conda-forge has the complete burden to maintain forks to basically all tools and libs which assume "win = VSC" , starting with python and conda-build itself (setup.py must find the right compiler). Basically every examples for porting work:
So basically you would have all the efforts to support VC windows in addition to even more effort to support a msys2 port. So what are the gains? |
I'm currently using MinGW as part of my VS builds in two situations:
I consider MinGW (64-bit version) a mature compiler that doesn't cause any headaches -- it feels like its Linux counterpart. Linking of MinGW C/Fortran binaries into VS modules works mostly out of the box with two minor tweaks:
If MinGW worked for the entire Python ecosystem, one could mostly reuse the effort that was already spent on creating Linux recipes. People who are primarily working on Linux (i.e. many in the scientific computing community) might prefer (or rather: will love) this approach to porting their work to Windows. Of course, whether this gain is worth the effort is a matter of judgement. At the very least, some experiments have to be done to determine exactly how big the effort would be. This certainly isn't something I want to pursue now -- I just wanted to point out the possibility. |
|
Interesting that you mention OpenBLAS and needing a Fortran compiler. That's part of what sparked this discussion ( see PR conda-forge/staged-recipes#364 ). A bit of an aside: It is worth noting that the BLAS part of OpenBLAS can be built with CMake (started near the end of 2014/beginning of 2015). However, that doesn't necessarily help people that want the LAPACK part too, which requires a Fortran compiler. Also, it means the loss of the assembly optimizations, which really puts the value in question IMHO. Though for using something easily on Windows, I suppose it still has value. |
|
On Tue, Apr 19, 2016 at 1:08 PM, jakirkham notifications@github.com wrote:
the numpy/scipy folks have been working hard on Mingw and OpenBLAS, and -CHB |
|
On Tue, Apr 19, 2016 at 12:33 PM, Ullrich Koethe notifications@github.com
And does it all work with py3.5 and VS2015?
yup -- that would be nice. In practice, if it's easy to build stuff on Windows at the command line, -CHB Christopher Barker, Ph.D. Emergency Response Division |
|
Sorry I've been silent on this. +1 to investigating using MSYS2 and seeing how hard it is to build everything using it before committing to using it. I'm happy to help with this. |
The The Mingwpy project page list some information on this. Still seems to be a work in progress. Worth keeping an eye on though. |
|
I was about to ask where to follow this progress on OpenBLAS, @ChrisBarker-NOAA, but it looks like @jjhelmus has the answer already. Or are there more places to look? |
|
As you referred to the more general problem of packaging and working with MinGW-64 compilers, I am cc'ing you here @mikofski. Also, I am cc'ing the NumFocus grant authors ( @carlkl @rgommers @matthew-brett @njsmith @ogrisel ) here, as well. Here we are discussing how best to explore the interaction between conda-forge and MSYS2. We are interested in the ability of working with Windows in nearly the same way we do with any *NIX platform (many of whose builds tools are packaged already). While the idea has been floating around, our attempt of building OpenBLAS at conda-forge has raised this more general point of how we work with MinGW-64 compilers and MSYS2. As there now are quite a few packages available to start exploring this problem and some tools to aid in their automated addition, we are very eager to see where and how we can put them to use. Also, related is how this system is maintained. We are planning to have a meeting where we discuss this and other issue at conda-forge. Anyone who is interested is welcome to come. |
|
I think there are sort of two separate issues:
The Intel Fortran Compiler is not free or available to us, so if we want to compile Fortran we need the msys2 toolchain. I think we should focus on how we enable the use of that toolchain for projects that already require it, like ffmpeg and OpenBLAS - rather than patching all projects to build with msys2. |
Last I heard, @mingwandroid wasn't actively working on this. (Correct me if I'm wrong?) The problem is on its way to being solved for python 2.6-3.4 (that's Carl Kleffner's mingwpy work). The problem for 3.5 is that it requires deeper surgery on the guts of mingw-w64, and we don't yet have a real plan for how to make that happen. The most likely prospect atm is indeed getting @mingwandroid to implement this, since he understands the issues and is familiar with mingw-w64's guts; the blocker to this is that higher-ups at Continuum don't consider it a priority, so the best we've managed to get out of them is that they might let him work on it for a few hours a week starting a few months from now if we figure out how to get someone else to pay for it. If you think this should be higher priority then I guess you should tell them that :-).
An intermediate step would be to use msys2 for scripting but continue to call msvc. This would at least let you get rid of the .bat files |
|
I thought that only issue with Python 3.5 was if we want to link in a 'clever' way to the new versionless MSVCRT? If we just want to do the old way and link directly to msvcrt14 that should be possible, no? Seems like a reasonable stop gap - since we have that issue with the other VS versions anyway, so for the time being I don't see the problem with continuing it into Python 3.5 until mingw is sorted. |
|
@njsmith You are correct, I'm not actively working on it. MSYS2 is coming to Conda to address some of the other issues of building software on Windows (POSIX shell, Autotools, GNU toolchain with gfortran, git, CMake, consistency over all those things - we'll end up with more consistency here than the Unixes ironically since there's only one MSYS2). @patricksnape UCRT is quite a lot different to the other CRTs and it'll require a reasonable amount of effort above what was required for e.g. to upgrade to msvc2012. @njsmith and @carlkl are far more knowledgeable about the ins and outs of it since my involvement so far has mostly been trying to figure out when I might get time to look into it properly. Linking directly to the ucrt14 is exactly what you shouldn't do AFAIUI as your executables will break on future updates to the operating system. |
|
Right, starting with msvc 2015 there is no msvcrt at all :-). They changed the names and semantics of a bunch of symbols, they removed a few (or rather: moved them to a private helper library, vcruntime140) that we will want to replace with our own versions, and in general all symbols got rearranged into a whole collection of new libraries. AFAIK there's nothing deeply difficult here, but it's complicated, especially since we would like to do the work upstream in mingw-w64 rather than make a fork that we will have to maintain forever. (mingwpy is effectively a fork, but that work is much less intrusive so tracking upstream is relatively easy, and for several reasons upstream is less interested in helping with mingwpy and more interested in helping with the msvc 2015 stuff, so the calculation is different.) |
|
OTOH if someone wanted to try just wading in and hacking mingw-w64 to work with the UCRT then we might at least learn something. Maybe it will turn out to not be that hard :-). I'm happy to share what I know and give hints (and I guess @mingwandroid and others are as well). |
So how is continuum doing Fortran on Win64 for python 3.5? The intel and while I"m at it what's the best practice for Fortran with Python2.7 an -CHB Christopher Barker, Ph.D. Emergency Response Division |
|
I'm not completely certain, but I'm pretty sure we use the Intel compiler. |
|
That would make sense. FWICT the Intel compiler is pretty nicely compatible with VC. Though I have no experience using it. Also, I don't know what would go into getting that into the VM(s). |
|
@njsmith @mingwandroid and others interested in coming to the meeting next week the date has been corrected from Thursday to Friday. The time is still the same (1400 UTC). |
|
Ok thanks, I'll await the link to the Hangout then. |
|
Hangout link can be found at https://conda-forge.hackpad.com |
|
Thanks @pelson. |
|
I'm a little surprised no one has mentioned this here, but it is possible to build CPython 3.5 using mingw-w64 GCC (and 2.7 as well with a somewhat different set of patches). If you install mingw-w64-python3 in MSYS2, this is what you get. However it requires 90-odd patches that upstream CPython-dev has been very resistant to even consider reviewing the first handful, much less merge and work towards getting this to work out of the box. See https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-python3/PKGBUILD for all the gory details. This is the extreme case of forking a whole new platform for ease of building scientific extensions that need things MSVC doesn't support like Fortran, inline assembly, building with autotools, etc. But it has the advantage that it works today, unlike getting mingw-w64 to link against the VC 2015 runtime. In Julia's case we choose to completely ignore MSVC as it's a lousy compiler that doesn't meet our needs, and it's way too foreign to expect open-source developers who mostly use Linux and Mac to work with. |
|
@tkelman: unfortunately, a Python distribution for Windows that can neither build nor use pypi wheels is unlikely to see much uptake. Also, conda-forge has no control over what compiler is used to build Python itself, unless they abandon anaconda compatibility entirely. |
|
Depends where you'd rather invest your effort. Building a custom python ecosystem of your own, or building a custom compiler toolchain of your own for (partial) compatibility with a different compiler that you don't actually want to use. You're letting the way python.org chooses to build Python binaries drive the decisions here, and it's not a foregone conclusion that it has to stay that way. |
I don't think so. If you want to do serious scientific computing in Windows, pypi is just not good enough, and I'm skeptical that its design can be fixed. conda is the only package manager on Windows whose approach is sufficiently powerful to make this work (although it's still quite a way to go). If conda lives up to our expectations, I believe it will replace pypi on Windows entirely. So, let's make conda and conda-forge as good as possible and not constrain ourselves by upstream's prejudice. |
|
Pip and the wheels situation and the pynativelib proposal are all rapidly improving and I'm sure everyone appreciates the work Nathaniel and others have done to make that possible. And mingwpy will hopefully reach its goals sooner or later w.r.t. MSVC 2015 compatibility, and once it's there the mingw-w64-using community will evaluate whether it's something that will be more broadly useful for things like MSYS2, Julia, CRAN, etc where currently MSVC is just ignored. In the meantime, it seems there are more people capable of working on rebuild-the-world packaging infrastructure than hacking on the guts of compiler runtime compatibility issues. It's at least worth experimenting with, since there are examples of other ecosystems that rebuild everything on their own and do fine without worrying about ABI compatibility with what's posted on pypi. Cygwin has peacefully been building its own set of python packages for years and it works decently for its users. |
|
I've not mentioned MSYS2's mingw-w64-python ports yet since they're not something that we plan to add to Conda as it would cause confusion for our users and also add significantly to my workload, ending up with another Conda Python ecosystem that bears no resemblance to the Linux and OS X ones. That time would be better spent on the effort to make the toolchains compatible. Guido said that if we identified mingw-w64 Python as different from MSVC Python from pip and wheels perspective) then the idea of a mingw-w64 Python wouldn't be rejected outright, but there's still the problem that CPython developers have mostly displayed reactions ranging from disinterest to contempt and fear at the notion of compiling Python on Windows using anything but MSVC. The patches will continue to be maintained, as time allows, by the MSYS2 project and interested contributors are welcome to lend a hand there.. We should discuss this at the conda-forge meeting tomorrow I think. |
|
@mingwandroid, can you make a summary of features/properties you would expect from mingwpy to be included into conda?
|
|
My personal preferences are:
|
|
@Carlk: conda isn't conda-forge. If the conda-forge guys want to use mingwpy then they will. Getting things into conda is something that Continuum try to do so much, instead we try to host our recipes in conda-forge. An exception to this was R on Windows, where we use MSYS2's mingw-w64 libraries directly. I un-package the Pacman tar.xzs and create conda tar.bz2s. The reasons were:
I'm not sure about the "gcc runtime code" qualification. Shared linking is a bit of a requirement for me, otherwise I'm forever building software. Large scale software distributions with static linking don't err, scale. We'd need to track what libraries got statically linked to each package (which conda doesn't do yet) just so we know that when we build library A we also need to rebuild package B, C, D and E. You can't mix -static-libgcc and non -static-libgcc. With MSYS2's mingw-w64 packages we never use -static-libgcc because you can't fix issues in your toolchain (or other gcc libraries, like the CRLF bug in gfortran recently) without rebuilding everything.
This doesn't matter so much in practice. winpthreads is built on top of win32 threads so they're just native threads anyway. You can't share the sharing/exclusion primitives between the two directly, but that's not unexpected. A program can happily run with threads spawned by each though, if it linked to libraries configured each way.
dw2 is faster, esp. for exception heavy C++ so we picked that. Thiago from the Qt project did benchmarks a while ago. sjlj adds a significant register-dumping overhead to every
Internal x87 80bit precision has never struck me as anything of interest. Like you say, those bits get thrown away finally, and more precision can only help while the calculations are being done in the x87 registers. That it's different from MSVC is expected anyway.
I can't see much problem with changing MSYS2's i686 mingw-w64-gcc to use |
If fact we already do 😄 https://github.com/conda-forge/pyspharm-feedstock/blob/master/recipe/meta.yaml#L21
|
|
Just 2 links for all developers expecting extra precision from long double arguments with FPU utilization:
Unfortunately Intel FPU trigonometric functions offers much less precision than expected if used with specific argument domains or used with argument range reduction. This affects long double precision values as well. Today you are better of using SSE2 math code libraries or use dedicated multiprecision libraries if double precision is not sufficient. |
|
Sorry, being really lazy here: does this have implications for which gcc CHB On May 6, 2016, at 4:29 AM, carlkl notifications@github.com wrote: Just 2 links for all developers expecting extra precision from long double https://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion Unfortunately Intel FPU trigonometric functions offers much less precision Today you are better of using SSE2 math code libraries or use dedicated — |
|
Going to close this out. |
@mingwandroid has done an amazing job of packaging MSYS2 so that is works within
conda. See his channel here. Also, see his comment about the work he has done and what still remains for more details.Given how this greatly reduces the gap between Windows and the *NIXes, this makes Windows packaging very accessible to *NIX devs. This should not be undervalued as it has the potential to really increase our bandwidth on Windows by basically the size of our community. Packages that couldn't be built due to CMake files missing or other problems also dissolve away. This has the potential to substantially reduce the effort required to get a package working on all platforms, which makes us more likely to do exactly that.
Also, it has the potential to free this quite painful constraint of compiler and CPython version. That has raised such issues as making custom patched versions of CPython ( #64 ) just to have C++11 features when using Python 2.7. Further this can free us from thinking about VC features entirely, which have had various issues.
It is honestly making me seriously considering switching to MSYS2 with everything.
Would be very curious to hear perspectives from a wide range of people on this. Please feel free to share your thoughts and also bring other people over that I have missed. I have put this on our docket for next weeks meeting.
cc @ocefpaf @pelson @ChrisBarker-NOAA @patricksnape @gillins @msarahan @mcg1969 @isuruf @ngoldbaum @ukoethe @jasongrout @SylvainCorlay
The text was updated successfully, but these errors were encountered: