conda smithy issue on Windows

[ivoflipse]

I tried to rerender a feedstock using conda smithy on Windows, but I got the following error:

> conda smithy rerender
Using Anaconda API: https://api.anaconda.org
Fetching package metadata ............Could not find URL: https://repo.continuum.io/pkgs/msys2/linux-64/

My conda configuration is as follows:

> conda info
Current conda install:

             platform : win-64
        conda version : 4.1.11
    conda-env version : 2.5.2
  conda-build version : 1.21.14+0.g4dfebe9.dirty
       python version : 3.5.1.final.0
     requests version : 2.9.1
     root environment : C:\Users\ivofl\Miniconda3  (writable)
  default environment : C:\Users\ivofl\Miniconda3
     envs directories : C:\Users\ivofl\Miniconda3\envs
        package cache : C:\Users\ivofl\Miniconda3\pkgs
         channel URLs : https://conda.anaconda.org/t/<TOKEN>/clinicalgraphics/win-64/
                        https://conda.anaconda.org/t/<TOKEN>/clinicalgraphics/noarch/
                        https://conda.anaconda.org/t/<TOKEN>/conda-forge/win-64/
                        https://conda.anaconda.org/t/<TOKEN>/conda-forge/noarch/
                        https://repo.continuum.io/pkgs/free/win-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/win-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
                        https://repo.continuum.io/pkgs/msys2/win-64/
                        https://repo.continuum.io/pkgs/msys2/noarch/
          config file : C:\Users\ivofl\.condarc
         offline mode : False
    is foreign system : False

[ocefpaf]

I believe that MSYS2 is now in defaults so I guess you can remove the channel:

https://repo.continuum.io/pkgs/msys2/win-64/
://repo.continuum.io/pkgs/msys2/noarch/

Also, https://repo.continuum.io/pkgs/msys2/linux-64/ will never exists as that is Windows only.

However, it would be nice if conda-smithy could be robust enough to still rerender even when a channel does not exist.

[ivoflipse]

Strange though, since it's not in my .condarc

[ocefpaf]

Strange though, since it's not in my .condarc

Not sure how things work on Windows but multiple installations of Ana/miniconda can put that file and different places. I recommend using the conda-config CLI to deal with things like this.

Do you want to try conda config --remove channels msts2 --force as a workaround?

(Still conda-smithy should not fail because of that.)

[ivoflipse]

Error: 'msys2' is not in the 'channels' key of the config file

I presume it's hard coded into conda or one of the others at the moment.

Indeed in conda/base/constants.py:

if Platform.from_sys() is Platform.win:
    DEFAULT_CHANNELS = ('https://repo.continuum.io/pkgs/free',
                        'https://repo.continuum.io/pkgs/pro',
                        'https://repo.continuum.io/pkgs/msys2',
                        )

[ocefpaf]

So you cannot do anything there. We really need to get that fixed in conda-smithy. Thanks for finding that!

[jakirkham]

cc @Korijn

[jakirkham]

This is a quite a bit more complicated than a bugfix for conda-smithy. In fact, this involves conda-smithy, conda, and how the msys2 channel was setup. To really explain why this happens, one must first actually understand what conda-smithy does.

To re-render a feedstock, conda-smithy has to figure out in advance what will be built on each platform. For example, does it use Python, does it use numpy x.x, which versions of these does it use, does it only build on certain platforms, etc. However, as we always use one platform to run conda-smithy, it must actually masquerade as the other platforms on our OS and must convince conda and conda-build that we are actually on those platforms when we are not. This is done through the fudge_subdir function.

Now we don't actually specify the channels to use, but we do specify the platform. It is then conda that takes the channel listing given (i.e. DEFAULT_CHANNELS listed above) and attaches the platform to each one. So when masquerade as linux-64, it will check for the following channels. The last one it only checks on Windows (note it is intentionally a 404 as it does not exist hence this issue).

• https://repo.continuum.io/pkgs/free/linux-64/
• https://repo.continuum.io/pkgs/pro/linux-64/
• https://repo.continuum.io/pkgs/msys2/linux-64/

For both the free and pro URLs, we know that all the platforms are defined and fetching them has never been a problem. Some packages may not exist in each one, but that is more of a support question. However, msys2 is a different story. We only find the following URLs for noarch, win-32, and win-64. There are no other URLs for other platforms. While this isn't surprising this, it is the crux of our problem.

• https://repo.continuum.io/pkgs/msys2/noarch/
• https://repo.continuum.io/pkgs/msys2/win-32/
• https://repo.continuum.io/pkgs/msys2/win-64/

So how do we fix this? There isn't really a fix for conda-smithy per se as this ability to masquerade as a different OS is core to its functionality. Perhaps if conda exposed this functionality better that would help, but it wouldn't necessarily fix the channel issue that we are encountering. Instead the easiest fix IMHO would be for msys2 to get dummy directories for linux-64, osx-64 and possibly their 32-bit variants as well. This would allow conda-smithy to continue doing its job and be the least invasive change.

cc @msarahan @mcg1969 @kalefranz @mingwandroid

[jakirkham]

Raised this upstream issue ( https://github.com/Anaconda-Platform/support/issues/76 ).

[jakirkham]

Just put together PR ( #401 ), which is overriding channel settings. While I suspect it might fix this issue, I don't have any way of testing. So would appreciate if others struggling with this issue take a look and see if it does fix it or not. Note there may be other Windows issues that need to be addressed. Am merely referring to the msys2 issue during re-rendering.

[jakirkham]

The fix for this is released in 2.0.0 FYI. Am very excited that Windows users will be able to re-render. 🎉 Am very interested to hear feedback from all of you on your experiences using the new conda-smithy and iterate on it as needed. 😄