Cannot import requests anymore

[gabm]

Issue: Since requests is a noarch package, i cannot import requests anymore... Is there any "trick"?

Environment (conda list):

$ conda list
# packages in environment at /root/miniconda3:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                     py_1    conda-forge
beautifulsoup4            4.6.0                    py35_0    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
certifi                   2018.4.16                py35_0    conda-forge
cffi                      1.11.5                   py35_0    conda-forge
chardet                   3.0.4                      py_1    conda-forge
conda                     4.5.6                    py35_0    conda-forge
conda-build               3.10.9                   py35_0    conda-forge
conda-env                 2.6.0                         0    conda-forge
conda-verify              2.0.0                    py35_0    conda-forge
cryptography              2.2.1                    py35_0    conda-forge
filelock                  3.0.4                    py35_0    conda-forge
glob2                     0.6                        py_0    conda-forge
idna                      2.7                        py_1    conda-forge
jinja2                    2.10                     py35_0    conda-forge
libffi                    3.2.1                         3    conda-forge
markupsafe                1.0                      py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
openssl                   1.0.2o                        0    conda-forge
patchelf                  0.9                           2    conda-forge
pip                       9.0.3                    py35_0    conda-forge
pkginfo                   1.4.2                      py_1    conda-forge
psutil                    5.4.6                    py35_0    conda-forge
pycosat                   0.6.3                    py35_0    conda-forge
pycparser                 2.18                     py35_0    conda-forge
pycrypto                  2.6.1                    py35_1    conda-forge
pyopenssl                 18.0.0                   py35_0    conda-forge
pysocks                   1.6.8                    py35_1    conda-forge
python                    3.5.5                         1    conda-forge
pyyaml                    3.12                     py35_1    conda-forge
readline                  7.0                           0    conda-forge
requests                  2.19.1                     py_1    conda-forge
ruamel_yaml               0.15.35                  py35_0    conda-forge
setuptools                40.0.0                   py35_0    conda-forge
six                       1.11.0                   py35_1    conda-forge
sqlite                    3.20.1                        2    conda-forge
tk                        8.6.7                         0    conda-forge
urllib3                   1.23                     py35_0    conda-forge
wheel                     0.31.1                   py35_0    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.7                         0    conda-forge
zlib                      1.2.11                        0    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : None
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.5.6
    conda-build version : 3.10.9
         python version : 3.5.5.final.0
       base environment : /root/miniconda3  (writable)
           channel URLs :
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /root/miniconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.6 requests/2.18.4 CPython/3.5.5 Linux/4.4.0-122-generic ubuntu/16.04 glibc/2.23
                UID:GID : 0:0
             netrc file : /root/.netrc
           offline mode : False

[ocefpaf]

We can revert that. It is unfortunate that we cannot use noarch there. Thanks for reporting.

[gabm]

Thanks for your fast answer..

But one question comes to my mind: How is python supposed to find that module in site-packages .. its not even in lib/..?

I see it in ~/miniconda3/pkgs/requests-2.19.1-py_1/site-packages/ while an older version was in ~/miniconda3/pkgs/requests-2.11.1-py35_0/lib/python3.5/site-packages/

[jakirkham]

Thanks @gabm. Agree it should be reverted. Thanks for addressing this, @ocefpaf.

[jakirkham]

Basically conda does the magic necessary to install noarch: python packages. One part of this is moving the site-packages contents into lib/pythonX.Y/site-packages. However this runs into trouble if the package in question is a dependency of conda. Hence the issue you have seen.

cc @conda-forge/core (for awareness)

[gabm]

Thanks for the exeplanation, the revert solves this

[mbargull]

@jakirkham / @ocefpaf, can you elaborate on this issue? Do you think conda did not install the noarch: python variants of these packages correctly? If that really is the case, I'd like to have a reproducer so I can fix this in conda. Currently I have no idea what this is all about..
Also, many of those pull requests that reference this issue are for packages that are in no way direct or transitive dependencies of conda!?! So why say they would be ❓ 😕 😵 ❓

[CJ-Wright]

xref: regro/cf-scripts#265 (comment)

[mbargull]

xref: regro/cf-scripts#265 (comment)

But a build dependency is not "a dependency of conda"...

[CJ-Wright]

I'm confused by your comment.

That list is are all the packages who are required by conda it are requirements of the requirements. If my reading of the conda meta is correct the only build requirements are python and m2 things. Are many of the packages in that list m2 dependencies?

I'm very curious where this issue comes from as well. Is the restriction of arch only for the build/host dependencies of conda? Is it recursive upwards?

[mbargull]

Build dependencies, no matter if for conda itself or one of its (transitive) dependencies, should not play any role. The list of packages you provided in the comment you referenced contains quite a few build tools like m4, automake, setuptools_scm, ... It's very unlikely that those will ever be installed by a conda create -nenv conda..

[mbargull]

I'm very curious where this issue comes from as well. Is the restriction of arch only for the build/host dependencies of conda? Is it recursive upwards?

No idea. There shouldn't be any restriction at all...

[mbargull]

(only things like constructor can't cope with noarch: python packages -- conda should be fine by installing those, even if they are dependencies of itself)

[mbargull]

Without any way to reproduce, I don't understand what

i cannot import requests anymore

is supposed to mean at all..

[isuruf]

conda dependencies not being noarch was an old restriction. conda=4.2 didn't support noarch. To update to 4.3 which supported noarch, you need to do conda update conda, but that means dependencies are updated too. If the deps were noarch, conda=4.2 would upgrade itself to conda=4.3, but the deps would be broken.

I don't know why requests installation was broken in this case.

[mbargull]

Oh wait, maybe it's was that recent issue with 4.5.7 monkeypatching requests.

[mbargull]

Just look at the recent issues from https://github.com/conda/conda/issues?q=sort%3Aupdated-desc+requests. But that was not related to noarch: python at all... Maybe the OP just came to a false conclusion about it being noarch: python, but it was just coincidence?

[gabm]

It could very well be that I was mistaken. I just encountered this issue and the first (very fast answer) indicated that the reason might be that requests are direct dependencies of conda..

I was on conda 4.5.6 and I have a custom python app that we use since many months without problem. Suddenly, after the noarch update, my app stopped working with ImportError: No module named 'requests'. This was a direct consequence of the update...

Now I tried to reproduce this. As the requests 2.19.1 py_1 disappeared from the repo i rebuilt it and installed it.. Now, (un)fortunately the error cannot be reproduced anymore. Is there any way to get this version from the conda ci?

[gabm]

Okay guys.. I found the problem... I am deeply sorry for the confusion i caused, it is a problem on our side... for future reference, here the whole story:

• we use docker images to build our stuff
• when the docker image gets built, we use an old installer (conda 4.2.12)
• as part of the build process we update the conda and conda-build version with conda update --all

Using this procedure, we installed the noarch packages requests (idna, chardet) using conda 4.2.12 which does not throw errors, but somehow messes up the layout on the harddrive. Our workaround will either be "using a newer installer" or "only updating conda first, then the rest".

Sorry again and thanks for pointing out that the theory didnt make sense..

[mbargull]

Our workaround will either be "using a newer installer" or "only updating conda first, then the rest".

Yeah, that "conda<4.3 installs but not handles noarch: python packages correctly" issue is a bad one that bit quite a few people previously. Since the (non-prerelease) version of 4.3 got out one and a half years or so, we usually don't see this popping up anymore. And I hope we won't have such changes that break existing installations in the future at all anymore.

@conda-forge/core: IMHO (and unless you're still keen on us constructor and such with conda-forge packages), you can revert those noarch: python reverts again, if you want.

[isuruf]

@mbargull, there's plenty of travis.yml files (1973 in fact) using https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh which has conda 4.0.5. Fortunately most of them do a conda update conda.

[mbargull]

plenty of travis.yml files

What do you mean by that? If I grep on feedstocks I get none. Do you mean in some non-master branches? Either way 4.0.5 is really outdated, so I'm not sure what are you getting at and thus am a bit confused.

[isuruf]

I meant in github not in conda-forge

[mbargull]

Ok, I'm not sure if you really want to concern yourself with those, but whatever you see fit 😉

[isuruf]

I'm just saying that there are conda < 4.3 users and we should make an announcement that we are doing this. I'm in favour of deprecating the use of conda<4.3 even for upgrading.

[jjhelmus]

I'm discussing with the team to see if we can redirect Miniconda-latest-Linux-x86_64.sh to Miniconda3-latest... so that anyone who uses the older links gets a newer noarch supporting conda version.

[isuruf]

@jjhelmus, thanks. I'd suggest redirect to Miniconda2-latest as people expect python2 with it.

[jakirkham]

Is there some reason you were using conda 4.2.12, @gabm? Just haven't had a chance to upgrade or were there some issues that you were trying to avoid in later condas?

[gabm]

actually we had problems when noarch packages didn't work properly (or didn't exist yet?). People used installers with python 3.6 and were not able use our Python software built for 3.5.. that's why we defined to use a certain installer... and never upgraded that as long as it worked.