Integrate this project into psutil release workflow

[giampaolo]

This is the continuation of giampaolo/psutil#824
More or less, the goal should be to consolidate the workflow of this repository so that when a new psutil version is out (or we're close to it) we should be able to easily do the following:

1. build Linux wheels for all supported Python versions (2.7, 3.3, 3.4, 3.5, 3.6 both 32 and 64 bit)
2. same for OSX
3. uploads them on rackspace or whatever hosting solution as long as it's publicly accessible
4. download them from rackspace and upload them on pypi

Ideally, I would like to do this in most automated way as possible (e.g. via make pre-release and make release commands).

The hypothetical make pre-release command should execute steps 1, 2 and 3. IMO it is not necessary that psutil tests run because they're slow, especially on OSX and we already have a travis/appveyor setup configured in the original psutil project.

The hypothetical make release command should download the wheels from rackspace (all of them) and upload them on PYPI via twine. This task is already solved in the psutil project (for windows wheels).

Extra: (note: still not sure) once this is done and proven to be robust enough we may probably want to also upload sources (sdist) from here and possibly also Windows wheels, removing those 2 responsibilities from the original psutil project.

[matthew-brett]

For the projects I am involved in, I really want to run the tests on the installed wheels, to catch problems like the ones we hit here. Just for example, this setup runs tests on 32-bit Linux, which is somewhat difficult to as part of a typical travis-ci setup. For example, Scipy just hit a set of 32-bit Linux failures, that we only found with the daily wheel tests.

For the release procedure, what do you think of the procedure here : https://github.com/MacPython/cython-wheels/blob/master/README.rst#quickstart . Is that automated enough?

[giampaolo]

OK, fine with me (tests for all combos).
And yes, something like trigger_build.py looks clever enough for this purpose.
I suppose the first step would be to fix that segfault error. I don't see it happening on giampaolo/psutil's Travis. Maybe there's something particular about the setup (I mean in travis.yml)?.

Also I see:

    - os: linux
      env:
        - MB_PYTHON_VERSION=2.7
        - UNICODE_WIDTH=16

What is it? Do we need it?

[matthew-brett]

I can replicate the segfault with:

# Get manylinux1 docker image, start temporary container
docker pull quay.io/pypa/manylinux1_x86_64
docker run -ti --rm -v $PWD:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

Then, inside the container:

# Select Python 3.5 (just for example)
export PATH=/opt/python/cp35-cp35m/bin:$PATH
git clone https://github.com/giampaolo/psutil
cd psutil
pip install .
PSUTIL_TESTING=1 python psutil/tests/__main__.py 

So - I guess it must be something about the code linked in this old Linux container?

[giampaolo]

I cannot debug this problem right now but in order to move this forward I temporarily disabled that test:
giampaolo/psutil#1126

[giampaolo]

I did this:

# Get manylinux1 docker image, start temporary container
docker pull quay.io/pypa/manylinux1_x86_64
docker run -ti --rm -v $PWD:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

...and tested with Python 2.7 but can't replicate the segfault.

[giampaolo]

...and I completely forgot how this is supposed to work. =)
I am working in a branch (giamptest). I manage to run travis builds:
https://travis-ci.org/MacPython/psutil/builds/297178281
What next? Am I supposed to see a wheel uploaded on http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com/?

[matthew-brett]

Hmm - at the end of the log it says that the key for the travis upload is missing. I'm not sure why that is. I can try resetting it again.

[matthew-brett]

See #2

[giampaolo]

I still see:

0.19s$ python -m wheelhouse_uploader upload --local-folder ${TRAVIS_BUILD_DIR}/wheelhouse/ $UPLOAD_ARGS $CONTAINER
WARNING: secret API key missing: skipping package upload

[giampaolo]

Sorry for being annoying but... any chance you can take a look at this any time soon?

[matthew-brett]

That's OK - where do you see that message? Can you point to the build? If on your repo, that's expected, if on MacPython/psutil, then not expected.

[giampaolo]

Here at the very bottom: https://travis-ci.org/MacPython/psutil/builds/300553111
The message is collapsed so you have to click on it.

[matthew-brett]

Strange - I can't explain that. Try this instead?

  secure: "CJoUuosDNWnMgdbF8V4N+7x0L1o77KspnUfkRS0bK8Sd2TtLaImrtf1UdOzGSoL/EknG/m8lRHE9znKGwPWKfMPbWVOtWhfYT4xrnoUQ2yKW02fx/m5OiozQoCGYiONws9Nv1nFvzJ6PnyLGdhR5SKv4rWWqEZPSGGEZ4KtU51s="

[giampaolo]

Same probelm.

[matthew-brett]

Sorry - my fault - see #3

[giampaolo]

Thanks Matthew. It now uploads wheels!
I have another problem though. I'm trying to debug a segfault occurring on CentOS:
https://travis-ci.org/MacPython/psutil/jobs/307405624
I'm not sure I understand how this docker/centos thing works. Is there a way for me to run the same centos version locally so that I can debug the problem?

[matthew-brett]

If you are running on Linux or Mac, it shouldn't be too much trouble getting docker working on your machine. After that, if you're lucky, the segfault also occurs when building on the test image (an Ubuntu Trusty image), and you can do all the debugging there. Otherwise, you'll have to build the wheel first on the manylinux image, and then test it on the test image, as the tests are doing.

For example, here's some instructions for scikit-image, in this case debugging a 32-bit error:

https://github.com/scikit-image/scikit-image/wiki/Testing-in-32-bit-environment

Please let me know if I can help with the machinery. Are you on Linux or MacOS?

[giampaolo]

Thanks Matthew.
Oh is that a Ubuntu 14.04? I thought it was CentOS.
Anyway, I just want be able to replicate this segfault manually:
https://travis-ci.org/MacPython/psutil/jobs/307405624
I am a Ubuntu 16.04 user (and not exactly familiar with docker =)).

[matthew-brett]

Do these commands do the job?

sudo apt-get install     linux-image-extra-$(uname -r)     linux-image-extra-virtual
sudo apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common 7  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
apt-get update
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get install docker-ce
sudo docker run hello-world
sudo docker run -ti --rm matthewbrett/trusty:64 /bin/bash

That should start you up in the testing container - then you'd build and install psutil and see if it segfaults. I got the instructions from https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/

[matthew-brett]

Sorry - in more detail - here is me trying to replicate the error with docker containers.

I make sure I'm in the pustil directory, with a git checkout.

First I build the wheel in the manylinux docker container:

# Start the container
docker run -ti --rm -v $PWD:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

# In the container
cd /io  # The psutil directory that was my working directory outside the container
/opt/python/cp27-cp27mu/bin/pip wheel .
auditwheel repair psutil-5.4.2-cp27-cp27mu-linux_x86_64.whl

Then I test in the test docker container (trusty):

# Start the container
docker run -ti --rm -e PYTHON_VERSION=2.7 -v $PWD:/io matthewbrett/trusty:64 /bin/bash

# In the container
pip install /io/wheelhouse/psutil-5.4.2-cp27-cp27mu-manylinux1_x86_64.whl 
pip install ipaddress mock
PSUTIL_TESTING=1 python -Wa /io/psutil/tests/__main__.py

For these, I get lots of test errors - I'm hoping you understand those better than I do ...

[giampaolo]

Thanks for the accurate instructions. I cannot reproduce the segfault. Were you?

[matthew-brett]

I got lots of errors, but not the segfault. I was wondering whether the errors had disguised the segfault, but I didn't understand the errors well enough to fix them.

[dpeschman]

Hello. I found this while looking for a way to use psutil without installing gcc. Were you guys successful in your efforts?