[BUGFIX] Use pip==20.0.2 to match core st2 pin #687
Conversation
We have been running tests with pip==20.0.2 in v3.3.0 and v3.2.0. But we have still been installing pip 19 in production. That is very likely to cause installation errors for packs in production that would not exist during testing (eg if a pack's dep uses a pip feature that is not available in pip 19, which is becoming more common). See these commits in StackStorm/st2: - a6c4b89373e01f1c38396ce65d9dda3e3be0e842 - c9f6638c491383e7587ab44f76673c81805aaa13 - e1804fb666cd9d41a77a1e776fe6bb0a1e1b949b - 40827e7fdb04588bfc12ce4da7c7a9bc7e9c4ae3 We have not been testing with anything newer than pip 20.0.2, however, so upgrading to a newer version is dangerous. Since we've already been testing with 20.0.2, we should start with making pip consistent across all of our infrastructure and then we can worry about upgrading. Here is a list of places that need to be updated when we do upgrade pip: https://github.com/StackStorm/st2/wiki/Where-all-to-update-pip-and-or-virtualenv
|
Hm, it's failing with [package: st2] Collecting pymongo==3.10.0 (from -r ./requirements.txt (line 48))
[package: st2] ERROR: Could not find a version that satisfies the requirement pymongo==3.10.0 (from -r ./requirements.txt (line 48)) (from versions: none)
[package: st2] ERROR: No matching distribution found for pymongo==3.10.0 (from -r ./requirements.txt (line 48))
[package: st2] Traceback (most recent call last):
[package: st2] File "/usr/bin/dh_virtualenv", line 110, in <module>
[package: st2] sys.exit(main() or 0)
[package: st2] File "/usr/bin/dh_virtualenv", line 90, in main
[package: st2] deploy.install_dependencies()
[package: st2] File "/usr/lib/python2.7/dist-packages/dh_virtualenv/deployment.py", line 177, in install_dependencies
[package: st2] subprocess.check_call(self.pip('-r', requirements_path))
[package: st2] File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
[package: st2] raise CalledProcessError(retcode, cmd)
[package: st2] subprocess.CalledProcessError: Command '['/tmp/st2-20210304-6-9j53ti/st2/debian/st2/opt/stackstorm/st2/bin/python', '/tmp/st2-20210304-6-9j53ti/st2/debian/st2/opt/stackstorm/st2/bin/pip', 'install', '--log=/tmp/tmp78RH06', '--find-links=/tmp/wheelhouse', '--no-index', '-r', './requirements.txt']' returned non-zero exit status 1
[package: st2] debian/rules:65: recipe for target 'override_dh_virtualenv' failedOn Bionic. No idea why yet. It almost looks like it's trying to use unsupported Python version, but pymongo 3.10.0 claims it still supports Python 2.7. |
|
OK, I found the root cause: So the pip which is used for virtualenv is still old one which doesn't support manywheel2014 wheel format used by pymongo. Looking into a fix (I believe we just need to specify right pip path so it uses Python 3 pip and not old system one). |
|
Finally got it to work - StackStorm/st2packaging-dockerfiles#104 (comment). Spent many more hours that I would hope on that yesterday. Even though we pinned pip in various places that was not enough and created virtualenv was still using incorrect versions (and those were not consistent across deb distros - Xenial was using 21.0 and Bionic was using 19.1 - nasty and a ticking time bomb for sure). In short - that code was broken and dangerous for a long time and some of my recent changes just exposed those long standing issues. |
Xenial and Bionic.
|
Yay, finally all green and correct pip versions being used across the board 🎉
What supposed to be 10 minute change turned into many hours :D |
We have been running tests with pip==20.0.2 in v3.3.0 and v3.2.0. But we have still been installing pip 19 in production.
That is very likely to cause installation errors for packs in production that would not exist during testing (eg if a pack's dep uses a pip feature that is not available in pip 19, which is becoming more common).
See these commits in StackStorm/st2:
We have not been testing with anything newer than pip 20.0.2, however, so upgrading to a newer version is dangerous. Since we've already been testing with 20.0.2, we should start with making pip consistent across all of our infrastructure and then we can worry about upgrading.
Here is a list of places that need to be updated when we do upgrade pip:
https://github.com/StackStorm/st2/wiki/Where-all-to-update-pip-and-or-virtualenv