Getting psutil installed is difficult

[ellieayla]

The dd-trace-py library depends on the psutil library since #819

dd-trace-py/setup.py

Line 70 in 2b9a086

'psutil>=5.0.0',

The psutil library package has C extensions for platform-specific functionality. The package requires a working compiler (eg GCC) and other C header files to build those platform-specific bits. If the package is installed in a container, the container needs all that stuff, which takes up space and provides no value at runtime. Building a wheel requires those same dependencies to build, but installing and using the wheel does not require those dependencies at runtime. That's better.

Installations of psutil on Mac/Windows have wheels already available in https://pypi.org/project/psutil/#files and can be used directly. But there are no wheels available for Linux.

We use dd-trace-py with Gunicorn+Flask in linux containers. As a result, those containers need to pull in GCC, and larger than they otherwise would be. Doing multi-stage builds (stage one has GCC and produces a wheel, stage two consumes the wheel and produces a web app) is possible, but further complicates docker image build-time caching.

Efforts in giampaolo/psutil#824 and giampaolo/psutil#1126 and MacPython/psutil#1 have been trying to produce a psutil wheel for a while. As Datadog is relying on this library, can Datadog's python experts help?

Related;

This text was originally written in a support ticket (288427) on 2019-12-10, and PR #1160 was created on 2019-12-11. @brettlangdon seeks to vendor psutil into the dd-trace-py wheels, rather than produce/maintain a standalone psutil wheel. I'm retroactively creating this issue to be closed by that PR, in hopes that the description and citations are helpful to someone else.

Vitae;

• Using Python 3.6 almost everywhere, though we're planning on switching to 3.8 (for PEP 589 typed dictionaries).
• Tracer version is pinned to ddtrace=0.27.1 right now, recognize 0.31.0 is available and there's been a lot of changes.
• Build and runtime environments for are all containers. Production is in kubernetes; CICD testing is in bespoke docker on ephemeral Jenkins worker VMs.
• We are not using ddtrace-run. The entrypoint for our python webapp does other things that preclude it. Instead we import datadog & ddtrace and setting config from environment variables very similar to how ddtrace-run does.
• Linux flavour is Ubuntu, with the oldest being ubuntu:bionic-20190307 with many modifications. Newer microservices are using a slim variant.

[brettlangdon]

hey @alanjcastonguay thanks for opening up this issue, indeed I opened #1160 as a potential solution to this problem.

This doesn't mean we are against helping the psutil maintainers get wheel building setup and going, but the vendoring approach at least seemed like something in our control for now that we can try and push through without many roadblocks.

[ellieayla]

Certainly, there's less red tape this way. Anything learned from this effort can be borrowed too.

[brettlangdon]

Probably won't be able to borrow as much as you'd think.

Building wheels isn't terrible, we have scripts that do that. The tricky part here is telling our setup.py how to build the c-extensions for psutil and making sure our installation doesn't fail if psutil install fails.

This is mostly all we do to build wheels: https://github.com/DataDog/dd-trace-py/blob/a5d21865f8b9ff3806c1253a72b91d2f201965e5/scripts/build-dist

That followed by a twine upload is all it takes.

[brettlangdon]

(but, we also only publish manylinux wheels, not sure if that will cause issues for psutil since it has distro specific c-extensions)

[ellieayla]

That's far simpler. I think the early parts of other efforts predate manylinux1 working, so they were fighting with CentOS for a while while disentangling OSX-specific things? I'm happy with manylinux1, though maybe manylinux2010 will become important soon.

[brettlangdon]

@alanjcastonguay we have just released 0.34.0 which vendors psutil so it gets built into our wheel.

Give it a try and let us know if it solves your issue!

[ellieayla]

Fantastic; ddtrace==0.34.0 installs perfectly everywhere I wanted it to.