How to use rules_nixpkgs for python packages

[smukherj1]

Hi repo owners & @mboes,

I'm from the remote build execution team at Google and we are evaluating nix as a hermetic package manager for remote builds. I'm trying to create a project that uses rules_nixpkgs to download the flask python package and use it in a python hello world server app.

I created https://github.com/smukherj1/eval_nixpkgs. This project has a Dockerfile that creates a container with bazel and the nix package manager. I have a script "docker_run.sh" which takes a single argument which should be the name of the image created by the Dockerfile. Note that this script needs to be in your current directory when you execute it. The script tries to build the python app using bazel in a docker container.

Currently, I get the following error:-
ERROR: Analysis of target '//:app' failed; build aborted: no such package '@python2.7-Flask-1.0.2//': Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/ec321eb2cc2d0f8f91b676b6d4c66c29/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 82 ":".join([((path_name + "=") + str(ctx.pa...()]) File "/root/.cache/bazel/_bazel_root/ec321eb2cc2d0f8f91b676b6d4c66c29/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 83, in ":".join str(ctx.path(target)) File "/root/.cache/bazel/_bazel_root/ec321eb2cc2d0f8f91b676b6d4c66c29/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 83, in str ctx.path(target) Not a regular file: /workdir/external/nixpkgs

Now this is almost certainly because I created the bazel project wrong. Any guidance on where I went wrong would be greatly appreciated!

Thanks in advance for your help :)

[mboes]

Thanks for reaching out. I can certainly see that using Nix as the package manager in the context of RBE could be a great fit.

The error you're seeing is because you flipped the dictionary order. Instead of

    repositories = { "@nixpkgs": "nixpkgs" },

it should be

    repositories = { "nixpkgs": "@nixpkgs//:default.nix" },

The source of your confusion was likely a documentation bug in an example in the README (now fixed).

Now, it turns out that the above fix is not sufficient. There's on more change you have to make, which I'll explain shortly. You can find a working Flask app in my fork of your repo: https://github.com/mboes/eval_nixpkgs.

The role of rules_nixpkgs is just to expose Nix packages to Bazel. But then what to actually do with these packages is language-specific. In the case of system libraries like libzlib.so, they are consumed by C++ rules using cc_import. In the case of Haskell packages like conduit, they are consumed by Haskell rules using haskell_import. In the case of Python packages, we're stuck because Bazel's Python rules don't currently provide an analogous python_import rule. Ideally, I'd say something like the below, if I had such a make believe python_import rule:

python_import(
  name = "flask",
  lib_path = "@python2.7-Flask-1.0.2//:lib",
)

py_binary(
  name="app",
  deps=[":flask"],
)

But are we really stuck? It turns out that we can work around this. Since the Python rules provide a py_runtime rule allowing us to select a different Python interpreter, what we can do is ask Nix to concoct a custom Python interpreter with all the Python packages we want in-scope, as if they were packages that just ship with Python and are available by default like the standard library. With this "runtime" in hand, we can ask all Python rules to always use, by passing --python_top on the command-line.

In the repository I link to above, I can run your Flask app as follows:

bazel run --python_top=//:python-2.7.15 :app

Now, as a meta-comment, it's not entirely clear to me that this particular example is a great use case for Nix, if you had a pip_import rule or somesuch, which could build third party dependencies. But perhaps you are evaluating Nix because you want to avoid calling out to Pip? Or because you want to tightly control C dependencies of the Python packages?

[mboes]

@smukherj1 Can you confirm? If so we can close this ticket.

[smukherj1]

@mboes Yes confirming my test project built successfully when I implemented your suggestions. Thanks for the help! Sorry for forgetting to follow up here