add a nix flake #1624

teto · 2023-07-27T11:19:40Z

when I packaged alot for poetry, I did it within a ubuntu container and I realized it was not that easy to setup (I've updated instructions in the doc in that same PR). Nix is a package manager aiming at reproducibility and it excels in it.

With nix installed one can run nix run 'github:teto/alot/flake' to test the PR (though it is currently failing).
Likewise nix develop 'github:teto/alot/flake' will give you a development environment with all dependencies available.

Note: this is easy to maintain out of tree. The 2 flake.* files can be moved in a contrib/ folder to not crowd the root directory.

lucc · 2023-07-28T09:43:01Z

I would welcome a flake in the alot repo.

I did set up a wrapping flake repo a while ago with a gtihub action to fetch the latest alot: https://github.com/lucc/alot-flake. Having the flake in the main alot repo would simplify a few things in my opinion:

the workflow in lucc/alot-flake is regularly deactivated by github because the repo does not change for a while, so it would be nice to get rid of the extra repo and the workflow
I have to maintain the building logic outside of the alot repo resulting in divergent states (right now my flake does not handle the new pyproject file I think)
the flake in the alot main repo could be used in CI which would fix some of my trouble in Use GitHub actions #1617

But

the flake is mostly for the benefit of people using the nix package manager which is not python specific (but can be used in ci)
we can still maintain the flake in a separate repo (like I already do), so that's no problem if @pazz is against it.

pazz · 2023-07-28T10:10:20Z

I am not against this at all and happy to facilitate whatever makes usage and development easier. If you continue to maintain this then all the better :) P Should I just merge this then? Quoting Lucas Hoffmann (2023-07-28 10:43:11)

…

I would welcome a flake in the alot repo. I did set up a wrapping flake repo a while ago with a gtihub action to fetch the latest alot: https://github.com/lucc/alot-flake. Having the flake in the main alot repo would simplify a few things in my opinion: • the workflow in lucc/alot-flake is regularly deactivated by github because the repo does not change for a while, so it would be nice to get rid of the extra repo and the workflow • I have to maintain the building logic outside of the alot repo resulting in divergent states (right now my flake does not handle the new pyproject file I think) • the flake in the alot main repo could be used in CI which would fix some of my trouble in #1617 But • the flake is mostly for the benefit of people using the nix package manager which is not python specific (but can be used in ci) • we can still maintain the flake in a separate repo (like I already do), so that's no problem if @pazz is against it. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.*Message ID: <pazz/alot/pull/ ***@***.***>

lucc · 2023-07-28T10:12:53Z

From my point of view we can merge as soon as @teto has fixed draft state.

teto · 2023-07-28T11:37:44Z

Should I just merge this then?

dont it's not ready yet, I wanted to get your sentiment first but I've seen @lucc active on nixpkgs before so that's a positive point. I will have a look at @lucc's flake. This PR "autogenerates" the nix code from the pyproject.toml to avoid duplicating the code dependencies but it still needs some overrides/tweaks.

lucc · 2023-07-28T11:52:39Z

@teto our CI was defuct for quite some time and it might be that some tests are broken. You might have to disable some test cases explicitly. The alot derivation in nixpkgs does it, my flake also does it.

teto · 2023-07-30T17:29:50Z

@lucc so it should be ok to test. Let me know if nix run github:teto/alot/flake wokrs for you.

We can improve the flake afterwards. I will try to finish #1603 and will see what is missing then

lucc

Otherwise it looks good to me and I can run it with nix.

flake.nix

lucc · 2023-08-08T14:05:16Z

flake.nix

+        };
+
+        devShells.default = pkgs.mkShell {
+          packages = [ poetry2nix.packages.${system}.poetry ];


How is this supposed to be used? I tried

nix develop github:teto/alot/flake poetry run alot

which results in an import error (configobj not found). But when I run

nix develop github:teto/alot/flake poetry install

it fails to install the gpg and the notmuch2 bindings because it can not find the gpgme and the notmuch development packages. Should these be added here?

lucc · 2023-08-19T13:46:32Z

@teto do we even need a dedicated devShell? When I run nix develop .\#packages.x86_64-linux.default I have a python with all dependencies (I can run python -m alot in that shell). So how about we get rid of the devShell stuff and merge this?

Once merged, running `nix run 'github:pazz/alot'` will be able to run a master version of alot. This assumes you have the nix package manager available with flakes enabled.

teto · 2023-08-19T14:17:50Z

Agreed, let's merge we can refine later.

lucc

I think we can merge this

pazz · 2023-08-29T09:12:46Z

I should have reported this here earlier:

I just tried to install via poetry on a new system (debian testing) and it fails with gpg related errors.
Did we overlook any dependencies?

poetry install --no-root

Installing dependencies from lock file    

Package operations: 1 install, 0 updates, 0 removals

  • Installing gpg (1.10.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Could not find gpgme-config.  Please install the libgpgme development package.
  

  at ~/.venv/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with gpg (1.10.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "gpg (==1.10.0)"'.

pip wheel --use-pep517 "gpg (==1.10.0)"
Collecting gpg==1.10.0
  Using cached gpg-1.10.0.tar.gz (39 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      Could not find gpgme-config.  Please install the libgpgme development package.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm on debian 12 and successfully installed the dependencies indicated in this commit using

sudo apt get install python3-setuptools python3-magic python3-configobj python3-notmuch python3-urwid python3-urwidtrees python3-gpg python3-twisted python3-dev swig

There has been some issues/discussions about gpgme-config missing in current versions of libgpgme-dev
and other projects have worked around this.

Is this simply a debian testing issue with gpgme and pip or do we need to take action?

teto · 2023-08-29T09:23:43Z

It's possible I missed libgpgme-dev in the list of dependencies. On nixos it appears in the python package:

nix-locate bin/gpgme-config
libsForQt5.qgpgme.dev                             5,393 x /nix/store/a33nb2h0145dc0yhwih1dzxdik7b574d-gpgme-1.20.0-dev/bin/gpgme-config
python310Packages.gpgme.dev                       5,389 x /nix/store/i98bybk6d6ci3rcmmvhw7pw4mypif57q-gpgme-1.20.0-dev/bin/gpgme-config
python311Packages.gpgme.dev                       5,389 x /nix/store/jjg0kr6z221mqkmm0rz5cg5da9frf88s-gpgme-1.20.0-dev/bin/gpgme-config
gpgme.dev                                         5,390 x /nix/store/h7x6m11chlxbd5i9jd4saxkv1jbn4yf8-gpgme-1.20.0-dev/bin/gpgme-config

maybe apt has a python3-gpgme on top of python3-gpg ?

pazz · 2023-08-29T14:03:28Z

I have installed all those packages where available, but think this binary simply has been removed from debian (and Arch?): pspdev/pspdev#44 Quoting Matthieu Coudron (2023-08-29 10:23:54)

…

It's possible I missed libgpgme-dev in the list of dependencies. On nixos it appears in the python package: nix-locate bin/gpgme-config libsForQt5.qgpgme.dev 5,393 x /nix/store/a33nb2h0145dc0yhwih1dzxdik7b574d-gpgme-1.20.0-dev/bin/gpgme-config python310Packages.gpgme.dev 5,389 x /nix/store/i98bybk6d6ci3rcmmvhw7pw4mypif57q-gpgme-1.20.0-dev/bin/gpgme-config python311Packages.gpgme.dev 5,389 x /nix/store/jjg0kr6z221mqkmm0rz5cg5da9frf88s-gpgme-1.20.0-dev/bin/gpgme-config gpgme.dev 5,390 x /nix/store/h7x6m11chlxbd5i9jd4saxkv1jbn4yf8-gpgme-1.20.0-dev/bin/gpgme-config maybe apt has a python3-gpgme on top of python3-gpg ? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.*Message ID: <pazz/alot/pull/ ***@***.***>

lucc · 2023-08-29T16:39:18Z

@pazz I am confused: is your comment about the nix flake or about the poetry integration which is from #1621?

pazz · 2023-08-30T07:49:59Z

@lucc I am confused as to where the difference is tbh.
Fact is, I cannot install the development version (used to be ./setup.py develop) using poetry nor pip.

I am also a little confused as to where we declare dependencies now. I hope not three times?
I know this complaint comes late, apologies.

By the way, If I change pyproject.toml like so, then I can install via pip install -e .

diff --git a/pyproject.toml b/pyproject.toml
index 52feecf6..d7cfddb3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -29,7 +29,7 @@ urwidtrees = ">=1.0.3"
 twisted = ">=18.4.0"
 python-magic = "*"
 configobj = ">=4.7.0"
-gpg = "*"
+gnupg = "*"
 
 [tool.poetry.dev-dependencies]
 pycodestyle = "*"

On debian testing, there should be no need to install gpgme wia pip or poetry because the documented dependencies (in alot's install docs) include python3-gpg (nowadays python-gpg) which comes with the python bindings to gpg as used in alot.

lucc · 2023-08-30T13:52:37Z

The core difference between poetry and nix to my understanding is:

scope:
- poetry is a package manager for python packages
- nix is a general package manager
files:
- poetry uses pyproject.toml and poetry.lock
- nix uses flake.nix and flake.lock

I think the fuzzyness comes from the fact that we use the poetry integration for nix, i.e we let nix read the poetry.lock file in order to read and build the dependencies of alot. (If you check flake.lock -- its quite short -- you can see that it does not list any dependencies of alot)

So where do we list dependencies:

we list our python deps in pyproject.toml/poetry.lock in a machine readable format
we list our non python deps in flake.nix in a format readable by the nix package manager
we (apparently) list all deps in the user manual for debian/ubuntu/fedora users

~~The patch for pyproject.toml looks like it should also fix this override in flake.nix: https://github.com/pazz/alot/blob/67ca49a/flake.nix#L32~~

pazz · 2023-09-01T08:39:18Z

Thank you, this clears up a lot for me.
Still, the current status is kind of broken on debian, not due to us:

alot uses the gpg module, which are the python bindings for direct calls to libgpg.
This needs to be compiled during the installation via pip install gpg, which fails on debian testing and derivates because updated libgpg development packages come withou the gpgme-config tool.

I can install alot via pip install -e . or poetry install --no-root if I change the dependency as in my patch above. But then it fails to run due to a missing import og gpg.

poetry run alot
Warning: 'alot' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.

The support to run uninstalled scripts will be removed in a future release.

Run `poetry install` to resolve and get rid of this message.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/patrick/projects/alot/alot/__main__.py", line 10, in <module>
    from alot.settings.const import settings
  File "/home/patrick/projects/alot/alot/settings/const.py", line 4, in <module>
    from .manager import SettingsManager
  File "/home/patrick/projects/alot/alot/settings/manager.py", line 17, in <module>
    from ..utils import configobj as checks
  File "/home/patrick/projects/alot/alot/utils/configobj.py", line 14, in <module>
    from .. import crypto
  File "/home/patrick/projects/alot/alot/crypto.py", line 5, in <module>
    import gpg
ModuleNotFoundError: No module named 'gpg'

Not surprising.

Now, I do have a working gpg python module installed via my package manager. Somehow this system-wide package is not picked up by pip nor poetry though..

As side-node: the line in the flake.lock file you linked to looks as if we already do some mocking around the gpg module.
And it looks to me that maybe it'd be a good idea to use the gpgme module instead of gpg because that's actively being maintained ?

lucc · 2023-09-01T09:06:19Z

To my understanding poetry does everything in a venv so it will not pick up your system python stuff.

About my comment: I was mixing up the strings gpg, gnupg and gpgme so not sure if that comment is any good.

lucc · 2023-09-01T16:04:03Z

I can reproduce the breakage on debian with this docker file:

FROM debian
RUN apt -yq update && apt install -yq python3-poetry libnotmuch-dev libmagic1
# used by the gnupg python package?
RUN apt install -yq libgpgme-dev
# not picked up by poetry?
RUN apt install -yq python3-gpg
# fails with "Could not find gpgme-config.  Please install the libgpgme development package."
RUN poetry install

What is the relation between the different gpg modules for python?
I thought that https://pypi.org/project/gpg/ was from the upstream people from https://www.gnupg.org, at least that's what they link as their homepage.

teto · 2023-09-02T23:44:29Z

#1624 (comment) couldn't be any better it's perfect.

Somehow this system-wide package is not picked up by pip nor poetry though..

You can disable the venv created by poetry python-poetry/poetry#110 (comment) but that feels wrong.

I had tested the poetry script in an ubuntu container, not debian. Do we need gpgme-config for email signature validation ? How is it used ?

lucc · 2023-09-03T10:02:28Z

Do we need gpgme-config for email signature validation ? How is it used ?

gpgme-config is used by the python gpg package during installation. The error message stems from poetry -> pip -> gpg package.

The gpg package is used for all things crypo in alot.

teto mentioned this pull request Jul 29, 2023

Use Github Actions #1616

Closed

teto force-pushed the flake branch from 7b051ba to d6e7259 Compare July 30, 2023 17:21

teto marked this pull request as ready for review July 30, 2023 17:28

lucc requested changes Aug 8, 2023

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

lucc reviewed Aug 8, 2023

View reviewed changes

build: adding flake

67ca49a

Once merged, running `nix run 'github:pazz/alot'` will be able to run a master version of alot. This assumes you have the nix package manager available with flakes enabled.

teto force-pushed the flake branch from d00f450 to 67ca49a Compare August 19, 2023 14:03

lucc approved these changes Aug 19, 2023

View reviewed changes

lucc merged commit 50fcd10 into pazz:master Aug 19, 2023

teto deleted the flake branch August 19, 2023 20:59

chenrui333 mentioned this pull request Aug 26, 2024

alot 0.11 Homebrew/homebrew-core#182573

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add a nix flake #1624

add a nix flake #1624

teto commented Jul 27, 2023

lucc commented Jul 28, 2023

pazz commented Jul 28, 2023 via email

lucc commented Jul 28, 2023

teto commented Jul 28, 2023

lucc commented Jul 28, 2023

teto commented Jul 30, 2023

lucc left a comment

lucc Aug 8, 2023

lucc commented Aug 19, 2023

teto commented Aug 19, 2023

lucc left a comment

pazz commented Aug 29, 2023

teto commented Aug 29, 2023

pazz commented Aug 29, 2023 via email

lucc commented Aug 29, 2023

pazz commented Aug 30, 2023 •

edited

Loading

lucc commented Aug 30, 2023 •

edited

Loading

pazz commented Sep 1, 2023

lucc commented Sep 1, 2023

lucc commented Sep 1, 2023

teto commented Sep 2, 2023

lucc commented Sep 3, 2023

add a nix flake #1624

add a nix flake #1624

Conversation

teto commented Jul 27, 2023

lucc commented Jul 28, 2023

pazz commented Jul 28, 2023 via email

lucc commented Jul 28, 2023

teto commented Jul 28, 2023

lucc commented Jul 28, 2023

teto commented Jul 30, 2023

lucc left a comment

Choose a reason for hiding this comment

lucc Aug 8, 2023

Choose a reason for hiding this comment

lucc commented Aug 19, 2023

teto commented Aug 19, 2023

lucc left a comment

Choose a reason for hiding this comment

pazz commented Aug 29, 2023

teto commented Aug 29, 2023

pazz commented Aug 29, 2023 via email

lucc commented Aug 29, 2023

pazz commented Aug 30, 2023 • edited Loading

lucc commented Aug 30, 2023 • edited Loading

pazz commented Sep 1, 2023

lucc commented Sep 1, 2023

lucc commented Sep 1, 2023

teto commented Sep 2, 2023

lucc commented Sep 3, 2023

pazz commented Aug 30, 2023 •

edited

Loading

lucc commented Aug 30, 2023 •

edited

Loading