fetchFromGitLab: support for private repositories

[panicgh]

Description of changes

This is an attempt to add support for private GitLab repositories (via password/token login), similar to fetchFromGitHub. I tried to maintain a similar structure as for GitHub, but some details in GitLab API seem to work differently: the .netrc file can only be used for the fetchgit approach; a custom header to curl needs to be used for fetchzip instead (API doc)

Let me know if it would be better to rebase on staging instead of master.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[panicgh]

I successfully tested the fetcher with a private GitLab instance and an access token for the "Reporter" role. If fetchgit is used (e.g. via forceFetchGit=true;), the token needs read_repository or read_api scope; otherwise (i.e. fetchzip), it needs read_api scope.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fetchfromgitlab-support-for-private-repositories/22900/1

[panicgh]

Hmm. I just discovered #43583 and wonder if it still makes sense to continue with this PR.

It makes sense to me to continue with this PR, since it is already implemented and it may still be useful in situations where fetchGit is not suitable.

[panicgh]

• Rebased to master
• Changed sparseCheckout from string to list

Questions:

• Are tests required? How to implement them best? (e.g. private repo with public access token at gitlab.com, ad-hoc test VMs with preconfigured gitlab instance?)
• Is documentation required? (But there is no documentation on the private repo support for fetchFromGitHub either)

[SuperSandro2000]

Are tests required? How to implement them best? (e.g. private repo with public access token at gitlab.com, ad-hoc test VMs with preconfigured gitlab instance?)

We don't have tests for most other more complex fetchers and I think a VM test is overkill.

Is documentation required? (But there is no documentation on the private repo support for fetchFromGitHub either)

It would be nice to have it otherwise other people don't know how to use it.

[SuperSandro2000]

Can we not express this in a netrc file?

[panicgh]

GitLab can use HTTP Basic Authentication and a .netrc file only when used via Git (e.g. forceFetchGit = true).

https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#project-access-tokens

Use a project access token to authenticate:

• With the GitLab API.
• With Git, when using HTTP Basic Authentication, use:
  • Any non-blank value as a username.
  • The project access token as the password.

Project access tokens are similar to group access tokens and personal access tokens.

It needs a custom PRIVATE-TOKEN header for access via the GitLab API (fetchzip/fetchurl):
https://docs.gitlab.com/ee/api/#personalprojectgroup-access-tokens

I added a short clarification to the code and to the commit message.

[panicgh]

If I manage to do in not too long time, I will also try to submit a test based on the GitLab NixOS test.

[panicgh]

Rebased and added an expires_at attribute to the API calls creating new tokens (otherwise the call fails). I was unsuccessful using nix-store --realise as suggested by @roberth (see discussion above).

If you think it makes sense, I could split out the first 3 commits (to add forceFetchGit and sparseCheckout to fetchFromGitLab) into a separate PR, which is less complex and more generally useful than this one.

[panicgh]

Rebased. Omitted the commits already upstreamed in #268304.

[viraptor]

It would be great if you could include an example of usage in practice. I've seen a linked example of nix-daemon environment variables file already, but some sample of non-daemon usage would be useful too. I'm going through a number of ways to try now, so I suspect questions like that will be raised later.

(confirming this works for me BTW)

[fliiiix]

Is there a reason this pr is still in draft?

[centromere]

@SuperSandro2000 Do you know why this hasn't been merged yet?

[panicgh]

Is there a reason this pr is still in draft?

I wanted to rebase it and align with the latest changes from the gitlab NixOS tests, but did not yet have the time to do so. And it seemed I would be the only one interested in this and have found a solution (apply the patch locally) that works for me for the past several years. But if now other people may want to use this, I can do the rebase/potential cleanup etc. next week or so.

[SuperSandro2000]

First of all it is still a draft

[Jappie3]

According to my testing and the comment you left, access to the Gitlab API only requires an access token - not a username and password. Besides that, using an access token is the only available option of the two if you have 2FA enabled. I suggest modifying the if statement like this:

netrcImpureEnvVars = ["${varBase}USERNAME" "${varBase}PASSWORD" "${varBase}TOKEN"];
netrcPhase =
  if useFetchGit
  then
    # GitLab supports HTTP Basic Authentication only when Git is used:
    # https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#project-access-tokens
    ''
      if [ -z "''$${varBase}USERNAME" -o -z "''$${varBase}PASSWORD" ]; then
        echo "Error: Private fetchFromGitLab using Git requires the nix building process (nix-daemon in multi user mode) to have the ${varBase}USERNAME and ${varBase}PASSWORD env vars set." >&2
        exit 1
      fi

      cat > netrc <<EOF
      machine ${domain}
              login ''$${varBase}USERNAME
              password ''$${varBase}PASSWORD
      EOF
    ''
  else
    # Access via the GitLab API requires a custom header and does not work
    # with HTTP Basic Authentication:
    # https://docs.gitlab.com/ee/api/#personalprojectgroup-access-tokens
    ''
      if [ -z "''$${varBase}TOKEN" ]; then
        echo "Error: Private fetchFromGitLab using the GitLab API requires the nix building process (nix-daemon in multi user mode) to have the ${varBase}TOKEN env var set." >&2
        exit 1
      fi

      # needed because fetchurl always sets --netrc-file if a netrcPhase is present
      touch netrc

      cat > private-token <<EOF
      PRIVATE-TOKEN: ''$${varBase}TOKEN
      EOF
      curlOpts="$curlOpts --header @./private-token"
    '';

[panicgh]

I have no strong argument, but would tend to keep username/password for git and API access. Using the same variables makes it easier to for example change for forceFetchGit = true without changing credentials and also makes the fetcher more similar to fetchFromGitHub. Even if the user name is technically unnecessary, maybe it can still be useful for example to print it in logs (I did not verify if this is actually possible).

[viraptor]

But if now other people may want to use this

Definitely still interested here.

[stuart-warren]

There is an upstream issue to support basic auth that could always do with more thumbs up to get it prioritised https://gitlab.com/gitlab-org/gitlab/-/issues/341157

[panicgh]

Rebased. It seems to work fine when I use this on NixOS, however the VM tests currently do not pass and nix-daemon coredumps in one of the VMs... Maybe someone has an idea?

It seems to fail on the first test "fetch-as-public" (which tries to download a private repo without token, which is expected to fail due to the missing of token). This seems to work in principle (curl errors with 404), but then nix-daemon crashes with "FATAL: exception not rethrown".

$ nix build -f . nixosTests.gitlab-fetch-private -L --no-link  --print-out-paths
...
vm-test-run-gitlab-fetch-private> client # trying https://acme.test/api/v4/projects/alice%2Ftest-alice/repository/archive.tar.gz?sha=refs%2Fheads%2Fmaster
vm-test-run-gitlab-fetch-private> client #   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
vm-test-run-gitlab-fetch-private> client #                                  Dload  Upload   Total   Spent    Left  Speed
vm-test-run-gitlab-fetch-private> gitlab # [  115.759861] gitlab-workhorse[1191]: acme.test 127.0.0.1 - - [2025/01/24:23:15:41 +0000] "GET /api/v4/projects/alice%2Ftest-alice/repository/archive.tar.gz?sha=refs%2Fheads%2Fmaster HTTP/1.1" 404 35 "" "curl/8.11.1 Nixpkgs/25.05" 80
vm-test-run-gitlab-fetch-private> client #   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
vm-test-run-gitlab-fetch-private> client # curl: (22) The requested URL returned error: 404
vm-test-run-gitlab-fetch-private> client # error: cannot download source-salted-7y4klz60xpdh from any mirror
vm-test-run-gitlab-fetch-private> client # error: builder for '/nix/store/c85yf1ab6v6m68wyx6q6nry2xjfkkd30-source-salted-7y4klz60xpdh.drv' failed with exit code 1;
vm-test-run-gitlab-fetch-private> client #        last 9 log lines:
vm-test-run-gitlab-fetch-private> client #        > error checking the existence of https://tarballs.nixos.org/sha256/sha256-HqozwmyEGzXUlSL6zgInlycalxROnD2QOSPRwkrw8sQ=:
vm-test-run-gitlab-fetch-private> client #        > curl: (6) Could not resolve host: tarballs.nixos.org
vm-test-run-gitlab-fetch-private> client #        >
vm-test-run-gitlab-fetch-private> client #        > trying https://acme.test/api/v4/projects/alice%2Ftest-alice/repository/archive.tar.gz?sha=refs%2Fheads%2Fmaster
vm-test-run-gitlab-fetch-private> client #        >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
vm-test-run-gitlab-fetch-private> client #        >                                  Dload  Upload   Total   Spent    Left  Speed
vm-test-run-gitlab-fetch-private> client #        >   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
vm-test-run-gitlab-fetch-private> client #        > curl: (22) The requested URL returned error: 404
vm-test-run-gitlab-fetch-private> client #        > error: cannot download source-salted-7y4klz60xpdh from any mirror
vm-test-run-gitlab-fetch-private> client #        For full logs, run 'nix-store -l /nix/store/c85yf1ab6v6m68wyx6q6nry2xjfkkd30-source-salted-7y4klz60xpdh.drv'.
vm-test-run-gitlab-fetch-private> client # [  127.221911] nix-daemon[946]: FATAL: exception not rethrown
vm-test-run-gitlab-fetch-private> client # [  127.237442] systemd-coredump[1621]: Process 946 (nix-daemon) of user 0 terminated abnormally with signal 6/ABRT, processing...