Composer: Updates require access to private repositories?

[mpdude]

Basic info:

Package ecosystem
Composer/PHP

Package manager version
Locally: 2.0.10

Language version
PHP 7.2.24-0ubuntu0.18.04.7

Manifest location and content prior to update
`/composer.{json,lock}

Updated dependency
Native package manager behavior
Images of the diff or a link to the PR, issue or logs
Due to the usage of a private registry, at this point I am unable to provide those 😥. If really necessary I can see if/how I could provide (partially) redacted ones.

Previous behavior in Dependabot Preview:

I am using a private Composer registry (run with Satis) protected with HTTP Auth. Dependabot Preview was configured to do lockfile updates only and could do so without problems.

FWIW, "Dependabot Preview" was an "Installed GitHub App" at https://github.com/organizations/my-org/settings/installations, and was configured to have access to all of my org's (private) repositories.

Current behavior in GitHub-native Dependabot:

I migrated by means of the automatically provided migration PR (awesome work 💪🏻 , by the way!).

However, updates fail with an error message like

Dependabot can't access my-org/my-repo
You can fix the issue by granting Dependabot access, which will enable any repository in @my-org to get automatic updates from my-org/my-repo. Be sure that you want to share my-org/my-repo with everyone in your organization.

Here are the lines from the update log that mention the package in question:

  proxy | 2021/05/03 11:45:37 [241] GET https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
  proxy | 2021/05/03 11:45:37 [241] * authenticating git server request (host: github.com)
  proxy | 2021/05/03 11:45:37 [241] 404 https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
  proxy | 2021/05/03 11:45:37 [241] * auth'd git request returned 404, retrying without auth
  proxy | 2021/05/03 11:45:37 [241] * de-auth'd request returned 401, replacing response
  proxy | 2021/05/03 11:45:37 [243] GET https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
  proxy | 2021/05/03 11:45:37 [243] 401 https://github.com:443/my-org/my-repo/info/refs?service=git-upload-pack
updater | INFO <job_116983183> Handled error whilst updating some/other-package: git_dependencies_not_reachable {:"dependency-urls"=>["ssh://git@github.com/my-org/my-repo"]}

I can see that access to the private registry was basically configured correctly, as I can see that the user account dedicated for "native" Dependabot successfully fetches the packages.json at the top level URL of the private registry. However, it never makes attempts to fetch any other URLs. That seems odd, since the contents of packages.json look like

{
    "packages": [],
    "includes": {
        "include/all$ddb6c5de69f9dbe693aac27bcab7eba3a38b2597.json": {
            "sha1": "ddb6c5de69f9dbe693aac27bcab7eba3a38b2597"
        }
    }
}

... and so I'd consider additional requests necessary.

Update: It seems there are requests to the other URLs, but only occasionally. Maybe there is some caching in place?

Issue/questions:

My issues/questions are, in order:

1. Why is it necessary in the first place for Dependabot to access the repositories themselves? To my knowledge, Composer registries contain all relevant metadata: A complete list of available package versions, the dependencies each of them declares etc. Also, you can run composer update --no-install to run only dependency updates without installing any packages at all; which seems to support my hypothesis that the dependency resolution step needs the package metadata only.
2. Will I really have to grant Dependabot access to every single repository through the organisation settings? In my case, that's about a 100 repos, and I do not really want to go through all of them manually... also, it would not be efficient to have failing Dependabot workflows, grant access and repeat... and repeat...
3. If really necessary, is there an API available I could use to script this process?

[jurre]

Thanks for the kind words and clear description of the problem!

I think that you'll need to allow Dependabot access to that private git repository that is referenced in the logs you shared.

Could you try following these docs and see if that resolves the issue for you?

Dependabot-preview automatically had access to a few more things than the native version does, and I think this should resolve it.

To answer your questions:

Why is it necessary in the first place for Dependabot to access the repositories themselves? To my knowledge, Composer registries contain all relevant metadata: A complete list of available package versions, the dependencies each of them declares etc. Also, you can run composer update --no-install to run only dependency updates without installing any packages at all; which seems to support my hypothesis that the dependency resolution step needs the package metadata only.

Yes I think ultimately there should be a way to do this without requiring a full download, but at the moment we've not been able to tackle it.

Will I really have to grant Dependabot access to every single repository through the organisation settings? In my case, that's about a 100 repos, and I do not really want to go through all of them manually... also, it would not be efficient to have failing Dependabot workflows, grant access and repeat... and repeat...

You can do this via the UI in the docs mentioned, I agree it's a bit cumbersome for that many repo's :( but it's a bit better than going through the "grant access" notices.

If really necessary, is there an API available I could use to script this process?

Currently there isn't, but it's come up before. We may look into this, but I can't offer a timeline of when this would be available/

@asciimike I think we may want to mention this in the migration docs, wdyt?

[asciimike]

The migration docs have a callout for private repos, which links to the private repo setup instructions. Happy to clarify them if they're not clear enough.

As for the API, totally agree that it's a pain to get a large number over, and we're thinking of some solutions to make it better, as @jurre has said.

[mpdude]

If I could grant access to all private repos with a single switch, that would probably be fine for now. Possible to expect before August?

[mpdude]

I missed that part of the migration docs before, but it’s completely clear.

[asciimike]

@mpdude unsure about it: I think we're looking at providing either a button to migrate all or an API at some point, as it seems to be blocking a lot of larger customers.

[mpdude]

I did the manual migration of 96 repos today and noticed something that might be self-evident for you if you know the internals at GitHub, but it was a bit surprising to me:

Granting the private repo access to Dependabot is not only necessary so it can perform its internal package update resolutions (which, as I wrote initially, should not even need repo access for Composer packages).

It will also enable workflows triggered by Dependabot to install (i.e. git clone) such private repos. That might be good to know in the context of #3253, because it removes the need to have SSH deploy keys or similar for private dependencies.

[driesvints]

So, if I have this correct from reading this issue: it's required to grant Dependabot access to private repositories which are hosted through Satis?

This is unfeasible for a few reasons but the main one is that you don't always own the private repos. For example, in the case of Laravel Nova the repository/composer package is offered through Satis and you get an API token which you can use as a password for Satis in order to download it. I'd expect it to be able to authenticate with the username/password flow of the composer-repository type registry but Dependabot doesn't allows for that it seems: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates#composer-repository

Is there really no way to allow Dependabot to access a Satis registry other than explicitly adding it to the repo in question?

[mpdude]

My initial observation was incomplete and in fact, Dependabot from time to time also accesses the detailed package data in the Satis repository. It seems to try, however, to also download the packages themselves to get additional data.

@driesvints In your case, when the Laravel Nova composer registry is accessible with HTTP Basic Auth, I think Dependabot should be able to read the package metadata from there.

But do you know how access to the packages themselves is secured? What URLs are provided by the package repository? How are these downloads authenticated?

[driesvints]

I'm trying again on a repo with Laravel Nova. Here's the config file I'm using: https://github.com/fullstackbelgium/fullstackbelgium.be/blob/1918171c6762053f3e462b03248a0d6278b43f8f/.github/dependabot.yml

I've also tried various variations for the url setting, (including the "http-basic" url from the Nova docs for CI environments](https://nova.laravel.com/docs/3.0/installation.html#authenticating-nova-in-continuous-integration-ci-environments).

When I run Dependabot PR creation I see the following:

updater | INFO <job_137268865> Handled error whilst updating laravel/framework: private_source_authentication_failure {:source=>"nova.laravel.com"}
updater | INFO <job_137268865> Finished job processing

I can also see that Dependabot sucessfully retrieved the version listing from Satis (hashes redacted):

  proxy | 2021/05/23 16:16:01 [014] GET https://nova.laravel.com:443/packages.json
  proxy | 2021/05/23 16:16:01 [014] * authenticating composer registry request (host: nova.laravel.com)
  proxy | 2021/05/23 16:16:01 [014] 200 https://nova.laravel.com:443/packages.json
  proxy | 2021/05/23 16:16:01 [016] GET https://packagist.org:443/p/laravel/framework.json
  proxy | 2021/05/23 16:16:01 [016] 200 https://packagist.org:443/p/laravel/framework.json
updater | INFO <job_137268865> Latest version is 8.42.1
  proxy | 2021/05/23 16:16:02 [018] GET https://nova.laravel.com:443/packages.json
  proxy | 2021/05/23 16:16:02 [018] * authenticating composer registry request (host: nova.laravel.com)
  proxy | 2021/05/23 16:16:02 [018] 200 https://nova.laravel.com:443/packages.json
  proxy | 2021/05/23 16:16:02 [020] GET https://nova.laravel.com:443/include/all%<redacted>.json
  proxy | 2021/05/23 16:16:02 [020] * authenticating composer registry request (host: nova.laravel.com)
  proxy | 2021/05/23 16:16:02 [020] 200 https://nova.laravel.com:443/include/all%<redacted>.json

But do you know how access to the packages themselves is secured? What URLs are provided by the package repository? How are these downloads authenticated?

I've asked this to my colleague who works on Nova. Hoping to get back to you soon. Thanks for helping out @mpdude!

In the meantime I can see that the manifest returns the following for a version:

    "v3.9.4": {
        "name": "laravel/nova",
        "version": "v3.9.4",
        "version_normalized": "3.9.4.0",
        "source": {
          "type": "git",
          "url": "git@github.com:laravel/nova.git",
          "reference": "<redacted>"
        },
        "dist": {
          "type": "zip",
          "url": "https://nova.laravel.com/dist/laravel/nova/laravel-nova-<redacted>-zip-348430.zip",
          "reference": "<redacted>",
          "shasum": "<redacted>"
        },

I see a 403 page when I visit that url.

[crynobone]

I see a 403 page when I visit that URL.

Accessing the page would also require HTTP Basic Auth, but that should be handled by Composer and I guess Dependabot will use it as well when fetching the artifact.

[driesvints]

@crynobone any idea then what I'm doing wrong?

[crynobone]

proxy | 2021/05/23 16:16:10 [225] GET https://nova.laravel.com:443/dist/laravel/nova/laravel-nova-<redacted>.zip
proxy | 2021/05/23 16:16:10 [225] * authenticating composer registry request (host: nova.laravel.com)

proxy | 2021/05/23 16:16:10 [225] 403 https://nova.laravel.com:443/dist/laravel/nova/laravel-nova-<redacted>.zip

proxy | 2021/05/23 16:16:10 [233] GET https://github.com:443/laravel/nova.git/info/refs?service=git-upload-pack
proxy | 2021/05/23 16:16:10 [233] * authenticating git server request (host: github.com)
proxy | 2021/05/23 16:16:10 [233] 404 https://github.com:443/laravel/nova.git/info/refs?service=git-upload-pack
proxy | 2021/05/23 16:16:10 [233] * auth'd git request returned 404, retrying without auth
proxy | 2021/05/23 16:16:10 [233] * de-auth'd request returned 401, replacing response
proxy | 2021/05/23 16:16:10 [235] GET https://github.com:443/laravel/nova.git/info/refs?service=git-upload-pack
proxy | 2021/05/23 16:16:10 [235] 401 https://github.com:443/laravel/nova.git/info/refs?service=git-upload-pack

It seem dependabot get the correct dist file but then failed to download it (403 authentication error) and then fallback to GitHub repository where it doesn't have access to.

[driesvints]

/cc @asciimike You asked me recently to take that private composer repo issue to the issue tracker here. We found this similar issue and did some digging. There's more info above ^

[jeffwidman]

Is this still an active issue?

As best I can tell from reading through this, the original issue was about migrating from Dependabot-preview to native Dependabot. That has long since been resolved.

But now there's a follow-on comments about a different issue, which is private auth struggles with PHP. And I'm unclear if that's still an active problem or now resolved?

This is part of why open source maintainers are always saying please file a new issue rather than trying to tag on comments for a semi-related issue... it's just really hard to keep track of things as a maintainer.

[mpdude]

I cannot tell for sure – I do not longer use Dependabot for regular updates, but for security-related things only. So I do no longer expect it to notice, update or see changes to my internal packages 🤷🏻

[driesvints]

I have to say I've given up on using Dependabot for regular package updates as well :-/

[jeffwidman]

Thanks for letting me know. I'm going to close but if you hit this in the future please open an issue and we can work together to figure it out.