Use dotnet credential provider #8927
Conversation
a36ce0b to
19be332
Compare
|
As I understand it, this will only work for Azure DevOps package feeds, but not private feeds from other providers (e.g., GitHub, AWS, Artifactory, MyGet, etc.) so the NuGet.Config mangling would still be necessary. |
|
Oh, I did not consider it. You are right |
19be332 to
6224025
Compare
|
It does not solve everything around authorization, but it fixes Azure DevOps auth. Can it be reviewed&merged? |
f2ae60e to
de22d37
Compare
39435b6 to
56dccc5
Compare
|
@trejjam I only recently learned about how credentials are used in the update process and my understanding is that this won't fix the issue. When a dependabot job is run, there are 2 separate docker containers that are started, the main one is the NuGet updater (which is this code) and despite there being I'm hesitant to modify the job proxy with a dotnet credential provider, since that job proxy is used by all update types (NPM, Python, etc.), not just .NET. Is there a situation you're seeing where tokens set in the # ...
registries:
azure-devops-feed:
type: nuget-feed
url: https://dev.azure.com/...<I forgot what goes here>.../index.json
username: ${{secrets.MY_FEED_USER_NAME}}
password: ${{secrets.MY_FEED_TOKEN}}
# __OR__
azure-devops-feed-2:
type: nuget-feed
url: https://dev.azure.com/...<I forgot what goes here>.../index.json
token: ${{secrets.MY_FEED_USER_NAME}}:${{secrets.MY_FEED_TOKEN}}
# ...All of that being said, the manual patching of the user-level |
|
tagging @jakecoffman in case he has some context on the proxy |
56dccc5 to
7610a8f
Compare
|
I am hitting the issue running our instance of Dependabot. I do not have knowledge of other runtimes, and honestly, I am not searching for it. The issue, as I stated above, is when we use |
|
If you are not familiar with |
|
I guess I'm not understanding what the intent is with this PR. The code makes no change to @trejjam It sounds like you're running dependabot directly from the Ruby code and not through the cli tool, is this correct? If that's the case, then you could be getting tokens/authentication into these code paths (using the CLI tokens/auth is all stripped out.) So if the issue is just with the environment variable having the correct contents, I have a few comments that I'll add next to the code itself. |
| endpoint_credentials = [] | ||
|
|
||
| nuget_credentials.each do |c| | ||
| next unless c["token"] |
There was a problem hiding this comment.
Azure DevOps feeds aren't necessarily private and authentication isn't always specified via a token field, it could also come in via username/password, so that'll have to be accounted for.
| nuget_credentials.each do |c| | ||
| next unless c["token"] | ||
|
|
||
| exploded_token = T.must(c["token"]).split(":", 2) |
There was a problem hiding this comment.
Related to the above comment, this shouldn't assume a token will be of the form abc:123; it should be able to handle a token without a colon and/or it should handle separate username/password fields.
| password = exploded_token[1] | ||
|
|
||
| endpoint_credentials << <<~NUGET_ENDPOINT_CREDENTIAL | ||
| {"endpoint":"#{c['url']}", "username":"#{username}", "password":"#{password}"} |
There was a problem hiding this comment.
Here and a few lines below, use the .to_json method so we don't have to worry about any escaping. I'm not 100% certain on the syntax, but it's something like this:
endpoint_credentials << {
"endpoint" => c['url'],
"username" => username,
"password" => password,
}Then later on:
env["VSS_NUGET_EXTERNAL_FEED_ENDPOINTS"] = {
"endpointCredentials" => endpoint_credentials
}.to_json|
If you are running the Ruby code directly and not through the CLI, it might be worth reviving PR #9004. |
| def self.get_credentials_to_credential_helper_env(credentials) | ||
| env = {} | ||
|
|
||
| nuget_credentials = credentials.select { |cred| cred["type"] == "nuget_feed" } |
There was a problem hiding this comment.
This feature only works with NuGet feeds from Azure DevOps, right? If that's the case it would be a good idea to filter credentials to only those feeds so you don't accidentally expose feeds/tokens that aren't ADO to the ADO service.
If I recall, all Azure DevOps NuGet feeds look like https://pkgs.dev.azure.com/... or https://SOME-ORGANIZATION.pkgs.visualstudio.com/....
|
@brettfo I am still not able to get private NuGet feeds hosted in Azure DevOps to auth correctly using this setup. Can you let me know if I am missing something obvious?
My dependabot.yml config is: version: 2
registries:
my-private-feed:
type: nuget-feed
url: https://pkgs.dev.azure.com/my-org/_packaging/my-private-feed/nuget/v3/index.json
token: PAT:${{MY_PAT_TOKEN_ENV_VAR}}
# ...I am using https://github.com/tinglesoftware/dependabot-azure-devops to run Dependabot (Dockerised) in a DevOps Pipeline. However, whenever it encounters a package that needs to be updated and it starts building the project, the build fails with "error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/.../index.json". I know very little about the inner working here, but is there some disconnect between the auth provided to the "scanner" and the "updater" services? Is this an issue specific to Azure DevOps feeds? Is there something I'm doing wrong in the config? Any help would be really appreciated, this is the final barrier for me being able to use this in our Azure DevOps environment. |
|
@rhyskoedijk There are 2 things I can think of to try.
Either use user/pass separately: ...
type: nuget-feed
url: https://dev.azure.com/...<I forgot what goes here>.../index.json
username: ""
password: ${{ env.MY_ENV_VAR }}
...Or pre-combine the empty username and password, which really just means you need a leading colon character: ...
type: nuget-feed
url: https://dev.azure.com/...<I forgot what goes here>.../index.json
token: ":${{ env.MY_ENV_VAR }}" # I'm not sure if the leading colon requires surrounding quotes or not
...I've seen successful runs with private NuGet feeds of all types (Azure DevOps, AWS CodeArtifact, JFrog Artifactory, etc.) so there's nothing special about Azure DevOps. I don't know about the |
|
@rhyskoedijk I just thought of something else you could check to verify the token for the ADO package feed. At the very start of the job we try to authenticate against all private NuGet feeds and we log the results of that. Right at the top (commonly within the first 10 or 15 lines) you should see a log line like |
|
@brettfo from what I recall authentication inside the GitHub hosted version works differently. The credentials are not passed in plain text to the job. This is not open for anyone to inspect or copy. When running a self hosted version, this functionality is not present so the credentials may be passed as is. This is the case for the Azure DevOps extension mentioned by @rhyskoedijk that I maintain because Azure DevOps does not have an inbuilt version. Before migration to NuGet/dotnet CLI, this method of passing credentials worked just well enough. Admittedly, the migration needed to have happened to bring support for more scenarios. However, private feeds for azure DevOps seems to be not working as expected and this PR contains the actual solution. @rhyskoedijk offered a workaround at tinglesoftware/dependabot-azure-devops#921 (comment), and it proves as much. By the way, I may be wrong in my understanding of these components. |
|
@mburumaxwell based on #10360 (comment) from @brettfo, this pull request and the wider issue of private feed auth handling via the "updater" component aren't going to be resolved by making changes to dependabot-core. All remaining code related to private feed auth in the updater is likely to be removed in the future. The recommendation is to do something similar to dependabot-cli, where the "updater" container is wrapped inside a "proxy" container that auto-injects the auth to outgoing HTTP requests. |
|
@rhyskoedijk Have you found out anything else on it? The other alternative is to use of the CLI which seems to support Azure DevOps properly except for the additions like auto approve, policy bypass, auto complete, skip PR creation, etc., which I suspect is used by lots of people. |
|
The dependabot CLI starts a proxy container that handles all authentication and that's the way we're moving going forward. Could the behavior this PR is trying to introduce be covered in that scenario? |
|
@brettfo yes, if using Dependabot CLI, this change isn't required. This change is only required if using Something that tripped me up for a while is that there is still a lot of code in dependabot-core that attempts to inject credentials even though the values won't be populated under normal conditions. e.g. I assume this will be removed when time permits to avoid future confusion? |
|
@rhyskoedijk I think you're correct that the authentication stuff you pointed out could likely be removed. I'll have to experiment a bit to see if it might still be required, but in general we're doing the "proper" NuGet thing and honoring |
Using dotnet credential helper, we do not need to juggle with Nuget.config (current solution works only sometimes).
With Credential Provider, we can use it as it is and do not care much about it. It should just work.
When it does not work:
packageSourceMapping<packageSources><clear /></packageSources>As @brettfo pointed out, this fixes these issues only for Azure DevOps repositories. Not other providers