Where to put credentials for CI?

[davidcorrigan714]

What is your question?

Seeing a few recent Issues around this issue and a few things in work, figure I'd just start an issue to figure out the current state of things and concretely express our need since it seems there's still some ambiguity of what users need. In this case I'm one of the maintainers of our internal build pipeline processes & private registries (Artifactory) and don't use Conan at all myself. Our users want to use Conan in their projects so in our CI pipelines we really need a specific, configurable, spot on the filesystem to put credentials for our private registries for build processes using Conan for each build. That's in line with most other package managers out there - pip, docker, helm, & npm all support that in some fashion. They all have a credentials file that defaults to a user scoped directory of some kind, and an environment variable to override that location. Generally what we're doing is we have common tooling across our organization that installs appropriate credentials to the appropriate environments for any specific build running on our CI platform, just need to know where to put them for a given package manager. We also may not have access to the package manager binary itself yet at this stage since providing that is the responsibility of build steps further down the chain and separate from our steps that furnish credentials, so running a conan CLI command to reset them every run would be less-than ideal too.

Seems the existing way for Conan is to pass credentials in is with environment variables. We strongly discourage putting any credentials in environment variables - we see those get picked and stored up by logging, debugging, and profiling tools way more often than files on disk. Same with the CLI '-p' option which is also seems Conan supports, command lines are ridiculously leaky and love to be captured by build log files.

I'm seeing docs about a source_credentials.json file. Looks like that's in the "Conan cache"(?) Need some docs or links on that page explaining what that is and if it's configurable. Seems like it's currently in the conan home directory? Then I'm seeing the recent "Feature/credentials json" PR. Sorta looks like that's revising source_credentials.json ?

Reading through #7793 is scaring me too. Conan should absolutely be using credentials specified by the user, loaded on every invocation. The user shouldn't need to be aware that it's caching things, or that apparently some Conan servers use JWT tokens. Tokens should be opaque strings for all practical purposes to appropriately support other private registry implementations, which practically has to be done to support users that need to run private registries. Worst case maybe an environment variable to ignore cache could be added?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @davidcorrigan714

Thanks for your questions

I'm seeing docs about a source_credentials.json file. Looks like that's in the "Conan cache"(?) Need some docs or links on that page explaining what that is and if it's configurable. Seems like it's currently in the conan home directory? Then I'm seeing the recent #14392 PR. Sorta looks like that's revising source_credentials.json ?

It is not revising the source_credentials.json, because that is for external non-Conan repos, like download sources origins, regular download URLs. The PR is to create a new credentials.json that will store the credentials for Conan server repos. This is what aligns with other package managers.

Reading through #7793 is scaring me too. Conan should absolutely be using credentials specified by the user, loaded on every invocation. The user shouldn't need to be aware that it's caching things, or that apparently some Conan servers use JWT tokens. Tokens should be opaque strings for all practical purposes to appropriately support other private registry implementations, which practically has to be done to support users that need to run private registries. Worst case maybe an environment variable to ignore cache could be added?

Conan always uses JWT with all communication with all servers, not only with some servers. It is built-in in the protocol. The core idea was that Conan didn't have to store the user passwords on its infrastructure, but the password is transient, never stored by Conan, and instead an expirable (potentially short-lived) JWT token is used for all communications with the server. That allows to limit the live of the password definition in CI to just the first authentication, and can be dropped later. This limits a lot the concerns that the password might be leaked somewhere else, as the only possibility is the single moment the password exists, like conan remote login command. Up to my knowledge, using JWT is not uncommon, and for example Docker implements it (https://docs.docker.com/registry/spec/auth/jwt/)

our need since it seems there's still some ambiguity of what users need.

Regarding https://github.com/conan-io/conan/pull/14392/files#r1278897465, that should be mostly a small UX thing. The credentials.json is already a jinja template file. It can include/import other files, so if you have some other file in a known location in the system, it can be imported and injected in the current one. Not saying that we will not consider adding a conf to allow locating the file in another place, but it shouldn't be a blocker as there are workarounds, and we typically wait for more user feedback until adding this kind of configurability.

Recall that the main "configuration" of Conan is via conan config install that can install remotes, profiles, custom settings, deployers, custom commands, profile plugin, command wrappers, etc, together with credentials.json file. So it is true that it is typically not system-wide configuration, but cache-wide configuration. The idea is that many different projects use different remotes, different profiles, etc. Accordingly, they can also limit their credentials to the repos they use.

[davidcorrigan714]

Ah so I mis-understood what source_credentials.json is for and that PR is really implementing more of what I'm looking for.

Starting to get the picture here, the login command hits some login API, gets a JWT, caches that and uses it probably until a 401 or 403 come back. NuGet, and I believe Git too, support similar concepts but have a pluggable framework for obtaining the tokens. Without that, the scheme is effectively ignoring a lot of security implications in furnishing that initial secret. Doesn't really matter if Conan has a nice ephemeral token stashed away when we still have to supply a long-lived token to get it and don't have secure mechanisms to supply that to the login command for every build scenario. Sounds like credentials.json is supposed to help fix that gap though.

So conceptually at the start of a build we could redefine CONAN_HOME and put credentials.json there. We'll wipe the credentials folder after each build which should also remove Conan's stashed JWT in that case. Alternatively we could generate a credentials.json and have Conan users install that provided file in their build steps, but then our common tooling may not always clear Conan's cached jwt.

[memsharded]

Starting to get the picture here, the login command hits some login API, gets a JWT, caches that and uses it probably until a 401 or 403 come back.

Yes, correct

NuGet, and I believe Git too, support similar concepts but have a pluggable framework for obtaining the tokens.

I have looked for it, but couldn't find clear information, if you have some links that explain a bit about these frameworks, that would help.

So conceptually at the start of a build we could redefine CONAN_HOME and put credentials.json there. We'll wipe the credentials folder after each build which should also remove Conan's stashed JWT in that case. Alternatively we could generate a credentials.json and have Conan users install that provided file in their build steps, but then our common tooling may not always clear Conan's cached jwt.

It is typically different when talking about CI agents doing automated build and developers. In case of CI it is usual to work with blank caches, do the job, and remove things as necessary. But for developers it is not practical to clean things at every build/command. But also their permissions and tokens will be read-only (devs shouldn't have write/delete permissions on server repos). In that case, setting the JWT tokens expiry time to something reasonable (like 30 mins-1h or in that order) should take care of auto-invalidation of tokens. Conan also has some extra second obfuscation layer (based on an env-var CONAN_LOGIN_ENCRYPTION_KEY, that does a Vignere cypher over the JWT), but probably that 2nd factor is only necessary in unusual scenarios like open services like ConanCenter, but not for the typical enterprise case with VPNs, etc.

Ah so I mis-understood what source_credentials.json is for and that PR is really implementing more of what I'm looking for.

Yes, at the end of the day, there are 3 mechanisms that can be used to provide creds to a tool like Conan: env-vars, interactively via CLI and files. Conan already provided the first 2, and the last PR is adding support for the third one, file based.

[davidcorrigan714]

I have looked for it, but couldn't find clear information, if you have some links that explain a bit about these frameworks, that would help.

Here's the NuGet docs on authentication providers. The general gist of it is that at the beginning of a dotnet restore operation it queries all the authentication plugins. They can then obtain, store and track them as they needed for various login situations (OAuth, WIA, Env Vars, ect). Now that you have me thinking about it, I'm not sure what happens if it hits a 401 or 403 after that initial call to get a token while actually searching for or downloading packages. The really annoying part about NuGets is there's at least 3 different clients that seem to all look for the authentication plugins in different ways.

Git has some good docs and summary of their process.

Generally all these credential providers are trying to hook back into a corporate or platform identity system that supports something like OAuth which ties a identity back to a central authentication platform like AzureAD, Okta, keycloak, GH Actions, Azure Pipelines etc. Human logins tend favor some kind of OAuth flow for corporate environments. For CI/CD stuff it's really a mix of files on disk, environment variables, and bespoke processes from what I've seen. I just keep finding environment variables captured in build logs & build profiling tools so we're really trying to ditch that, and move to shorter lived access tokens once our platforms support those features.

[memsharded]

Thanks for the feedback, now I understand better what you meant, I recall now the Git credentials helpers.
I think it is doable to provide something like Conan credentials plugin. Conan already implements many other plugins (command_wrapper, compatibility, profile, custom commands, deployers), so this would be aligned with the Conan design.

However I would way a bit until the credentials.json file is out there and we get more usage feedback, then revisit this possibility. Depending on the usage, we might want to hook in the credentials.json file via jinja2 template to provide the user credentials helpers, or bypass completely the credentials.json, but we will see. I am tagging this as a 2.X roadmap feature.

[memsharded]

Hi @davidcorrigan714

The credentials.json file was added in https://docs.conan.io/2/changelog.html#aug-2023, in Conan 2.0.10 in PR #14392, but for some reason this was not correctly closed back then.

Docs in https://docs.conan.io/2/reference/config_files/credentials.html

I think we can close this ticket as implemented. Please check it if you haven't yet, and if there are any issues please open new tickets. Thanks very much for your feedback again!