Recommended workflow for switching between multiple API tokens?

[nlhkabu]

Hi folks

I am working on adding instructions to the 'Add API token' page on PyPI (see pypi/warehouse#6615). Currently, the instructions look like this:

API token for an entire PyPI account

We suggest that users create a .pypirc file in their home directory with the token details:

Token scoped to a PyPI project

We suggest that users can switch between multiple tokens with <code>twine --repository PROJECT_NAME</code>

Questions

1. Is this accurate?
2. Is this the recommended workflow?
3. Am I missing anything that it would be useful to document here?

Thanks :)

[sigmavirus24]

That looks accurate. I'd say that it's a hack but it's the first thing I thought of when I saw your question.

It would probably be good to have another way of managing these though instead of forcing folks to use --repository

[nlhkabu]

Ok, thanks @sigmavirus24. I will go ahead and get these instructions merged into Warehouse.

[bhrutledge]

@sigmavirus24 and @nlhkabu is it okay to close this? And/or should we open an issue for improving the UX for using Twine with API tokens?

[bhrutledge]

FYI @sigmavirus24 and @nlhkabu (and @di for good measure):

After trying this out for myself, I think the instructions might not be sufficient. Given this ~/.pypirc:

[pypi]
username = __token__

[testpypi]
username = __token__

[example-pkg]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-...

I get this output:

$ twine upload --repository example-pkg dist/*
InvalidConfiguration: Missing 'example-pkg' section from the configuration file
or not a complete URL in --repository-url.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

It works if add I this to the top of~/.pypirc:

[distutils]
index-servers =
    pypi
    testpypi
    example-pkg

I discovered that twine.utils.get_config only uses values for defined index-servers, which defaults to ["pypi", "testpypi"].

twine/twine/utils.py

Lines 78 to 85 in 330c0a2

	# optional configuration values for individual repositories
	for repository in index_servers:
	for key in [
	"username", "repository", "password",
	"ca_cert", "client_cert",
	]:
	if parser.has_option(repository, key):
	config[repository][key] = parser.get(repository, key)

A quick search didn't uncover any issues on Warehouse, which surprised me. It seems like a quick addition to the relevant Warehouse docs, but seems to add to the case for improving Twine's handling of tokens.

[bhrutledge]

Related: #565 - Not obvious how to use multiple project API tokens with keyring

[di]

While this works, I agree with @sigmavirus24 that it's a bit of a hack. I think it's probably worthwhile for us to think of a new and better way to do this in the .pypirc file (and maybe standardize the format while we're at it?)

Maybe something like this for repo-wide tokens:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

And this for project-specific tokens:

[distutils]
index-servers =
    pypi
    other

[other]
repository = http://example.com/pypi

[pypi:example-pkg]
token = pypi-ABC...

[other:example-pkg]
token = pypi-DEF...

[bhrutledge]

Thanks, @di. I wondered how much freedom we might have with the .pypirc format. I like how this removes the need for repository = https://upload.pypi.org/legacy/ in the config for project-scoped tokens.

FYI, there's an evolving discussion that started at pypa/packaging.python.org#297 (comment), where @jaraco and @takluyver have proposed some ideas on how to make this work with keyring and flit.

[di]

I could be wrong, but I think these extra fields/sections would be ignored by any tool that doesn't know about them, so it should be fully backwards-compatible.

[bhrutledge]

Marked this and #565 as blocked until the discussion started at pypa/packaging.python.org#297 (comment) is resolved.

[sigmavirus24]

The other thing to keep in mind is that the ini format is fairly flexible and projects like pytest and flake8 are able to do things like:

option =
   something: value
   something-else: another-value

So we could keep this super simple and have a:

tokens =
   *: pypi<general token>
   twine: pypi<project-token>
   ...

That keeps us from proliferating sections into the file, keeps the implementation simple, and is farily explicit without trying to hack around things. Then we can select the tokens out of there as necessary without doing "repository" based nonsense. Getting this to work with Keyring will be harder, as the discussion Brian linked indicates

[di]

@sigmavirus24 How would that handle the same project name with different tokens across multiple repos?

[sigmavirus24]

@di A more fully fledged example would be:

[distutils]
index-servers =
    pypi
    other
    another

[pypi]
tokens = 
   *: pypi-ABC...
   twine: pypi-BCD...
   etc: pypi-CDE...

[another]
repository = http://example.com/pypi
tokens = 
   *: pypi-DEF...
   twine: pypi-EFG...
   etc: pypi-FGH...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...
    etc: pypi-HIJ...

[di]

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

[distutils]
index-servers =
    pypi
    other

[pypi]
tokens = 
   *: pypi-ABC...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...

which feels clunky compared to:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

[bhrutledge]

From pypa/packaging.python.org#297, it seems like the ideal scenario is that tokens are not stored in .pypirc, but rather in keyring. Does that influence the choice of format?

[sigmavirus24]

I think you're always going to have people who would prefer not to use keyring. And we'd have to go through a lengthy deprecation period to kill that off which isn't worth the effort frankly.

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

The fact that we don't know what the most common use-case is suggests that we don't know enough to continue pushing forward on making the right choice

[di]

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects;
 count
--------
 103550
(1 row)

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects where projects.count = 1;
 count
-------
 62803
(1 row)

[bhrutledge]

Related (I think): I just put up a draft PR that documents the current spec of .pypirc: pypa/packaging.python.org#734.

Also, we're talking a lot about .pypirc, but I'm curious how this would manifest in configuration via the command line and/or environment variables. For example, does this imply --token and TWINE_TOKEN? It seems friendlier for those to be aligned.

I'm wondering if it might be simpler to keep the current convention, but mitigate the hack by defaulting username to __token__, as suggested in #561.

I think that would also require fewer updates to the docs in Twine, PPUG, and Warehouse.

[sigmavirus24]

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

Is there a way to tell what % of that 60% use tokens at all?

Here's the point: If we go with token= then we're implicitly endorsing 1 global token that everyone should be using and thus we're going to calcify that use-case. Is the goal of project-scoped tokens not to (beyond enabling CI) to also encourage keeping tokens scoped appropriately to projects?

We should be meeting folks where they are today, certainly, but we should also be striving to push the community forward with the gentlest nudges we can think of.

[bhrutledge]

@sigmavirus24 I think @di's original suggestion handles project-scoped tokens via additional sections that inherit from the "base" section:

[pypi]
token = pypi-ABC...

[pypi:example-pkg]
token = pypi-DEF...

Another idea:

[pypi]
token = pypi-ABC...
project_tokens = 
    example-pkg: pypi-DEF...

That said, I'd still like to consider sticking with the current username/password hack, such that Twine's default configuration becomes:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__

which I think leans more towards something like this:

[pypi]
password = pypi-ABC...

[pypi:example-pkg]
password = pypi-DEF...

[bhrutledge]

Building on my last suggestion, combined with my second proposal in #565 and pypa/packaging.python.org#297 (comment), I think this would allow CLI commands like:

$ keyring set pypi:example-pkg __token__
Password for '__token__' in 'pypi:example-pkg':

$ twine upload -r pypi:example-pkg dist/*

Aside: looking over those other issues makes me wonder if this is conversation should be moved to https://discuss.python.org/c/packaging/.

[di]

A few things on the password/token "hack":

1. I'd like us to imagine a world where using password in .pypirc is deprecated or discouraged, and I think that overloading password with the token value would make that hard or impossible.

2. I think that a default username of __token__ might produce some confusing behavior in some edge cases, for example consider the following:

   [pypi]
   usenrame = my_username
   password = my_password

   Twine now has no way to tell the user their username is "missing" because it happily falls back on the default of __token__, which fails authentication, even though the user has included the correct username/password (unless we want to start warning the user every time Twine uses the default, which I think would get annoying for users who legitimately want to use tokens)

3. Using token over password allows the user to explicitly tell us "I expect to use tokens" which allows us to potentially do some preflight checking of this value (i.e. whether it's a valid token). Otherwise, we'd have to implicitly infer this through how the user is using the username field.

4. Having a clear delineation between what is a token and what is a password in configuration will help provide more context to whatever mechanism implements Pluggable credential manager framework proposal [PoC included] #362, which would allow an authentication provider to provide a different value based on what's being requested. E.g.:

   [pypi]
   password_provider = keyring
   
   [other]
   token_provider = some_other_module

   (where we can probably just consider keyring the default password/token provider, but I'm including it here just as an example)

[shon]

I faced the similar issue today

To reproduce

OS: Ubuntu 22.04
Python: 3.10.4
Twine: 4.0.1

$ cat ~/.pypirc

[distutils]
index-servers =
  pypi

[<package-name>]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-<snip>

Twine command

$ twine upload -r <my-package>  dist/*.tar.gz --verbose

INFO     Using configuration from /home/shon/.pypirc                                                                                                           
ERROR    InvalidConfiguration: Missing '<my-package>' section from ~/.pypirc.                                                                                       
         More info: https://packaging.python.org/specifications/pypirc/        

What worked

• Rename [<my-package>] to [pypi]
• twine upload -r <my-package> dist/*.tar.gz
  But then obviously I can't use this for other packages.

[bhrutledge]

@shon Can you try adding <my-package> to index-servers in your .pypirc?

[distutils]
index-servers =
  pypi
  <my-package>

[<my-package>]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-<snip>

I think that will allow you to run:

twine upload --verbose -r <my-package>  dist/*.tar.gz

[shon]

Thanks @bhrutledge that worked!
Wasn't very intuitive to me that my-package should be mentioned index-servers.

[bhrutledge]

@shon Me neither, which led to the discussion starting at #496 (comment).

[kentbull]

Agree with @shon. I like @bhrutledge and @di's comments on a path forward.

[miketheman]

Chiming in briefly to inquire if there's appetite for removing the need for the entire [distutils] section and index-servers keys. (I slammed face-first into this issue today.)

distutils has been removed from Python as of 3.12+, and twine hasn't used distutils directly for over a decade.

Is there still a benefit to preserving usage of this indirection, vs setting sections and looking for those? (Potentially replace configparser.RawConfigParser which is a legacy form of configparser.ConfigParser )