With token, 403 Invalid or non-existent authentication information

[jaraco]

In pypa/setuptools#1817, I reported an issue where the latest version of setuptools failed to release. Setuptools uses a proven mechanical release process, but in this latest release, I switched to using token-based authentication. This wasn't my first time using token-based auth. I'd previously cut a release of configparser using a (all projects) token. I used the same token and code to inject that token into the .travis config:

def gen_pypi_token():
	pw = $(keyring get "PyPI upload" jaraco).rstrip()
	pw = 'TWINE_PASSWORD="' + pw + '"'
	repo_url = $(git remote get-url --push origin).rstrip()
	repo = repo_url.replace('https://github.com/', '')
	crypt_pw = $(echo @(pw) | travis encrypt -r @(repo)).rstrip()
	jaraco.clipboard.copy(crypt_pw.strip('"'))

aliases['gen-pypi-token'] = gen_pypi_token

Yet, when it ran, the job produced this error:

HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://upload.pypi.org/legacy/

Reverting the change and going back to password-based releases worked around the issue.

It's not yet obvious to me where the error lies. Benoit suggested maybe it's because Setuptools uses the default PyPI deployer and twine 1.13 but configparser uses the latest. It's also possible that I fat-fingered something in the process (though unlikely as most of the process is mechanized).

I'm filing this ticket here to capture the issue and solicit any advice and to capture any solution once identified. This issue is specifically about token-based authentication failures on Travis uploads.

[jaraco]

Does the /legacy/ endpoint support token-based auth? What about version 1.13? Reading the twine issues, I find pypa/twine#482, which indicates that @token is the right username, although I think I've read elsewhere that __token__ is the right form (and that's what worked for configparser).

This is the kind of issue that I'd be happy to dive in and find a solution for, but I feel uneasy with the circumstances. How can I test the production release process without creating a production release? Perhaps I should consider cutting 41.1.0.postN releases for the purpose of testing. Advice here would be welcomed.

[jaraco]

Is there anything on the backend of warehouse that you could correlate with that timestamp (2019-08-13T09:57:04, time zone unknown but probably UTC or UTC-0400)?

edit: corrected to Aug 13, not Aug 3.

[woodruffw]

@jaraco The original deployment of API tokens used @token as the username, but as of #6342 and #6345 the current username is __token__ and the current token prefix is pypi-.

Can you confirm that you're using the latest username and prefix, while still seeing the error?

[jaraco]

Yes. I confirmed the token is prefixed with pypi- and I’m using the dunderscore token username.

[jaraco]

I've created a test build that is likely to replicate the issue.

[jaraco]

In this build, I was able to make a release using the token and bypassing DPL (I subsequently deleted the release from PyPI). So the issue lies with using DPL or something that DPL does. I did make other changes, such as building with pep517, but I suspect that's not pertinent.

[samuelcolvin]

I've just had a similar issue uploading to pypi from travis. I've getting

HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://upload.pypi.org/legacy/

However the upload to pypi succeeded! (I'm not sure if this is the same behaviour @jaraco saw?)

All files seem to have been uploaded and the release seems to have worked fine.

I got this twice, the first time with my password, I thought it might be related to the new token workflow, so I switched to a token and got the same result.

Links:

• Package: https://github.com/samuelcolvin/aiohttp-toolbox
• first release log: https://travis-ci.com/samuelcolvin/aiohttp-toolbox/builds/123885202
• first release: https://pypi.org/project/aiohttp-toolbox/0.5/
• second release log: https://travis-ci.com/samuelcolvin/aiohttp-toolbox/builds/123888117
• second release: https://pypi.org/project/aiohttp-toolbox/0.5.1/

[woodruffw]

Thanks for the report @samuelcolvin! Can you confirm that the token you're feeding to Travis includes the pypi- prefix? We currently reject non-prefixed tokens, so I'm wondering if that could be the root cause 🙂

[jaraco]

However the upload to pypi succeeded!

Interesting. That’s very different from the behavior I saw. In my failures, no artifacts were uploaded. My build would attempt to upload both wheel and sdist artifacts. Not sure if that’s relevant.

[jaraco]

Okay. I now think I understand the crux of my problem. It's this line:

pw = 'TWINE_PASSWORD="' + pw + '"'

In the non-DPL upload technique, the most straightforward way I've found to pass the credentials to twine is through environment variables, but because Travis expects variables to include both the key and the value, it's not possible to encrypt just the password (and travis is uninterested in supporting this use case); one is forced to encrypt the TWINE_PASSWORD= as part of the payload.

But when I tried to apply this routine to generate the encrypted form of the token for the setuptools project, which uses DPL, it expects the password/token only, and I didn't account for that in the setuptools project when changing the credential. I'm 90% certain that if I omit that prefix from the token that it will work.

That doesn't explain the issue samuelcolvin experienced, so they may wish to file a separate ticket for that.

I'm closing this ticket and will re-open if my explanation above doesn't pan out.

[jaraco]

I've revised my gen-pypi-token alias thus:

def switch_env(pw):
	"""
	Does the .travis.yml need the environment variable treatment?
	If so, wrap the password in the environment variable.

	https://github.com/pypa/warehouse/issues/6422#issuecomment-523355317
	"""
	uses_env = 'password:' not in pathlib.Path('.travis.yml').read_text()
	if not uses_env:
		print("Using simple password (without env)")
		return pw
	return 'TWINE_PASSWORD="' + pw + '"'

def gen_pypi_token():
	pw = switch_env($(keyring get "PyPI upload" jaraco).rstrip())
	repo_url = $(git remote get-url --push origin).rstrip()
	repo = repo_url.replace('https://github.com/', '')
	crypt_pw = $(echo @(pw) | travis encrypt -r @(repo)).rstrip()
	jaraco.clipboard.copy(crypt_pw.strip('"'))

[jaraco]

I've created the 41.2 release using this new technique.

[samuelcolvin]

@woodruffw yes, the token definitely included the prefix: looking back at my bash history I see travis encrypt -r samuelcolvin/aiohttp-toolbox pypi-***.

Remember also, I got the exact same failure with the v0.5 release where I was still using my standard password, as you can see from the history of .travis.yml, the deploy setup hasn't changed this year. I've made 20 or so release of the package this year, none of which failed until v0.5.

It looks like the release used setuptools-41.1.0, pip-19.2.2, wheel-0.33.6 and twine-1.13.0 if that helps.

[samuelcolvin]

my issue seems to have been fixed by setuptools-v41.2.

[samuelcolvin]

sorry, ignore that, the error seems the same: https://travis-ci.com/samuelcolvin/aiohttp-toolbox/builds/124041776