2FA/API tokens: staging/testing rollout

[brainwane]

What's the problem this feature will solve?
To finish #996 (see #5567), we need to test MFA with real users on real packages; asking them to spin up dev environments is too hard and won't help multi-maintainer projects reason well about what MFA policies they want to set up.

Describe the solution you'd like
My tentative suggestion is:

• We first roll out MFA on test.pypi.org and publicize it to project maintainers for a designated 2-week testing period, and we allot extra time for dealing with support requests during that time
• We then roll it out to pypi.org and announce it on the PyPI announcement email list, PSF blog, etc.

Additional context

• If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?
• Are there particular categories of user we need to make sure we get in the beta test? Ideas that come to mind for me: Windows, mobile, users of particular TOTP implementations and U2F implementations, people on very slow connections, people who habitually block a lot of cookies/JS, maintainers who maintain 20+ projects, people with very old and weird PyPI accounts (e.g., we do not have verified email for them, their passwords do not adhere to current policy), users with upwards of 10 MFA methods they want to add to their accounts, multi-maintainer projects, multi-owner projects, organizations where users share an auth token within a group.
• Is test.pypi.org the right place for this? Should we spin up some other instance for this particular kind of testing, since maintainers do use Test PyPI for real uploads (of packages that need testing)?

Cc @ewdurbin .

[di]

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.

It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.

This could allow us to do a timeline like:

• 1 week period on TestPyPI for everyone
• 2 week beta on PyPI for select users
• Open to all users

If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?

Possibly? I think we should be explicit in our "beta" announcement that this might happen, so users aren't surprised.

Are there particular categories of user we need to make sure we get in the beta test?

I think all the groups you listed would be ideal.

people who habitually block a lot of cookies/JS

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency. (cc @woodruffw)

Is test.pypi.org the right place for this?

I think it is. We don't really claim that TestPyPI should be depended on for much.

[ewdurbin]

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.

It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.

This could allow us to do a timeline like:

• 1 week period on TestPyPI for everyone
• 2 week beta on PyPI for select users
• Open to all users

+1 to this approach.

[woodruffw]

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency.

It should be relatively easy to add a fallback here: we can display the otpauth:// URL directly, and the user can copy-and-paste that into their TOTP application or use their QR generator of choice. Right now that URL is only available via the aria-label of the canvas that we render the QR in, but that wouldn't be hard to change at all. (Sidenote: I'm not a frontend developer at all, so I have no idea how to check whether a user has disabled Javascript. Is it just <noscript>?)

This could allow us to do a timeline like:

• 1 week period on TestPyPI for everyone
• 2 week beta on PyPI for select users
• Open to all users

This looks good to me!

[brainwane]

@nlhkabu - does this feel ok to you? If so then I will declare consensus and start planning this, writing sets of "hey come test this" announcement text, etc.

[nlhkabu]

Hi @brainwane - sounds very reasonable to me.

I'd like to run some direct user tests alongside open testing - sessions where I am able to observe users and understand anywhere they get stuck.

I've set up a Google form for registering users for this. @brainwane would you be able to:

1. review the form
2. add a link to it in your announcement

To be clear, I don't think that manual user testing should block the launch of 2FA, unless I find any major technical issues that are not raised by open beta testing.

[di]

@nlhkabu One small comment on the form: might want to ask for their PyPI username as well, so we can enable 2FA for them during the private beta.

[di]

#5567 is close to being done. I'm going to plan on merging it on or before Friday, May 3rd, which would give us the following timeline:

• On or before 5/3 - Merge 2FA (TOTP) support #5567
• 5/3 - enable 2FA across all of TestPyPI, send announcements
• 5/3-5/20 - private beta on PyPI, add users as necessary
• 5/20 - enable for all PyPI users

[brainwane]

OK, draft announcements:

For distutils-sig/discuss.python.org and other longform places:

Subject line: PyPI two-factor auth (2FA) trial May 3-20

Dear PyPI users:

To increase the security of PyPI downloads, we're beginning to introduce two-factor authentication (2FA) as a login security option, and want project maintainers and owners to start testing it.

Starting this Friday, May 3rd, you'll be able to use 2FA on Test PyPI. And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

PyPI currently supports a single 2FA method: generating a code through a Time-based One-time Password (TOTP) application. After you set up 2FA on your PyPI account, then you must provide a TOTP (along with your username and password) to log in. Therefore, to use 2FA on PyPI, you'll need to provision an application (usually a mobile phone app) in order to generate authentication codes; our our testing wiki page gives you suggestions and pointers.

This change only applies to the login step, not package uploads.

More details at our testing wiki page.

During this testing period, if things go awry, there's a chance we will need to wipe tokens from users' accounts, so if you choose to try it, please be forewarned. We suggest you make sure you have a PyPI-verified email address on your user account before trying the feature, to make potential account recovery smoother.

And please let us know if you run into glitches.

We expect to end this testing period on May 20th, then enable the optional 2FA feature for all PyPI users, and move on to working on WebAuthn support.

Thanks to the Open Technology Fund for funding this work. More progress reports at the Packaging Working Group's wiki page.

-the PyPI team

for pypi-announce:

Subject line: PyPI two-factor auth (2FA) trial May 3-20

Dear PyPI users:

To increase the security of PyPI downloads, we're beginning to introduce two-factor authentication (2FA) as a login security option, and want project maintainers and owners to start testing it.

Starting this Friday, May 3rd, you'll be able to use 2FA on Test PyPI. And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

More details at our testing wiki page.

We expect to end this testing period on May 20th, then enable the 2FA feature for all PyPI users, and move on to working on WebAuthn support.

Thanks to the Open Technology Fund for funding this work. More progress reports at our wiki page.

-the PyPI team

[brainwane]

(I need to add a note calling out that this doesn’t affect the upload endpoint currently.)

[nlhkabu]

Looks good to me @brainwane - thank you. You can probably change this sentence to be shorter:

If you set up 2FA on your PyPI account, you must provide a second method of identity verification (other than your username and password) to log in.

[mlissner]

First:

And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

Should this say that it's really more of a user test? It's not just joining a beta, the plan is to sit with these folks for an hour, right? I eagerly clicked the link, then was disappointed since I didn't have the hour.

Second: Is there a plan to require 2FA eventually, or, if there isn't a plan, is it worth threatening that that might happen eventually? I think it makes sense to have 2FA for software repos like PyPi, but I know requiring it would be difficult. Maybe there's a middle ground to start pushing for.

[di]

@mlissner You don't need to be available for that hour to be in the beta, just respond "no". :)

[mlissner]

I'm like 10% sure that changed? Anyway, good!

[brainwane]

When we flipped the switch for this feature on Test PyPI & canon PyPI, we did not initially update the user model so that existing Test PyPI accounts had the feature on. Ernest (I believe) ran some SQL to update all the Test PyPI accounts yesterday and turn on the flag.

Right now, new Test PyPI accounts (as in, created since that time yesterday) do not have the flag set so they don't see 2FA in their account settings.

I'd like @ewdurbin to run the "turn this on for everybody on Test PyPI" SQL again just to get that taken care of, then figure out what to do next.

I think @di told me that turning it on by default for all Test PyPI accounts would also turn it on for all canon PyPI accounts (does this mean we ought to have a better feature flag system in general, or just for this feature?) (and I presume that constraint would also stop us from doing something to fix this like: verifying the email address triggers turning on 2FA support).

[ewdurbin]

Reran the enabling of 2FA on test.pypi.org

[brainwane]

Today's the 20th and it looks like we aren't done with the beta yet, because #5866 is a blocker.

Once we finish that, there are some UI issues that we should fix, but it's ok to roll out 2FA to everyone on pypi.org before fixing them.

And then a note for next time: let's address #5869 (or having "flip bit in user model" in the runbook for the WebAuthN rollout).

[brainwane]

@ewdurbin @dstufft @di do we have any way of knowing how many users on PyPI have turned on 2fa? Is that even a useful statistic?

I naively assume that it would be, and that we would want the number to go up over time, and that we would want a rough estimation in particular of how many users have it turned on who own or maintain at least one package.

(I am ok with the answer being "it is a pain to check that" in which case I will probably ask that we check it, like, nowish and then again in a couple months, as we check how well our staging/rollout/publicity are going. I am of course also ok with the answer being "we cannot know that" or "it is a vulnerability to say this publicly" or what have you.)

[ewdurbin]

warehouse=> SELECT ur.role_name, 
warehouse->        Count(*) 
warehouse-> FROM   users AS u 
warehouse->        JOIN (SELECT DISTINCT user_id, 
warehouse(>                              role_name 
warehouse(>              FROM   roles) AS ur 
warehouse->          ON u.id = ur.user_id 
warehouse-> WHERE  u.totp_secret IS NOT NULL 
warehouse-> GROUP  BY ur.role_name; 
 role_name  | count 
------------+-------
 Maintainer |    62
 Owner      |   214
(2 rows)

[ewdurbin]

Update as of this time:

warehouse=> SELECT ur.role_name, 
warehouse->        Count(*)    
warehouse-> FROM   users AS u  
warehouse->        JOIN (SELECT DISTINCT user_id, 
warehouse(>                              role_name   
warehouse(>              FROM   roles) AS ur 
warehouse->          ON u.id = ur.user_id  
warehouse-> WHERE  u.totp_secret IS NOT NULL 
warehouse-> GROUP  BY ur.role_name; 
 role_name  | count 
------------+-------
 Maintainer |    88
 Owner      |   326
(2 rows)

[brainwane]

@nlhkabu For the WebAuthn rollout, @ewdurbin suggested that we initially add a badge in the Account Settings marking the WebAuthn 2FA method as "beta". I've thus filed #5976 just now.

[ewdurbin]

Update as of today:

 role_name  | count 
------------+-------
 Maintainer |   105
 Owner      |   446
(2 rows)

[brainwane]

Per our conversation in the Friday meeting we plan to roll out the WebAuthn feature on test.pypi.org and on pypi.org with the "beta" badge (#5976) but without doing a bunch of messy bit-flipping. I've also updated https://wiki.python.org/psf/WarehousePackageMaintainerTesting so it'll be ready to publicize.

Working on copy for blog post(s), mailing list posts, social media, and so on.

[brainwane]

OK, today we're rolling WebAuthn out as beta and asking a few people to test it, and in the afternoon we'll post blog posts and I'll notify distutils-sig et alia. Later this week I'll publicize it to some more communities, and then in maybe 10 days we'll remove the "beta" badge and I'll email pypi-announce and python-announce. Once that is done I figure we can close the issue.

[brainwane]

Percentage of Logins (to pypi.org via browser, I think) using Two-Factor Authentication:

May: 2.25%
June: 3.08%
July 1-21: 4.62%
July 21-25: 9.72% (July 25th was our post to Discourse about API tokens)
July 25-28: 15.45%

And, for logins in the past 2 days, it's 23.1%. That is mostly a weekend, so it'll be interesting to see whether the trend changes on weekdays...

[ewdurbin]

We have updated the token username and prefix in #6342.

username: @token => __token__
password/token: pypi:<base64 token body> => pypi-<base64 token body>

These changes should alleviate the need for escaping heroics.

The previous format will continue to work for now, but users will be notified to update their configurations to match the new syntax before the beta period is over.

[brainwane]

Contractors on the OTF-funded work need to stop/deprioritize work on the security features in order to ensure we complete the accessibility and internationalization work by the end of the month. Therefore, even though some security features are still in beta, I'm closing the milestone.

[brainwane]

I'm writing a discuss.python.org/distutils-sig post now to update our community on the end of the OTF-funded work. I'd like for us to have a few more beta blockers closed before we send out a pypi-announce email, to reduce how much time we and other volunteers spend on support issues; it's ok with me if that means the email doesn't go out for a few more weeks.

[ewdurbin]

Update for today:

• how many users have turned on 2FA at all?

warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count 
-------
  3544
(1 row)

• how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP

warehouse=> select count(*) from users where totp_secret is not null;
 count 
-------
  3398
(1 row)

WebAuthn

warehouse=> select count(distinct user_id) from user_security_keys ;
 count 
-------
   336
(1 row)

• how many API tokens currently exist?

warehouse=> select count(*) from macaroons ;
 count 
-------
  1127
(1 row)

[brainwane]

Almost ready for API tokens to leave beta! Just waiting to land pypa/packaging.python.org#687 .

@ewdurbin may I ask for an updated count, on how many users have turned on 2FA at all, and how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

[brainwane]

My current draft of an announcement email:

---

Subject: Start using 2FA and API tokens on PyPI

Dear PyPI users:

To increase the security of PyPI downloads, we have added two-factor authentication (2FA) as a login security option, and API tokens for uploading packages.

If you maintain or own a project on the Python Package Index pypi.org , you should start using these features. Click "help" on PyPI for instructions.

Details and plans for the future:

2FA: PyPI's implementation of the WebAuthn standard means you can use any 2FA device that meets the FIDO standard. 2FA only affects logging in via a web browser, and not (yet) package uploads.

API tokens: use these (instead of username and password) to authenticate when uploading packages to PyPI. You can make tokens that work for all your uploads. You can also make tokens whose scope is limited to one specific package. That way, if a token is compromised, you can just revoke and recreate that token, instead of having to change your password in lots of automated processes.

For more details and instructions, click "help" on PyPI: https://pypi.org/help/ . (These features are also available on Test PyPI.)

In the future, PyPI will set and enforce a policy requiring users with two-factor authentication enabled to use API tokens to upload (rather than just their password, without a second factor). We do not yet know when we will make this policy change.

Thanks to the Open Technology Fund for funding this work.
More work is in progress on pip and PyPI -- see https://wiki.python.org/psf/PackagingWG .

-Sumana Harihareswara on behalf of the PyPI team

[ewdurbin]

Update as of this moment:

• how many users have turned on 2FA at all?

warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count 
-------
  5362
(1 row)

• how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP

warehouse=> select count(*) from users where totp_secret is not null;
 count 
-------
  5090
(1 row)

WebAuthn

warehouse=> select count(distinct user_id) from user_security_keys ;
 count 
-------
   571
(1 row)

• how many API tokens currently exist?

warehouse=> select count(*) from macaroons ;
 count 
-------
  3232
(1 row)

[brainwane]

I've made the launch announcement on pypi-announce and thus I now declare this issue closed. Thank you, everybody.