How to get/store users & groups?

[twrichards]

In order to facilitate mentions (both individuals and teams) we are going to be using Google users/groups, but in order to be resilient to Google outages (which is when Pinboard could be invaluable) we've decided to store a list of users and groups (in a DynamoDB table, which we can easily access via GraphQL [AWS AppSync]). There are various options for how we can do this...

• we have a new lambda, which runs on a schedule (e.g. every 30mins) to retrieve users & groups and store to our DB, but there are various options for where to get the data from...
  • permissions tool already retrieves users & groups from Google, but renders with HTML templates (e.g. https://permissions.gutools.co.uk/admin/users), so we could expose a new endpoint which presents the data we need in JSON format, however we would ideally want to authenticate this interaction between the proposed lambda and this new endpoint on permissions via IAM (rather than out over the internet with further API keys etc) - which might be tricky in Play 🤷‍♂️
  • the lambda could speak to google directly (with node lib), either using
    • the same api key/secret as permissions tool (they're in the same account so could share)
    • a new api key/secret specifically for this purpose
• OR, add a write-behind cache to permissions tool which is the dynamo table, i.e. in the process of returning normal responses it would store the users and groups it retrieves to the dynamo DB table that Pinboard would also read from, meaning no new lambda and would mean that if google is down permissions tool itself could read from this shared cache (making it more robust). We might need some timer to ensure it's hit frequently enough for the cache to be useful to Pinboard).
• slight variant on the above, but instead of write behind, Permissions tool has a scheduled task within it that keeps the cache up to date and it serves its responses directly from the cache (probably a speed improvement there)

would really appreciate the views of the @guardian/digital-cms (especially @sihil)

[twrichards]

In lieu of any preferences from the team (but also taking into account what @sihil said offline about potentially overhauling all of our tools authentication/authorisation [pan-domain-auth, login, permissions etc.] potentially moving to Okta) I'm going to go ahead with...

• we have a new lambda, which runs on a schedule (e.g. every 30mins) to retrieve users & groups and store to our DB, but there are various options for where to get the data from...

  • the lambda could speak to google directly (with node lib), either using

    • the same api key/secret as permissions tool (they're in the same account so could share)

... because this is the simplest and most self-contained, so this can unblock the completion of phase 1 of pinboard mentions.

[twrichards]

Implementation in...

• set up database table for users #44 addition of users table
• users-refresher-lambda #46 users-refresher-lambda to keep 'users' table up to date
• Implement 'mentions' (for individuals) #45 'mentions' implementation which makes use of these 'users'