Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multi-user auth and management support. #2082

Merged
merged 66 commits into from
Oct 13, 2024
Merged

Add multi-user auth and management support. #2082

merged 66 commits into from
Oct 13, 2024

Conversation

knadh
Copy link
Owner

@knadh knadh commented Oct 13, 2024

No description provided.

Marc Bärtschi and others added 30 commits October 13, 2024 16:59
This is a simple OIDC implementation. It's very basic and just logs the user in. Access control needs to be done on the IDP side.
knadh added 25 commits October 13, 2024 16:59
- Filter lists by permitted list IDs in DB get calls.
- Split getLists() handlers into two (one, all) for clarity.
- Introduce new `subscribers:get_by_list` permission.
- Tweak UI rendering to work with new per-list permssions.
This commit splits roles into two, user roles and list roles, both of which
are attached separately to a user.

List roles are collection of lists each with read|write permissions, while
user roles now have all permissions except for per-list ones.

This allows for easier management of roles, eliminating the need to clone and
create new roles just to adjust specific list permissions.
This has been a hair-pulling rabbit hole of an issue. #1931 and others.
When the `next-campaign-subscribers` query that fetches $n subscribers
per batch for a campaign returns no results, the manager assumes
that the campaign is done and marks as finished.

Marathon debugging revealed fundamental flaws in qyery's logic that
would incorrectly return 0 rows under certain conditions.
- Based on the "layout" of subscribers for eg: a series of blocklisted
  subscribers between confirmed subscribers.
  A series of unconfirmed subscribers in a batch belonging to a double
  opt-in list.
- Bulk import blocklisting users, but not marking their subscriptions
  as 'unsubscribed'.
- Conditions spread across multiple CTEs resulted in returning an
  arbitrary number of rows and $N per batch as the selected $N rows
  would get filtered out elsewhere, possibly even becoming 0.

After fixing this and testing it on our prod instance that has
15 million subscribers and ~70 million subscriptions in the
`subscriber_lists` table, ended up discovered significant inefficiences
in Postgres query planning. When `subscriber_lists` and campaign list IDs
are joined dynamically (CTE or ANY() or any kind of JOIN that involves)
a query, the Postgres query planner is unable to use the right indexes.

After testing dozens of approaches, discovered that statically passing
the values to join on (hardcoding or passing via parametrized $1 vars),
the query uses the right indexes. The difference is staggering.
For the particular scenario on our large prod DB to pull a batch,
~15 seconds vs. ~50ms, a whopping 300x improvement!

This patch splits `next-campaign-subscribers` into two separate queries,
one which fetches campaign metadata and list_ids, whose values are then
passed statically to the next query to fetch subscribers by batch.

In addition, it fixes and refactors broken filtering and counting logic
in `create-campaign` and `next-campaign` queries.

Closes #1931, #1993, #1986.
@knadh knadh merged commit 6fe47b2 into master Oct 13, 2024
ai-armageddon pushed a commit to Twitter-outreach/ListMonk that referenced this pull request Mar 11, 2025
Add multi-user auth, permissions, and user management.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant