From 94aa65da04a1258c63f6efb9725490657747f517 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 6 Jan 2019 01:10:26 +0000 Subject: [PATCH 1/4] MSC1776: a proposal for implementing peeking via /sync --- proposals/1776-peeking-via-sync.md | 136 +++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 proposals/1776-peeking-via-sync.md diff --git a/proposals/1776-peeking-via-sync.md b/proposals/1776-peeking-via-sync.md new file mode 100644 index 00000000000..648c275b629 --- /dev/null +++ b/proposals/1776-peeking-via-sync.md @@ -0,0 +1,136 @@ +# Proposal for replacing peeking in the CS API + +## Problem + +Peeking into rooms without joining them currently relies on the deprecated v1 +/initialSync and /events APIs. + +This poses the following issues: + + * Servers and clients must implement two separate sets of event-syncing logic, + doubling complexity. + * Peeking a room involves setting a stream of long-lived /events requests going. + Having multiple streams is racey, competes for resources with the /sync stream, + and doesn't scale given each room requires a new /events stream. + +## Solution + +We extend /sync to support peeking into rooms on demand. + +This can be done as part of extending /sync to also support paginating results +per-room, which is desirable for reducing the size of initial /syncs and making +them more granular, in order to speed up login. + +Proposal: we add a POST form of /sync, which lets the user add/remove additional +filters on the fly in order to aid peeking into specific sets of rooms with +specific filters. Filtering on rooms which you are not joined to is considered +a request to peek. We also let it support paginating results by grouping them +into batches of rooms. + +**Conceptually** this could look like something like: +```json +POST /sync +{ + "filters": [ + "normal sync filter (limit results to 10 rooms at a time)", + "peek just m.flair on this set of rooms", // to discover flair for users + "peek full room that we're previewing", + "peek full room a profile-room we're looking at", + "peek just m.room.name, m.room.avatar, m.subgroup on this set of group-rooms", // to display groups-as-rooms + ], + "pagination": { + "room_limit": 10 // return a maximum of 10 rooms at a time + } +} +``` + +The `pagination` field would support two fields: + * `room_limit` (which limits the number of rooms returned in a given /sync + response) parameter + * `room_order` (a constant that defines the ordering of the rooms within a /sync + response. only "m.origin_ts" would be defined). + +To avoid the hassle of manually maintaining filters, the act of specifying a +filter could automatically memoize it on the server so that it can be +efficiently referred to in future. (The resulting filter ID could be returned +in the /sync response). + +So, **concretely**, this would look something like: + +```json +POST /sync +{ + "filters": [ + { + "room": { + "state": { + "lazy_load_members": true + }, + "timeline": { + "limit": 20 + }, + } + }, + { + "room": { + "rooms": [ + "!pr0file1:example.com", + "!pr0file2:example.com", + "!pr0file3:example.com", + ], + "state": { + "types": [ + "m.flair" + ] + } + } + }, + ], + "paginate": { + "room_limit": 10 + } +} +``` + +...to request a sync which gives us both the normal timeline, as well as peek +changes in flair events for 3 users we're stalking because they're in the timeline +currently displayed on screen. + +In case of receiving large sync responses (initial or catchup sync), we request +them to be batched with no more than 10 rooms per response. + +The sync response would include: + +```json +{ + "filter_ids": [123, 124] +} +``` + +...to give the user the IDs of the filter objects which were implicitly created +from the POST, so that they can be reused in subsequent invocations of /sync +directly. Alternatively, the filter IDs could be encoded into the sync token so +that subsequent `GET /sync` did the right thing without needing to restate the +filters. + +## Hybrid sync + +This pattern also enables "hybrid sync" (proposed during the June 2018 +brainstorm with Manu/Erik/Matthew and others when considering how to speed up +initial sync), in that a slow large granular incremental sync could be +interrupted and later resumed in order to prioritise loading specific rooms as +the user navigates the app. + +## Security considerations + +Need to worry about clients DoSing by requesting overly unpleasant filters? + +## Issues + +How do we represent peek failures? + +## History + +This takes some inspiration from Erik's original 'paginated sync' PR to Synapse +from 2016: https://github.com/matrix-org/synapse/pull/893/files + From 96d30fdbcd888e6fb12ca2917e2237ee52d411a3 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 6 Jan 2019 01:16:01 +0000 Subject: [PATCH 2/4] tweaks --- proposals/1776-peeking-via-sync.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/proposals/1776-peeking-via-sync.md b/proposals/1776-peeking-via-sync.md index 648c275b629..6e987a91132 100644 --- a/proposals/1776-peeking-via-sync.md +++ b/proposals/1776-peeking-via-sync.md @@ -1,4 +1,4 @@ -# Proposal for replacing peeking in the CS API +# Proposal for implementing peeking via /sync in the CS API ## Problem @@ -33,13 +33,13 @@ POST /sync { "filters": [ "normal sync filter (limit results to 10 rooms at a time)", - "peek just m.flair on this set of rooms", // to discover flair for users + "peek just m.flair on this set of rooms (to discover flair for users)" "peek full room that we're previewing", "peek full room a profile-room we're looking at", - "peek just m.room.name, m.room.avatar, m.subgroup on this set of group-rooms", // to display groups-as-rooms + "peek just m.room.{name,avatar}, m.subgroup on this set of rooms (to display groups-as-rooms)", ], "pagination": { - "room_limit": 10 // return a maximum of 10 rooms at a time + "room_limit": 10 } } ``` @@ -129,8 +129,11 @@ Need to worry about clients DoSing by requesting overly unpleasant filters? How do we represent peek failures? +## Dependencies + +This unblocks MSC1769 (profiles as rooms) and MSC1772 (groups as rooms) + ## History This takes some inspiration from Erik's original 'paginated sync' PR to Synapse from 2016: https://github.com/matrix-org/synapse/pull/893/files - From 731f0b356c8ea027ceb96b9529bb040001997b57 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 6 Jan 2019 01:22:05 +0000 Subject: [PATCH 3/4] tradeoffs --- proposals/1776-peeking-via-sync.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/proposals/1776-peeking-via-sync.md b/proposals/1776-peeking-via-sync.md index 6e987a91132..763a5461a9e 100644 --- a/proposals/1776-peeking-via-sync.md +++ b/proposals/1776-peeking-via-sync.md @@ -1,4 +1,4 @@ -# Proposal for implementing peeking via /sync in the CS API +# MSC1776: Proposal for implementing peeking via /sync in the CS API ## Problem @@ -121,6 +121,11 @@ initial sync), in that a slow large granular incremental sync could be interrupted and later resumed in order to prioritise loading specific rooms as the user navigates the app. +## Tradeoffs + +The API shape here is picked quite naively as it seems to do the job. There are +probably many other ways to skin the cat. + ## Security considerations Need to worry about clients DoSing by requesting overly unpleasant filters? From dce4931a5035e7f4603da7ad54210cf9773d7bbc Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 6 Jan 2019 01:58:41 +0000 Subject: [PATCH 4/4] dep --- proposals/1776-peeking-via-sync.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/1776-peeking-via-sync.md b/proposals/1776-peeking-via-sync.md index 763a5461a9e..4febd6cbfd0 100644 --- a/proposals/1776-peeking-via-sync.md +++ b/proposals/1776-peeking-via-sync.md @@ -137,6 +137,7 @@ How do we represent peek failures? ## Dependencies This unblocks MSC1769 (profiles as rooms) and MSC1772 (groups as rooms) +This depends on MSC1777 (peeking over federation) to work well in practice. ## History