regression: we now show a spinner at the bottom of every room for ~30s after launching the app

[ara4n]

No description provided.

[dbkr]

I looked into this a bit: my suspicion is that it's waiting for an additional /keys/changes request that we're doing because the device list sync token is older than the sync token in the stored sync request. In practice this request should do nothing, but we still need it because we can't be sure the device list store saved correctly. If this is indeed what is causing this, we'll have to think about how to fix it safely.

[dbkr]

Update: having got Matthew to repro, it looks like my hypothesis is broadly correct (plus we have a shedload of groups requests going on at the same time for the Tag Panel):

https://matrix.to/#/!WCHwNGGsrVwPeRtAnq:matrix.org/$15174950301118850fPpTb:matrix.org

so the spinner-at-bottom-of-page looks to be blocked on a bunch of groups which took 22s :|
(having just reloaded)
and also a keys/changes req that took 48s
each group req took ~20s, of which there were about 20

summary / rooms / invited_users per each group i'm in

[lukebarnard1]

shedload of groups requests going on at the same time for the Tag Panel

These will be a red herring - the requests don't effect the state of the sync, which ultimately is controlling visibility of the spinner here.

[ara4n]

the spinner now shows for about 2 minutes.

[turt2live]

I usually see up to 10-15 minutes to be honest.

[dbkr]

So, this is Riot actually having a newer token in the device list store than in the sync store, because we write the device list immediately, whereas we only write the sync store every 5 minutes. We're also writing it unnecessarily often, so the first thing that will help this is matrix-org/matrix-js-sdk#623

[dbkr]

...there is actually more to this. Having debugged this over Matthew's shoulder, this is now the first incremental sync request getting queued behind all the groups requests which number in the tens and take a couple of seconds each. Easiest fix for this is probably to limit the number of concurrent groups requests.

[ara4n]

just dug into this with dave; looks like what's happening is that the groups stuff spins up a whole flood of GET requests to get the users, invited_users, summary and rooms for the groups you're in. This means about 20 concurrent HTTP hits start, and for whatever reason these requests take ages (to the extent that they often 504). This in turn stalls the first initial sync.

I am bemused and sad as to why the groups API makes us poll for all this rather than shoving it down /sync like everything else.

[ara4n]

see also matrix-org/synapse#2966

[lukebarnard1]

Mitigated by matrix-org/matrix-js-sdk#629