Merge remote-tracking branch 'origin/develop' into rav/sw1v-hotfixes

matrix-org · Aug 3, 2021 · c8a4565 · c8a4565
2 parents ae75ced + 42225aa
commit c8a4565
Show file tree

Hide file tree

Showing 50 changed files with 424 additions and 88 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,61 @@
+Synapse 1.40.0rc1 (2021-08-03)
+==============================
+
+Features
+--------
+
+- Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`. ([\#9918](https://github.com/matrix-org/synapse/issues/9918))
+- Update support for [MSC2716 - Incrementally importing history into existing rooms](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245), [\#10432](https://github.com/matrix-org/synapse/issues/10432), [\#10463](https://github.com/matrix-org/synapse/issues/10463))
+- Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](https://github.com/matrix-org/synapse/issues/10254), [\#10447](https://github.com/matrix-org/synapse/issues/10447), [\#10489](https://github.com/matrix-org/synapse/issues/10489))
+- Initial support for [MSC3244](https://github.com/matrix-org/matrix-doc/pull/3244), Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
+- Add a buffered logging handler which periodically flushes itself. ([\#10407](https://github.com/matrix-org/synapse/issues/10407), [\#10515](https://github.com/matrix-org/synapse/issues/10515))
+- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](https://github.com/matrix-org/synapse/issues/10411))
+- Support for [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-doc/pull/2285). Contributed by @SimonBrandner. ([\#10413](https://github.com/matrix-org/synapse/issues/10413))
+- Email notifications now state whether an invitation is to a room or a space. ([\#10426](https://github.com/matrix-org/synapse/issues/10426))
+- Allow setting transaction limit for database connections. ([\#10440](https://github.com/matrix-org/synapse/issues/10440), [\#10511](https://github.com/matrix-org/synapse/issues/10511))
+- Add `creation_ts` to "list users" admin API. ([\#10448](https://github.com/matrix-org/synapse/issues/10448))
+
+
+Bugfixes
+--------
+
+- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](https://github.com/matrix-org/synapse/issues/10410))
+- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\#10439](https://github.com/matrix-org/synapse/issues/10439))
+- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](https://github.com/matrix-org/synapse/issues/10455))
+- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](https://github.com/matrix-org/synapse/issues/10499))
+
+
+Improved Documentation
+----------------------
+
+- Fix hierarchy of providers on the OpenID page. ([\#10445](https://github.com/matrix-org/synapse/issues/10445))
+- Consolidate development documentation to `docs/development/`. ([\#10453](https://github.com/matrix-org/synapse/issues/10453))
+- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\#10464](https://github.com/matrix-org/synapse/issues/10464))
+- Document how to use Complement while developing a new Synapse feature. ([\#10483](https://github.com/matrix-org/synapse/issues/10483))
+
+
+Internal Changes
+----------------
+
+- Prune inbound federation queues for a room if they get too large. ([\#10390](https://github.com/matrix-org/synapse/issues/10390))
+- Add type hints to `synapse.federation.transport.client` module. ([\#10408](https://github.com/matrix-org/synapse/issues/10408))
+- Remove shebang line from module files. ([\#10415](https://github.com/matrix-org/synapse/issues/10415))
+- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](https://github.com/matrix-org/synapse/issues/10429))
+- Use a docker image cache for the prerequisites for the debian package build. ([\#10431](https://github.com/matrix-org/synapse/issues/10431))
+- Improve servlet type hints. ([\#10437](https://github.com/matrix-org/synapse/issues/10437), [\#10438](https://github.com/matrix-org/synapse/issues/10438))
+- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](https://github.com/matrix-org/synapse/issues/10442))
+- Update the `tests-done` Github Actions status. ([\#10444](https://github.com/matrix-org/synapse/issues/10444), [\#10512](https://github.com/matrix-org/synapse/issues/10512))
+- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](https://github.com/matrix-org/synapse/issues/10446), [\#10450](https://github.com/matrix-org/synapse/issues/10450))
+- Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](https://github.com/matrix-org/synapse/issues/10451))
+- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](https://github.com/matrix-org/synapse/issues/10468))
+- Additional type hints in the state handler. ([\#10482](https://github.com/matrix-org/synapse/issues/10482))
+- Update syntax used to run complement tests. ([\#10488](https://github.com/matrix-org/synapse/issues/10488))
+- Fix up type annotations to work with Twisted 21.7. ([\#10490](https://github.com/matrix-org/synapse/issues/10490))
+- Improve type annotations for `ObservableDeferred`. ([\#10491](https://github.com/matrix-org/synapse/issues/10491))
+- Extend release script to also tag and create GitHub releases. ([\#10496](https://github.com/matrix-org/synapse/issues/10496))
+- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\#10500](https://github.com/matrix-org/synapse/issues/10500))
+
+
 Synapse 1.39.0 (2021-07-29)
 ===========================
 

diff --git a/changelog.d/10245.feature b/changelog.d/10245.feature
diff --git a/changelog.d/10254.feature b/changelog.d/10254.feature
diff --git a/changelog.d/10283.feature b/changelog.d/10283.feature
diff --git a/changelog.d/10390.misc b/changelog.d/10390.misc
diff --git a/changelog.d/10407.feature b/changelog.d/10407.feature
diff --git a/changelog.d/10408.misc b/changelog.d/10408.misc
diff --git a/changelog.d/10410.bugfix b/changelog.d/10410.bugfix
diff --git a/changelog.d/10411.feature b/changelog.d/10411.feature
diff --git a/changelog.d/10413.feature b/changelog.d/10413.feature
diff --git a/changelog.d/10415.misc b/changelog.d/10415.misc
diff --git a/changelog.d/10426.feature b/changelog.d/10426.feature
diff --git a/changelog.d/10429.misc b/changelog.d/10429.misc
diff --git a/changelog.d/10431.misc b/changelog.d/10431.misc
diff --git a/changelog.d/10432.misc b/changelog.d/10432.misc
diff --git a/changelog.d/10437.misc b/changelog.d/10437.misc
diff --git a/changelog.d/10438.misc b/changelog.d/10438.misc
diff --git a/changelog.d/10439.bugfix b/changelog.d/10439.bugfix
diff --git a/changelog.d/10440.feature b/changelog.d/10440.feature
diff --git a/changelog.d/10442.misc b/changelog.d/10442.misc
diff --git a/changelog.d/10444.misc b/changelog.d/10444.misc
diff --git a/changelog.d/10445.doc b/changelog.d/10445.doc
diff --git a/changelog.d/10446.misc b/changelog.d/10446.misc
diff --git a/changelog.d/10447.feature b/changelog.d/10447.feature
diff --git a/changelog.d/10448.feature b/changelog.d/10448.feature
diff --git a/changelog.d/10450.misc b/changelog.d/10450.misc
diff --git a/changelog.d/10451.misc b/changelog.d/10451.misc
diff --git a/changelog.d/10453.doc b/changelog.d/10453.doc
diff --git a/changelog.d/10455.bugfix b/changelog.d/10455.bugfix
diff --git a/changelog.d/10463.misc b/changelog.d/10463.misc
diff --git a/changelog.d/10468.misc b/changelog.d/10468.misc
diff --git a/changelog.d/10482.misc b/changelog.d/10482.misc
diff --git a/changelog.d/10483.doc b/changelog.d/10483.doc
diff --git a/changelog.d/10488.misc b/changelog.d/10488.misc
diff --git a/changelog.d/10489.feature b/changelog.d/10489.feature
diff --git a/changelog.d/10490.misc b/changelog.d/10490.misc
diff --git a/changelog.d/10491.misc b/changelog.d/10491.misc
diff --git a/changelog.d/10499.bugfix b/changelog.d/10499.bugfix
diff --git a/changelog.d/10500.misc b/changelog.d/10500.misc
diff --git a/changelog.d/10511.feature b/changelog.d/10511.feature
diff --git a/changelog.d/10512.misc b/changelog.d/10512.misc
diff --git a/changelog.d/9918.feature b/changelog.d/9918.feature
diff --git a/debian/changelog b/debian/changelog
@@ -1,8 +1,12 @@
-matrix-synapse-py3 (1.39.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.40.0~rc1) stable; urgency=medium
 
+  [ Richard van der Hoff ]
   * Drop backwards-compatibility code that was required to support Ubuntu Xenial.
 
- -- Richard van der Hoff <richard@matrix.org>  Tue, 20 Jul 2021 00:10:03 +0100
+  [ Synapse Packaging team ]
+  * New synapse release 1.40.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 03 Aug 2021 11:31:49 +0100
 
 matrix-synapse-py3 (1.39.0) stable; urgency=medium
 

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -79,6 +79,7 @@
     - [Single Sign-On]()
       - [SAML](development/saml.md)
       - [CAS](development/cas.md)
+    - [Room DAG concepts](development/room-dag-concepts.md)
     - [State Resolution]()
       - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
     - [Media Repository](media_repository.md)

diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md
@@ -0,0 +1,79 @@
+# Room DAG concepts
+
+## Edges
+
+The word "edge" comes from graph theory lingo. An edge is just a connection
+between two events. In Synapse, we connect events by specifying their
+`prev_events`. A subsequent event points back at a previous event.
+
+```
+A (oldest) <---- B <---- C (most recent)
+```
+
+
+## Depth and stream ordering
+
+Events are normally sorted by `(topological_ordering, stream_ordering)` where
+`topological_ordering` is just `depth`. In other words, we first sort by `depth`
+and then tie-break based on `stream_ordering`. `depth` is incremented as new
+messages are added to the DAG. Normally, `stream_ordering` is an auto
+incrementing integer, but backfilled events start with `stream_ordering=-1` and decrement.
+
+---
+
+ - `/sync` returns things in the order they arrive at the server (`stream_ordering`).
+ - `/messages` (and `/backfill` in the federation API) return them in the order determined by the event graph `(topological_ordering, stream_ordering)`.
+
+The general idea is that, if you're following a room in real-time (i.e.
+`/sync`), you probably want to see the messages as they arrive at your server,
+rather than skipping any that arrived late; whereas if you're looking at a
+historical section of timeline (i.e. `/messages`), you want to see the best
+representation of the state of the room as others were seeing it at the time.
+
+
+## Forward extremity
+
+Most-recent-in-time events in the DAG which are not referenced by any other events' `prev_events` yet.
+
+The forward extremities of a room are used as the `prev_events` when the next event is sent.
+
+
+## Backwards extremity
+
+The current marker of where we have backfilled up to and will generally be the
+oldest-in-time events we know of in the DAG.
+
+This is an event where we haven't fetched all of the `prev_events` for.
+
+Once we have fetched all of its `prev_events`, it's unmarked as a backwards
+extremity (although we may have formed new backwards extremities from the prev
+events during the backfilling process).
+
+
+## Outliers
+
+We mark an event as an `outlier` when we haven't figured out the state for the
+room at that point in the DAG yet.
+
+We won't *necessarily* have the `prev_events` of an `outlier` in the database,
+but it's entirely possible that we *might*. The status of whether we have all of
+the `prev_events` is marked as a [backwards extremity](#backwards-extremity).
+
+For example, when we fetch the event auth chain or state for a given event, we
+mark all of those claimed auth events as outliers because we haven't done the
+state calculation ourself.
+
+
+## State groups
+
+For every non-outlier event we need to know the state at that event. Instead of
+storing the full state for each event in the DB (i.e. a `event_id -> state`
+mapping), which is *very* space inefficient when state doesn't change, we
+instead assign each different set of state a "state group" and then have
+mappings of `event_id -> state_group` and `state_group -> state`.
+
+
+### Stage group edges
+
+TODO: `state_group_edges` is a further optimization...
+      notes from @Azrenbeth, https://pastebin.com/seUGVGeT
diff --git a/docs/sample_log_config.yaml b/docs/sample_log_config.yaml
@@ -28,17 +28,14 @@ handlers:
     # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
     # logs will still be flushed immediately.
     buffer:
-        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
+        class: logging.handlers.MemoryHandler
         target: file
         # The capacity is the number of log lines that are buffered before
         # being written to disk. Increasing this will lead to better
         # performance, at the expensive of it taking longer for log lines to
         # be written to disk.
         capacity: 10
         flushLevel: 30  # Flush for WARNING logs as well
-        # The period of time, in seconds, between forced flushes.
-        # Messages will not be delayed for longer than this time.
-        period: 5
 
     # A handler that writes logs to stderr. Unused by default, but can be used
     # instead of "buffer" and "file" in the logger handlers.