From 2bae2c632ff595bda770212678521e04288f00a9 Mon Sep 17 00:00:00 2001
From: Eric Eastwood <erice@element.io>
Date: Tue, 3 Aug 2021 05:08:57 -0500
Subject: [PATCH 1/7] Add developer documentation to explain room DAG concepts
 like `outliers` and `state_groups` (#10464)

---
 changelog.d/10464.doc                 |  1 +
 docs/SUMMARY.md                       |  1 +
 docs/development/room-dag-concepts.md | 79 +++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)
 create mode 100644 changelog.d/10464.doc
 create mode 100644 docs/development/room-dag-concepts.md

diff --git a/changelog.d/10464.doc b/changelog.d/10464.doc
new file mode 100644
index 000000000000..764fb9f65c23
--- /dev/null
+++ b/changelog.d/10464.doc
@@ -0,0 +1 @@
+Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc.
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index f1bde91420fc..10be12d63865 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -79,6 +79,7 @@
     - [Single Sign-On]()
       - [SAML](development/saml.md)
       - [CAS](development/cas.md)
+    - [Room DAG concepts](development/room-dag-concepts.md)
     - [State Resolution]()
       - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
     - [Media Repository](media_repository.md)
diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md
new file mode 100644
index 000000000000..5eed72bec662
--- /dev/null
+++ b/docs/development/room-dag-concepts.md
@@ -0,0 +1,79 @@
+# Room DAG concepts
+
+## Edges
+
+The word "edge" comes from graph theory lingo. An edge is just a connection
+between two events. In Synapse, we connect events by specifying their
+`prev_events`. A subsequent event points back at a previous event.
+
+```
+A (oldest) <---- B <---- C (most recent)
+```
+
+
+## Depth and stream ordering
+
+Events are normally sorted by `(topological_ordering, stream_ordering)` where
+`topological_ordering` is just `depth`. In other words, we first sort by `depth`
+and then tie-break based on `stream_ordering`. `depth` is incremented as new
+messages are added to the DAG. Normally, `stream_ordering` is an auto
+incrementing integer, but backfilled events start with `stream_ordering=-1` and decrement.
+
+---
+
+ - `/sync` returns things in the order they arrive at the server (`stream_ordering`).
+ - `/messages` (and `/backfill` in the federation API) return them in the order determined by the event graph `(topological_ordering, stream_ordering)`.
+
+The general idea is that, if you're following a room in real-time (i.e.
+`/sync`), you probably want to see the messages as they arrive at your server,
+rather than skipping any that arrived late; whereas if you're looking at a
+historical section of timeline (i.e. `/messages`), you want to see the best
+representation of the state of the room as others were seeing it at the time.
+
+
+## Forward extremity
+
+Most-recent-in-time events in the DAG which are not referenced by any other events' `prev_events` yet.
+
+The forward extremities of a room are used as the `prev_events` when the next event is sent.
+
+
+## Backwards extremity
+
+The current marker of where we have backfilled up to and will generally be the
+oldest-in-time events we know of in the DAG.
+
+This is an event where we haven't fetched all of the `prev_events` for.
+
+Once we have fetched all of its `prev_events`, it's unmarked as a backwards
+extremity (although we may have formed new backwards extremities from the prev
+events during the backfilling process).
+
+
+## Outliers
+
+We mark an event as an `outlier` when we haven't figured out the state for the
+room at that point in the DAG yet.
+
+We won't *necessarily* have the `prev_events` of an `outlier` in the database,
+but it's entirely possible that we *might*. The status of whether we have all of
+the `prev_events` is marked as a [backwards extremity](#backwards-extremity).
+
+For example, when we fetch the event auth chain or state for a given event, we
+mark all of those claimed auth events as outliers because we haven't done the
+state calculation ourself.
+
+
+## State groups
+
+For every non-outlier event we need to know the state at that event. Instead of
+storing the full state for each event in the DB (i.e. a `event_id -> state`
+mapping), which is *very* space inefficient when state doesn't change, we
+instead assign each different set of state a "state group" and then have
+mappings of `event_id -> state_group` and `state_group -> state`.
+
+
+### Stage group edges
+
+TODO: `state_group_edges` is a further optimization...
+      notes from @Azrenbeth, https://pastebin.com/seUGVGeT

From a7bacccd8550b45fc1fa3dcff90f36125827b4ba Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 3 Aug 2021 11:23:45 +0100
Subject: [PATCH 2/7] Extend the release script to tag and create the releases.
 (#10496)

---
 changelog.d/10496.misc |   1 +
 scripts-dev/release.py | 311 ++++++++++++++++++++++++++++++++++++-----
 setup.py               |   2 +
 3 files changed, 278 insertions(+), 36 deletions(-)
 create mode 100644 changelog.d/10496.misc

diff --git a/changelog.d/10496.misc b/changelog.d/10496.misc
new file mode 100644
index 000000000000..6d0d3e539126
--- /dev/null
+++ b/changelog.d/10496.misc
@@ -0,0 +1 @@
+Extend release script to also tag and create GitHub releases.
diff --git a/scripts-dev/release.py b/scripts-dev/release.py
index cff433af2af2..e864dc6ed53f 100755
--- a/scripts-dev/release.py
+++ b/scripts-dev/release.py
@@ -14,29 +14,57 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""An interactive script for doing a release. See `run()` below.
+"""An interactive script for doing a release. See `cli()` below.
 """
 
+import re
 import subprocess
 import sys
-from typing import Optional
+import urllib.request
+from os import path
+from tempfile import TemporaryDirectory
+from typing import List, Optional, Tuple
 
+import attr
 import click
+import commonmark
 import git
+import redbaron
+from click.exceptions import ClickException
+from github import Github
 from packaging import version
-from redbaron import RedBaron
 
 
-@click.command()
-def run():
-    """An interactive script to walk through the initial stages of creating a
-    release, including creating release branch, updating changelog and pushing to
-    GitHub.
+@click.group()
+def cli():
+    """An interactive script to walk through the parts of creating a release.
 
     Requires the dev dependencies be installed, which can be done via:
 
         pip install -e .[dev]
 
+    Then to use:
+
+        ./scripts-dev/release.py prepare
+
+        # ... ask others to look at the changelog ...
+
+        ./scripts-dev/release.py tag
+
+        # ... wait for asssets to build ...
+
+        ./scripts-dev/release.py publish
+        ./scripts-dev/release.py upload
+
+    If the env var GH_TOKEN (or GITHUB_TOKEN) is set, or passed into the
+    `tag`/`publish` command, then a new draft release will be created/published.
+    """
+
+
+@cli.command()
+def prepare():
+    """Do the initial stages of creating a release, including creating release
+    branch, updating changelog and pushing to GitHub.
     """
 
     # Make sure we're in a git repo.
@@ -51,32 +79,8 @@ def run():
     click.secho("Updating git repo...")
     repo.remote().fetch()
 
-    # Parse the AST and load the `__version__` node so that we can edit it
-    # later.
-    with open("synapse/__init__.py") as f:
-        red = RedBaron(f.read())
-
-    version_node = None
-    for node in red:
-        if node.type != "assignment":
-            continue
-
-        if node.target.type != "name":
-            continue
-
-        if node.target.value != "__version__":
-            continue
-
-        version_node = node
-        break
-
-    if not version_node:
-        print("Failed to find '__version__' definition in synapse/__init__.py")
-        sys.exit(1)
-
-    # Parse the current version.
-    current_version = version.parse(version_node.value.value.strip('"'))
-    assert isinstance(current_version, version.Version)
+    # Get the current version and AST from root Synapse module.
+    current_version, parsed_synapse_ast, version_node = parse_version_from_module()
 
     # Figure out what sort of release we're doing and calcuate the new version.
     rc = click.confirm("RC", default=True)
@@ -190,7 +194,7 @@ def run():
     # Update the `__version__` variable and write it back to the file.
     version_node.value = '"' + new_version + '"'
     with open("synapse/__init__.py", "w") as f:
-        f.write(red.dumps())
+        f.write(parsed_synapse_ast.dumps())
 
     # Generate changelogs
     subprocess.run("python3 -m towncrier", shell=True)
@@ -240,6 +244,180 @@ def run():
     )
 
 
+@cli.command()
+@click.option("--gh-token", envvar=["GH_TOKEN", "GITHUB_TOKEN"])
+def tag(gh_token: Optional[str]):
+    """Tags the release and generates a draft GitHub release"""
+
+    # Make sure we're in a git repo.
+    try:
+        repo = git.Repo()
+    except git.InvalidGitRepositoryError:
+        raise click.ClickException("Not in Synapse repo.")
+
+    if repo.is_dirty():
+        raise click.ClickException("Uncommitted changes exist.")
+
+    click.secho("Updating git repo...")
+    repo.remote().fetch()
+
+    # Find out the version and tag name.
+    current_version, _, _ = parse_version_from_module()
+    tag_name = f"v{current_version}"
+
+    # Check we haven't released this version.
+    if tag_name in repo.tags:
+        raise click.ClickException(f"Tag {tag_name} already exists!\n")
+
+    # Get the appropriate changelogs and tag.
+    changes = get_changes_for_version(current_version)
+
+    click.echo_via_pager(changes)
+    if click.confirm("Edit text?", default=False):
+        changes = click.edit(changes, require_save=False)
+
+    repo.create_tag(tag_name, message=changes)
+
+    if not click.confirm("Push tag to GitHub?", default=True):
+        print("")
+        print("Run when ready to push:")
+        print("")
+        print(f"\tgit push {repo.remote().name} tag {current_version}")
+        print("")
+        return
+
+    repo.git.push(repo.remote().name, "tag", tag_name)
+
+    # If no token was given, we bail here
+    if not gh_token:
+        click.launch(f"https://github.com/matrix-org/synapse/releases/edit/{tag_name}")
+        return
+
+    # Create a new draft release
+    gh = Github(gh_token)
+    gh_repo = gh.get_repo("matrix-org/synapse")
+    release = gh_repo.create_git_release(
+        tag=tag_name,
+        name=tag_name,
+        message=changes,
+        draft=True,
+        prerelease=current_version.is_prerelease,
+    )
+
+    # Open the release and the actions where we are building the assets.
+    click.launch(release.url)
+    click.launch(
+        f"https://github.com/matrix-org/synapse/actions?query=branch%3A{tag_name}"
+    )
+
+    click.echo("Wait for release assets to be built")
+
+
+@cli.command()
+@click.option("--gh-token", envvar=["GH_TOKEN", "GITHUB_TOKEN"], required=True)
+def publish(gh_token: str):
+    """Publish release."""
+
+    # Make sure we're in a git repo.
+    try:
+        repo = git.Repo()
+    except git.InvalidGitRepositoryError:
+        raise click.ClickException("Not in Synapse repo.")
+
+    if repo.is_dirty():
+        raise click.ClickException("Uncommitted changes exist.")
+
+    current_version, _, _ = parse_version_from_module()
+    tag_name = f"v{current_version}"
+
+    if not click.confirm(f"Publish {tag_name}?", default=True):
+        return
+
+    # Publish the draft release
+    gh = Github(gh_token)
+    gh_repo = gh.get_repo("matrix-org/synapse")
+    for release in gh_repo.get_releases():
+        if release.title == tag_name:
+            break
+    else:
+        raise ClickException(f"Failed to find GitHub release for {tag_name}")
+
+    assert release.title == tag_name
+
+    if not release.draft:
+        click.echo("Release already published.")
+        return
+
+    release = release.update_release(
+        name=release.title,
+        message=release.body,
+        tag_name=release.tag_name,
+        prerelease=release.prerelease,
+        draft=False,
+    )
+
+
+@cli.command()
+def upload():
+    """Upload release to pypi."""
+
+    current_version, _, _ = parse_version_from_module()
+    tag_name = f"v{current_version}"
+
+    pypi_asset_names = [
+        f"matrix_synapse-{current_version}-py3-none-any.whl",
+        f"matrix-synapse-{current_version}.tar.gz",
+    ]
+
+    with TemporaryDirectory(prefix=f"synapse_upload_{tag_name}_") as tmpdir:
+        for name in pypi_asset_names:
+            filename = path.join(tmpdir, name)
+            url = f"https://github.com/matrix-org/synapse/releases/download/{tag_name}/{name}"
+
+            click.echo(f"Downloading {name} into {filename}")
+            urllib.request.urlretrieve(url, filename=filename)
+
+        if click.confirm("Upload to PyPI?", default=True):
+            subprocess.run("twine upload *", shell=True, cwd=tmpdir)
+
+    click.echo(
+        f"Done! Remember to merge the tag {tag_name} into the appropriate branches"
+    )
+
+
+def parse_version_from_module() -> Tuple[
+    version.Version, redbaron.RedBaron, redbaron.Node
+]:
+    # Parse the AST and load the `__version__` node so that we can edit it
+    # later.
+    with open("synapse/__init__.py") as f:
+        red = redbaron.RedBaron(f.read())
+
+    version_node = None
+    for node in red:
+        if node.type != "assignment":
+            continue
+
+        if node.target.type != "name":
+            continue
+
+        if node.target.value != "__version__":
+            continue
+
+        version_node = node
+        break
+
+    if not version_node:
+        print("Failed to find '__version__' definition in synapse/__init__.py")
+        sys.exit(1)
+
+    # Parse the current version.
+    current_version = version.parse(version_node.value.value.strip('"'))
+    assert isinstance(current_version, version.Version)
+
+    return current_version, red, version_node
+
+
 def find_ref(repo: git.Repo, ref_name: str) -> Optional[git.HEAD]:
     """Find the branch/ref, looking first locally then in the remote."""
     if ref_name in repo.refs:
@@ -256,5 +434,66 @@ def update_branch(repo: git.Repo):
         repo.git.merge(repo.active_branch.tracking_branch().name)
 
 
+def get_changes_for_version(wanted_version: version.Version) -> str:
+    """Get the changelogs for the given version.
+
+    If an RC then will only get the changelog for that RC version, otherwise if
+    its a full release will get the changelog for the release and all its RCs.
+    """
+
+    with open("CHANGES.md") as f:
+        changes = f.read()
+
+    # First we parse the changelog so that we can split it into sections based
+    # on the release headings.
+    ast = commonmark.Parser().parse(changes)
+
+    @attr.s(auto_attribs=True)
+    class VersionSection:
+        title: str
+
+        # These are 0-based.
+        start_line: int
+        end_line: Optional[int] = None  # Is none if its the last entry
+
+    headings: List[VersionSection] = []
+    for node, _ in ast.walker():
+        # We look for all text nodes that are in a level 1 heading.
+        if node.t != "text":
+            continue
+
+        if node.parent.t != "heading" or node.parent.level != 1:
+            continue
+
+        # If we have a previous heading then we update its `end_line`.
+        if headings:
+            headings[-1].end_line = node.parent.sourcepos[0][0] - 1
+
+        headings.append(VersionSection(node.literal, node.parent.sourcepos[0][0] - 1))
+
+    changes_by_line = changes.split("\n")
+
+    version_changelog = []  # The lines we want to include in the changelog
+
+    # Go through each section and find any that match the requested version.
+    regex = re.compile(r"^Synapse v?(\S+)")
+    for section in headings:
+        groups = regex.match(section.title)
+        if not groups:
+            continue
+
+        heading_version = version.parse(groups.group(1))
+        heading_base_version = version.parse(heading_version.base_version)
+
+        # Check if heading version matches the requested version, or if its an
+        # RC of the requested version.
+        if wanted_version not in (heading_version, heading_base_version):
+            continue
+
+        version_changelog.extend(changes_by_line[section.start_line : section.end_line])
+
+    return "\n".join(version_changelog)
+
+
 if __name__ == "__main__":
-    run()
+    cli()
diff --git a/setup.py b/setup.py
index 1081548e0067..c47856351081 100755
--- a/setup.py
+++ b/setup.py
@@ -108,6 +108,8 @@ def exec_file(path_segments):
     "click==7.1.2",
     "redbaron==0.9.2",
     "GitPython==3.1.14",
+    "commonmark==0.9.1",
+    "pygithub==1.55",
 ]
 
 CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]

From f4ac934afe1d91bb0cc1bb6dafc77a94165aa740 Mon Sep 17 00:00:00 2001
From: reivilibre <38398653+reivilibre@users.noreply.github.com>
Date: Tue, 3 Aug 2021 11:30:39 +0100
Subject: [PATCH 3/7] Revert use of PeriodicallyFlushingMemoryHandler by
 default (#10515)

---
 changelog.d/10515.feature   | 1 +
 docs/sample_log_config.yaml | 5 +----
 synapse/config/logger.py    | 5 +----
 3 files changed, 3 insertions(+), 8 deletions(-)
 create mode 100644 changelog.d/10515.feature

diff --git a/changelog.d/10515.feature b/changelog.d/10515.feature
new file mode 100644
index 000000000000..db277d9ecde0
--- /dev/null
+++ b/changelog.d/10515.feature
@@ -0,0 +1 @@
+Add a buffered logging handler which periodically flushes itself.
diff --git a/docs/sample_log_config.yaml b/docs/sample_log_config.yaml
index b088c834057d..669e60008113 100644
--- a/docs/sample_log_config.yaml
+++ b/docs/sample_log_config.yaml
@@ -28,7 +28,7 @@ handlers:
     # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
     # logs will still be flushed immediately.
     buffer:
-        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
+        class: logging.handlers.MemoryHandler
         target: file
         # The capacity is the number of log lines that are buffered before
         # being written to disk. Increasing this will lead to better
@@ -36,9 +36,6 @@ handlers:
         # be written to disk.
         capacity: 10
         flushLevel: 30  # Flush for WARNING logs as well
-        # The period of time, in seconds, between forced flushes.
-        # Messages will not be delayed for longer than this time.
-        period: 5
 
     # A handler that writes logs to stderr. Unused by default, but can be used
     # instead of "buffer" and "file" in the logger handlers.
diff --git a/synapse/config/logger.py b/synapse/config/logger.py
index dcd3ed1dac12..ad4e6e61c3bf 100644
--- a/synapse/config/logger.py
+++ b/synapse/config/logger.py
@@ -71,7 +71,7 @@
     # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
     # logs will still be flushed immediately.
     buffer:
-        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
+        class: logging.handlers.MemoryHandler
         target: file
         # The capacity is the number of log lines that are buffered before
         # being written to disk. Increasing this will lead to better
@@ -79,9 +79,6 @@
         # be written to disk.
         capacity: 10
         flushLevel: 30  # Flush for WARNING logs as well
-        # The period of time, in seconds, between forced flushes.
-        # Messages will not be delayed for longer than this time.
-        period: 5
 
     # A handler that writes logs to stderr. Unused by default, but can be used
     # instead of "buffer" and "file" in the logger handlers.

From c8566191fcd7edf224db468f860c1b638fb8e763 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 3 Aug 2021 11:32:10 +0100
Subject: [PATCH 4/7]  1.40.0rc1

---
 CHANGES.md                | 60 +++++++++++++++++++++++++++++++++++++++
 changelog.d/10245.feature |  1 -
 changelog.d/10254.feature |  1 -
 changelog.d/10283.feature |  1 -
 changelog.d/10390.misc    |  1 -
 changelog.d/10407.feature |  1 -
 changelog.d/10408.misc    |  1 -
 changelog.d/10410.bugfix  |  1 -
 changelog.d/10411.feature |  1 -
 changelog.d/10413.feature |  1 -
 changelog.d/10415.misc    |  1 -
 changelog.d/10426.feature |  1 -
 changelog.d/10429.misc    |  1 -
 changelog.d/10431.misc    |  1 -
 changelog.d/10432.misc    |  1 -
 changelog.d/10437.misc    |  1 -
 changelog.d/10438.misc    |  1 -
 changelog.d/10439.bugfix  |  1 -
 changelog.d/10440.feature |  1 -
 changelog.d/10442.misc    |  1 -
 changelog.d/10444.misc    |  1 -
 changelog.d/10445.doc     |  1 -
 changelog.d/10446.misc    |  1 -
 changelog.d/10447.feature |  1 -
 changelog.d/10448.feature |  1 -
 changelog.d/10450.misc    |  1 -
 changelog.d/10451.misc    |  1 -
 changelog.d/10453.doc     |  1 -
 changelog.d/10455.bugfix  |  1 -
 changelog.d/10463.misc    |  1 -
 changelog.d/10464.doc     |  1 -
 changelog.d/10468.misc    |  1 -
 changelog.d/10482.misc    |  1 -
 changelog.d/10483.doc     |  1 -
 changelog.d/10488.misc    |  1 -
 changelog.d/10489.feature |  1 -
 changelog.d/10490.misc    |  1 -
 changelog.d/10491.misc    |  1 -
 changelog.d/10496.misc    |  1 -
 changelog.d/10499.bugfix  |  1 -
 changelog.d/10500.misc    |  1 -
 changelog.d/10511.feature |  1 -
 changelog.d/10512.misc    |  1 -
 changelog.d/10515.feature |  1 -
 changelog.d/9918.feature  |  1 -
 debian/changelog          |  8 ++++--
 synapse/__init__.py       |  2 +-
 47 files changed, 67 insertions(+), 47 deletions(-)
 delete mode 100644 changelog.d/10245.feature
 delete mode 100644 changelog.d/10254.feature
 delete mode 100644 changelog.d/10283.feature
 delete mode 100644 changelog.d/10390.misc
 delete mode 100644 changelog.d/10407.feature
 delete mode 100644 changelog.d/10408.misc
 delete mode 100644 changelog.d/10410.bugfix
 delete mode 100644 changelog.d/10411.feature
 delete mode 100644 changelog.d/10413.feature
 delete mode 100644 changelog.d/10415.misc
 delete mode 100644 changelog.d/10426.feature
 delete mode 100644 changelog.d/10429.misc
 delete mode 100644 changelog.d/10431.misc
 delete mode 100644 changelog.d/10432.misc
 delete mode 100644 changelog.d/10437.misc
 delete mode 100644 changelog.d/10438.misc
 delete mode 100644 changelog.d/10439.bugfix
 delete mode 100644 changelog.d/10440.feature
 delete mode 100644 changelog.d/10442.misc
 delete mode 100644 changelog.d/10444.misc
 delete mode 100644 changelog.d/10445.doc
 delete mode 100644 changelog.d/10446.misc
 delete mode 100644 changelog.d/10447.feature
 delete mode 100644 changelog.d/10448.feature
 delete mode 100644 changelog.d/10450.misc
 delete mode 100644 changelog.d/10451.misc
 delete mode 100644 changelog.d/10453.doc
 delete mode 100644 changelog.d/10455.bugfix
 delete mode 100644 changelog.d/10463.misc
 delete mode 100644 changelog.d/10464.doc
 delete mode 100644 changelog.d/10468.misc
 delete mode 100644 changelog.d/10482.misc
 delete mode 100644 changelog.d/10483.doc
 delete mode 100644 changelog.d/10488.misc
 delete mode 100644 changelog.d/10489.feature
 delete mode 100644 changelog.d/10490.misc
 delete mode 100644 changelog.d/10491.misc
 delete mode 100644 changelog.d/10496.misc
 delete mode 100644 changelog.d/10499.bugfix
 delete mode 100644 changelog.d/10500.misc
 delete mode 100644 changelog.d/10511.feature
 delete mode 100644 changelog.d/10512.misc
 delete mode 100644 changelog.d/10515.feature
 delete mode 100644 changelog.d/9918.feature

diff --git a/CHANGES.md b/CHANGES.md
index 653324928145..34274cfe9ca8 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,63 @@
+Synapse 1.40.0rc1 (2021-08-03)
+==============================
+
+Features
+--------
+
+- Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`. ([\#9918](https://github.com/matrix-org/synapse/issues/9918))
+- Make historical events discoverable from backfill for servers without any scrollback history (part of MSC2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245))
+- Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](https://github.com/matrix-org/synapse/issues/10254), [\#10447](https://github.com/matrix-org/synapse/issues/10447), [\#10489](https://github.com/matrix-org/synapse/issues/10489))
+- Initial support for MSC3244, Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
+- Add a buffered logging handler which periodically flushes itself. ([\#10407](https://github.com/matrix-org/synapse/issues/10407), [\#10515](https://github.com/matrix-org/synapse/issues/10515))
+- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](https://github.com/matrix-org/synapse/issues/10411))
+- Support for [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-doc/pull/2285). Contributed by @SimonBrandner. ([\#10413](https://github.com/matrix-org/synapse/issues/10413))
+- Email notifications now state whether an invitation is to a room or a space. ([\#10426](https://github.com/matrix-org/synapse/issues/10426))
+- Allow setting transaction limit for database connections. ([\#10440](https://github.com/matrix-org/synapse/issues/10440), [\#10511](https://github.com/matrix-org/synapse/issues/10511))
+- Add `creation_ts` to list users admin API. ([\#10448](https://github.com/matrix-org/synapse/issues/10448))
+
+
+Bugfixes
+--------
+
+- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](https://github.com/matrix-org/synapse/issues/10410))
+- Fix events with floating outlier state being rejected over federation. ([\#10439](https://github.com/matrix-org/synapse/issues/10439))
+- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](https://github.com/matrix-org/synapse/issues/10455))
+- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](https://github.com/matrix-org/synapse/issues/10499))
+
+
+Improved Documentation
+----------------------
+
+- Fix hierarchy of providers on the OpenID page. ([\#10445](https://github.com/matrix-org/synapse/issues/10445))
+- Consolidate development documentation to `docs/development/`. ([\#10453](https://github.com/matrix-org/synapse/issues/10453))
+- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\#10464](https://github.com/matrix-org/synapse/issues/10464))
+- Document how to use Complement while developing a new Synapse feature. ([\#10483](https://github.com/matrix-org/synapse/issues/10483))
+
+
+Internal Changes
+----------------
+
+- Prune inbound federation inbound queues for a room if they get too large. ([\#10390](https://github.com/matrix-org/synapse/issues/10390))
+- Add type hints to `synapse.federation.transport.client` module. ([\#10408](https://github.com/matrix-org/synapse/issues/10408))
+- Remove shebang line from module files. ([\#10415](https://github.com/matrix-org/synapse/issues/10415))
+- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](https://github.com/matrix-org/synapse/issues/10429))
+- Use a docker image cache for the prerequisites for the debian package build. ([\#10431](https://github.com/matrix-org/synapse/issues/10431))
+- Connect historical chunks together with chunk events instead of a content field (MSC2716). ([\#10432](https://github.com/matrix-org/synapse/issues/10432))
+- Improve servlet type hints. ([\#10437](https://github.com/matrix-org/synapse/issues/10437), [\#10438](https://github.com/matrix-org/synapse/issues/10438))
+- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](https://github.com/matrix-org/synapse/issues/10442))
+- Update the `tests-done` Github Actions status. ([\#10444](https://github.com/matrix-org/synapse/issues/10444), [\#10512](https://github.com/matrix-org/synapse/issues/10512))
+- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](https://github.com/matrix-org/synapse/issues/10446), [\#10450](https://github.com/matrix-org/synapse/issues/10450))
+- Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](https://github.com/matrix-org/synapse/issues/10451))
+- Disable `msc2716` Complement tests until Complement updates are merged. ([\#10463](https://github.com/matrix-org/synapse/issues/10463))
+- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](https://github.com/matrix-org/synapse/issues/10468))
+- Additional type hints in the state handler. ([\#10482](https://github.com/matrix-org/synapse/issues/10482))
+- Update syntax used to run complement tests. ([\#10488](https://github.com/matrix-org/synapse/issues/10488))
+- Fix up type annotations to work with Twisted 21.7. ([\#10490](https://github.com/matrix-org/synapse/issues/10490))
+- Improve type annotations for `ObservableDeferred`. ([\#10491](https://github.com/matrix-org/synapse/issues/10491))
+- Extend release script to also tag and create GitHub releases. ([\#10496](https://github.com/matrix-org/synapse/issues/10496))
+- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\#10500](https://github.com/matrix-org/synapse/issues/10500))
+
+
 Synapse 1.39.0 (2021-07-29)
 ===========================
 
diff --git a/changelog.d/10245.feature b/changelog.d/10245.feature
deleted file mode 100644
index b3c48cc2cc25..000000000000
--- a/changelog.d/10245.feature
+++ /dev/null
@@ -1 +0,0 @@
-Make historical events discoverable from backfill for servers without any scrollback history (part of MSC2716).
diff --git a/changelog.d/10254.feature b/changelog.d/10254.feature
deleted file mode 100644
index df8bb51167f7..000000000000
--- a/changelog.d/10254.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events.
diff --git a/changelog.d/10283.feature b/changelog.d/10283.feature
deleted file mode 100644
index 99d633dbfb86..000000000000
--- a/changelog.d/10283.feature
+++ /dev/null
@@ -1 +0,0 @@
-Initial support for MSC3244, Room version capabilities over the /capabilities API.
\ No newline at end of file
diff --git a/changelog.d/10390.misc b/changelog.d/10390.misc
deleted file mode 100644
index 911a5733eee8..000000000000
--- a/changelog.d/10390.misc
+++ /dev/null
@@ -1 +0,0 @@
-Prune inbound federation inbound queues for a room if they get too large.
diff --git a/changelog.d/10407.feature b/changelog.d/10407.feature
deleted file mode 100644
index db277d9ecde0..000000000000
--- a/changelog.d/10407.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a buffered logging handler which periodically flushes itself.
diff --git a/changelog.d/10408.misc b/changelog.d/10408.misc
deleted file mode 100644
index abccd210a924..000000000000
--- a/changelog.d/10408.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to `synapse.federation.transport.client` module.
diff --git a/changelog.d/10410.bugfix b/changelog.d/10410.bugfix
deleted file mode 100644
index 65b418fd35ee..000000000000
--- a/changelog.d/10410.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut.
diff --git a/changelog.d/10411.feature b/changelog.d/10411.feature
deleted file mode 100644
index ef0ab84b17b1..000000000000
--- a/changelog.d/10411.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/10413.feature b/changelog.d/10413.feature
deleted file mode 100644
index 3964db7e0eae..000000000000
--- a/changelog.d/10413.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support for [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-doc/pull/2285). Contributed by @SimonBrandner.
diff --git a/changelog.d/10415.misc b/changelog.d/10415.misc
deleted file mode 100644
index 3b9501acbbcb..000000000000
--- a/changelog.d/10415.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove shebang line from module files.
diff --git a/changelog.d/10426.feature b/changelog.d/10426.feature
deleted file mode 100644
index 9cca6dc456df..000000000000
--- a/changelog.d/10426.feature
+++ /dev/null
@@ -1 +0,0 @@
-Email notifications now state whether an invitation is to a room or a space.
diff --git a/changelog.d/10429.misc b/changelog.d/10429.misc
deleted file mode 100644
index ccb2217f64fd..000000000000
--- a/changelog.d/10429.misc
+++ /dev/null
@@ -1 +0,0 @@
-Drop backwards-compatibility code that was required to support Ubuntu Xenial.
diff --git a/changelog.d/10431.misc b/changelog.d/10431.misc
deleted file mode 100644
index 34b9b49da680..000000000000
--- a/changelog.d/10431.misc
+++ /dev/null
@@ -1 +0,0 @@
-Use a docker image cache for the prerequisites for the debian package build.
diff --git a/changelog.d/10432.misc b/changelog.d/10432.misc
deleted file mode 100644
index 3a8cdf0ae0e9..000000000000
--- a/changelog.d/10432.misc
+++ /dev/null
@@ -1 +0,0 @@
-Connect historical chunks together with chunk events instead of a content field (MSC2716).
diff --git a/changelog.d/10437.misc b/changelog.d/10437.misc
deleted file mode 100644
index a55757849903..000000000000
--- a/changelog.d/10437.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve servlet type hints.
diff --git a/changelog.d/10438.misc b/changelog.d/10438.misc
deleted file mode 100644
index a55757849903..000000000000
--- a/changelog.d/10438.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve servlet type hints.
diff --git a/changelog.d/10439.bugfix b/changelog.d/10439.bugfix
deleted file mode 100644
index 74e5a25126d9..000000000000
--- a/changelog.d/10439.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix events with floating outlier state being rejected over federation.
diff --git a/changelog.d/10440.feature b/changelog.d/10440.feature
deleted file mode 100644
index f1833b0bd726..000000000000
--- a/changelog.d/10440.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow setting transaction limit for database connections.
diff --git a/changelog.d/10442.misc b/changelog.d/10442.misc
deleted file mode 100644
index b8d412d73289..000000000000
--- a/changelog.d/10442.misc
+++ /dev/null
@@ -1 +0,0 @@
-Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages.
diff --git a/changelog.d/10444.misc b/changelog.d/10444.misc
deleted file mode 100644
index c012e89f4bbb..000000000000
--- a/changelog.d/10444.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update the `tests-done` Github Actions status.
diff --git a/changelog.d/10445.doc b/changelog.d/10445.doc
deleted file mode 100644
index 4c023ded7ca6..000000000000
--- a/changelog.d/10445.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix hierarchy of providers on the OpenID page.
diff --git a/changelog.d/10446.misc b/changelog.d/10446.misc
deleted file mode 100644
index a5a0ca80eb4d..000000000000
--- a/changelog.d/10446.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update type annotations to work with forthcoming Twisted 21.7.0 release.
diff --git a/changelog.d/10447.feature b/changelog.d/10447.feature
deleted file mode 100644
index df8bb51167f7..000000000000
--- a/changelog.d/10447.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events.
diff --git a/changelog.d/10448.feature b/changelog.d/10448.feature
deleted file mode 100644
index f6579e0ca85b..000000000000
--- a/changelog.d/10448.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add `creation_ts` to list users admin API.
\ No newline at end of file
diff --git a/changelog.d/10450.misc b/changelog.d/10450.misc
deleted file mode 100644
index aa646f0841c7..000000000000
--- a/changelog.d/10450.misc
+++ /dev/null
@@ -1 +0,0 @@
- Update type annotations to work with forthcoming Twisted 21.7.0 release.
diff --git a/changelog.d/10451.misc b/changelog.d/10451.misc
deleted file mode 100644
index e38f4b476dde..000000000000
--- a/changelog.d/10451.misc
+++ /dev/null
@@ -1 +0,0 @@
-Cancel redundant GHA workflows when a new commit is pushed.
diff --git a/changelog.d/10453.doc b/changelog.d/10453.doc
deleted file mode 100644
index 5d4db9bca29c..000000000000
--- a/changelog.d/10453.doc
+++ /dev/null
@@ -1 +0,0 @@
-Consolidate development documentation to `docs/development/`.
diff --git a/changelog.d/10455.bugfix b/changelog.d/10455.bugfix
deleted file mode 100644
index 23c74a3c8912..000000000000
--- a/changelog.d/10455.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty.
diff --git a/changelog.d/10463.misc b/changelog.d/10463.misc
deleted file mode 100644
index d7b4d2222e10..000000000000
--- a/changelog.d/10463.misc
+++ /dev/null
@@ -1 +0,0 @@
-Disable `msc2716` Complement tests until Complement updates are merged.
diff --git a/changelog.d/10464.doc b/changelog.d/10464.doc
deleted file mode 100644
index 764fb9f65c23..000000000000
--- a/changelog.d/10464.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc.
diff --git a/changelog.d/10468.misc b/changelog.d/10468.misc
deleted file mode 100644
index b9854bb4c16c..000000000000
--- a/changelog.d/10468.misc
+++ /dev/null
@@ -1 +0,0 @@
-Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header.
diff --git a/changelog.d/10482.misc b/changelog.d/10482.misc
deleted file mode 100644
index 4e9e2126e173..000000000000
--- a/changelog.d/10482.misc
+++ /dev/null
@@ -1 +0,0 @@
-Additional type hints in the state handler.
diff --git a/changelog.d/10483.doc b/changelog.d/10483.doc
deleted file mode 100644
index 0f699fafddba..000000000000
--- a/changelog.d/10483.doc
+++ /dev/null
@@ -1 +0,0 @@
-Document how to use Complement while developing a new Synapse feature.
diff --git a/changelog.d/10488.misc b/changelog.d/10488.misc
deleted file mode 100644
index a55502c16359..000000000000
--- a/changelog.d/10488.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update syntax used to run complement tests.
diff --git a/changelog.d/10489.feature b/changelog.d/10489.feature
deleted file mode 100644
index df8bb51167f7..000000000000
--- a/changelog.d/10489.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events.
diff --git a/changelog.d/10490.misc b/changelog.d/10490.misc
deleted file mode 100644
index 630c31adaeb9..000000000000
--- a/changelog.d/10490.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix up type annotations to work with Twisted 21.7.
diff --git a/changelog.d/10491.misc b/changelog.d/10491.misc
deleted file mode 100644
index 3867cf2682c1..000000000000
--- a/changelog.d/10491.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type annotations for `ObservableDeferred`.
diff --git a/changelog.d/10496.misc b/changelog.d/10496.misc
deleted file mode 100644
index 6d0d3e539126..000000000000
--- a/changelog.d/10496.misc
+++ /dev/null
@@ -1 +0,0 @@
-Extend release script to also tag and create GitHub releases.
diff --git a/changelog.d/10499.bugfix b/changelog.d/10499.bugfix
deleted file mode 100644
index 6487af6c9691..000000000000
--- a/changelog.d/10499.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances.
diff --git a/changelog.d/10500.misc b/changelog.d/10500.misc
deleted file mode 100644
index dbaff57364d2..000000000000
--- a/changelog.d/10500.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'.
diff --git a/changelog.d/10511.feature b/changelog.d/10511.feature
deleted file mode 100644
index f1833b0bd726..000000000000
--- a/changelog.d/10511.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow setting transaction limit for database connections.
diff --git a/changelog.d/10512.misc b/changelog.d/10512.misc
deleted file mode 100644
index c012e89f4bbb..000000000000
--- a/changelog.d/10512.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update the `tests-done` Github Actions status.
diff --git a/changelog.d/10515.feature b/changelog.d/10515.feature
deleted file mode 100644
index db277d9ecde0..000000000000
--- a/changelog.d/10515.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a buffered logging handler which periodically flushes itself.
diff --git a/changelog.d/9918.feature b/changelog.d/9918.feature
deleted file mode 100644
index 98f0a5089321..000000000000
--- a/changelog.d/9918.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`.
\ No newline at end of file
diff --git a/debian/changelog b/debian/changelog
index 341c1ac99268..f0557c35ef87 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,12 @@
-matrix-synapse-py3 (1.39.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.40.0~rc1) stable; urgency=medium
 
+  [ Richard van der Hoff ]
   * Drop backwards-compatibility code that was required to support Ubuntu Xenial.
 
- -- Richard van der Hoff <richard@matrix.org>  Tue, 20 Jul 2021 00:10:03 +0100
+  [ Synapse Packaging team ]
+  * New synapse release 1.40.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 03 Aug 2021 11:31:49 +0100
 
 matrix-synapse-py3 (1.39.0) stable; urgency=medium
 
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 5da6c924fcec..d6c176550897 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -47,7 +47,7 @@
 except ImportError:
     pass
 
-__version__ = "1.39.0"
+__version__ = "1.40.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

From c80ec5d15386a8d3db03d0064b4e87d52d38dff2 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 3 Aug 2021 11:48:48 +0100
Subject: [PATCH 5/7] Fixup changelog

---
 CHANGES.md | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 34274cfe9ca8..8b78fe92f6a0 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -5,22 +5,22 @@ Features
 --------
 
 - Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`. ([\#9918](https://github.com/matrix-org/synapse/issues/9918))
-- Make historical events discoverable from backfill for servers without any scrollback history (part of MSC2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245))
+- Make historical events discoverable from backfill for servers without any scrollback history (part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10245](https://github.com/matrix-org/synapse/issues/10245))
 - Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](https://github.com/matrix-org/synapse/issues/10254), [\#10447](https://github.com/matrix-org/synapse/issues/10447), [\#10489](https://github.com/matrix-org/synapse/issues/10489))
-- Initial support for MSC3244, Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
+- Initial support for [MSC3244](https://github.com/matrix-org/matrix-doc/pull/3244), Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
 - Add a buffered logging handler which periodically flushes itself. ([\#10407](https://github.com/matrix-org/synapse/issues/10407), [\#10515](https://github.com/matrix-org/synapse/issues/10515))
 - Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](https://github.com/matrix-org/synapse/issues/10411))
 - Support for [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-doc/pull/2285). Contributed by @SimonBrandner. ([\#10413](https://github.com/matrix-org/synapse/issues/10413))
 - Email notifications now state whether an invitation is to a room or a space. ([\#10426](https://github.com/matrix-org/synapse/issues/10426))
 - Allow setting transaction limit for database connections. ([\#10440](https://github.com/matrix-org/synapse/issues/10440), [\#10511](https://github.com/matrix-org/synapse/issues/10511))
-- Add `creation_ts` to list users admin API. ([\#10448](https://github.com/matrix-org/synapse/issues/10448))
+- Add `creation_ts` to "list users" admin API. ([\#10448](https://github.com/matrix-org/synapse/issues/10448))
 
 
 Bugfixes
 --------
 
 - Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](https://github.com/matrix-org/synapse/issues/10410))
-- Fix events with floating outlier state being rejected over federation. ([\#10439](https://github.com/matrix-org/synapse/issues/10439))
+- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\#10439](https://github.com/matrix-org/synapse/issues/10439))
 - Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](https://github.com/matrix-org/synapse/issues/10455))
 - Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](https://github.com/matrix-org/synapse/issues/10499))
 
@@ -37,12 +37,12 @@ Improved Documentation
 Internal Changes
 ----------------
 
-- Prune inbound federation inbound queues for a room if they get too large. ([\#10390](https://github.com/matrix-org/synapse/issues/10390))
+- Prune inbound federation queues for a room if they get too large. ([\#10390](https://github.com/matrix-org/synapse/issues/10390))
 - Add type hints to `synapse.federation.transport.client` module. ([\#10408](https://github.com/matrix-org/synapse/issues/10408))
 - Remove shebang line from module files. ([\#10415](https://github.com/matrix-org/synapse/issues/10415))
 - Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](https://github.com/matrix-org/synapse/issues/10429))
 - Use a docker image cache for the prerequisites for the debian package build. ([\#10431](https://github.com/matrix-org/synapse/issues/10431))
-- Connect historical chunks together with chunk events instead of a content field (MSC2716). ([\#10432](https://github.com/matrix-org/synapse/issues/10432))
+- Connect historical chunks together with chunk events instead of a content field ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10432](https://github.com/matrix-org/synapse/issues/10432))
 - Improve servlet type hints. ([\#10437](https://github.com/matrix-org/synapse/issues/10437), [\#10438](https://github.com/matrix-org/synapse/issues/10438))
 - Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](https://github.com/matrix-org/synapse/issues/10442))
 - Update the `tests-done` Github Actions status. ([\#10444](https://github.com/matrix-org/synapse/issues/10444), [\#10512](https://github.com/matrix-org/synapse/issues/10512))

From da6cd82106636d4c8b5143d7c2839f11fb40fbd2 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 3 Aug 2021 12:11:26 +0100
Subject: [PATCH 6/7] Fixup changelog

---
 CHANGES.md | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 8b78fe92f6a0..f2d694588618 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -5,7 +5,7 @@ Features
 --------
 
 - Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`. ([\#9918](https://github.com/matrix-org/synapse/issues/9918))
-- Make historical events discoverable from backfill for servers without any scrollback history (part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10245](https://github.com/matrix-org/synapse/issues/10245))
+- Add support for [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245), [\#10432](https://github.com/matrix-org/synapse/issues/10432), [\#10463](https://github.com/matrix-org/synapse/issues/10463))
 - Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](https://github.com/matrix-org/synapse/issues/10254), [\#10447](https://github.com/matrix-org/synapse/issues/10447), [\#10489](https://github.com/matrix-org/synapse/issues/10489))
 - Initial support for [MSC3244](https://github.com/matrix-org/matrix-doc/pull/3244), Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
 - Add a buffered logging handler which periodically flushes itself. ([\#10407](https://github.com/matrix-org/synapse/issues/10407), [\#10515](https://github.com/matrix-org/synapse/issues/10515))
@@ -42,13 +42,11 @@ Internal Changes
 - Remove shebang line from module files. ([\#10415](https://github.com/matrix-org/synapse/issues/10415))
 - Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](https://github.com/matrix-org/synapse/issues/10429))
 - Use a docker image cache for the prerequisites for the debian package build. ([\#10431](https://github.com/matrix-org/synapse/issues/10431))
-- Connect historical chunks together with chunk events instead of a content field ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10432](https://github.com/matrix-org/synapse/issues/10432))
 - Improve servlet type hints. ([\#10437](https://github.com/matrix-org/synapse/issues/10437), [\#10438](https://github.com/matrix-org/synapse/issues/10438))
 - Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](https://github.com/matrix-org/synapse/issues/10442))
 - Update the `tests-done` Github Actions status. ([\#10444](https://github.com/matrix-org/synapse/issues/10444), [\#10512](https://github.com/matrix-org/synapse/issues/10512))
 - Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](https://github.com/matrix-org/synapse/issues/10446), [\#10450](https://github.com/matrix-org/synapse/issues/10450))
 - Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](https://github.com/matrix-org/synapse/issues/10451))
-- Disable `msc2716` Complement tests until Complement updates are merged. ([\#10463](https://github.com/matrix-org/synapse/issues/10463))
 - Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](https://github.com/matrix-org/synapse/issues/10468))
 - Additional type hints in the state handler. ([\#10482](https://github.com/matrix-org/synapse/issues/10482))
 - Update syntax used to run complement tests. ([\#10488](https://github.com/matrix-org/synapse/issues/10488))

From 42225aa421efa9dd87fc63286f24f4697e4d2572 Mon Sep 17 00:00:00 2001
From: Erik Johnston <erik@matrix.org>
Date: Tue, 3 Aug 2021 12:12:50 +0100
Subject: [PATCH 7/7] Fixup changelog

---
 CHANGES.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CHANGES.md b/CHANGES.md
index f2d694588618..7ce28c4c182a 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -5,7 +5,7 @@ Features
 --------
 
 - Add support for [MSC2033](https://github.com/matrix-org/matrix-doc/pull/2033): `device_id` on `/account/whoami`. ([\#9918](https://github.com/matrix-org/synapse/issues/9918))
-- Add support for [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245), [\#10432](https://github.com/matrix-org/synapse/issues/10432), [\#10463](https://github.com/matrix-org/synapse/issues/10463))
+- Update support for [MSC2716 - Incrementally importing history into existing rooms](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10245](https://github.com/matrix-org/synapse/issues/10245), [\#10432](https://github.com/matrix-org/synapse/issues/10432), [\#10463](https://github.com/matrix-org/synapse/issues/10463))
 - Update support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](https://github.com/matrix-org/synapse/issues/10254), [\#10447](https://github.com/matrix-org/synapse/issues/10447), [\#10489](https://github.com/matrix-org/synapse/issues/10489))
 - Initial support for [MSC3244](https://github.com/matrix-org/matrix-doc/pull/3244), Room version capabilities over the /capabilities API. ([\#10283](https://github.com/matrix-org/synapse/issues/10283))
 - Add a buffered logging handler which periodically flushes itself. ([\#10407](https://github.com/matrix-org/synapse/issues/10407), [\#10515](https://github.com/matrix-org/synapse/issues/10515))