Add support for native e2ee

[Half-Shot]

This PR adds support for native e2ee within the bridge, and drops support for Pantalaimon based bridging.

N.B this doesn't work yet, seeing failures like:

ERROR 11:34:53:742 [Appservice] Decryption error on !ltmLLVxeBLdNUWQISs:beefy $8gHWPs07YA8Ao-Ormsn71WwhHdP37VonGS8gfsYwKsU Error: Decryption failed: decryption failed because the room key is missing
    at OlmMachine.decryptRoomEvent (/home/will/git/matrix-hookshot/node_modules/@turt2live/matrix-sdk-crypto-nodejs/src/ts/OlmMachine.ts:305:37)
    at CryptoClient.<anonymous> (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/e2ee/CryptoClient.js:157:50)
    at Generator.next (<anonymous>)
    at /home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/e2ee/CryptoClient.js:17:71
    at new Promise (<anonymous>)
    at __awaiter (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/e2ee/CryptoClient.js:13:12)
    at CryptoClient.decryptRoomEvent (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/e2ee/CryptoClient.js:156:16)
    at CryptoClient.descriptor.value (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/e2ee/decorators.js:33:35)
    at Appservice.<anonymous> (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/appservice/Appservice.js:540:83)
    at Generator.next (<anonymous>)
    at fulfilled (/home/will/git/matrix-hookshot/node_modules/matrix-bot-sdk/lib/appservice/Appservice.js:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'GenericFailure'
}

[Half-Shot]

Currently blocked on turt2live/matrix-bot-sdk#208

[B4dM4n]

The matrix-bot-sdk has recently released v0.6.1 which contains a fix for the mentioned issue.

I tried to rebase this PR onto main and update matrix-bot-sdk version. My current effort can be found here: main...B4dM4n:matrix-hookshot:native-e2ee

The Hookshot Bot can decrypt messages I send him, but Element can't decrypt messages send by the bot (** Unable to decrypt: The sender's device has not sent us the keys for this message. **).

I'm not sure why this happens, but a very simple bot using the matrix-bot-sdk is correctly working.

My current guess is that the RedisStorageProvider is mangling some values, but I didn't look really into it.

For easier testing I created a docker-compose file which can setup Synapse, Element, Redis and Hookshot: https://github.com/B4dM4n/hookkshot-e2ee-test

@Half-Shot Can you please have a look at the problem? I'm not too familiar with the Matrix API's / SDK's.

[Half-Shot]

Wow, thanks for looking into this. I'll be sure to take a look at your changes and see if I can get a working decryption going.

[B4dM4n]

I could track down the decryption issue to users not being tracked properly in the matrix-bot-sdk encryption handling: turt2live/matrix-bot-sdk#251

Up next is some general testing of the bot encryption.

[Half-Shot]

#417 lands support for bot-sdk 0.6.1 which means we can lean on that for some of this PR.

turt2live/matrix-bot-sdk#251 is gonna be a hard requirement though, given how hookshot is used.

[MatMaul]

It seems like SDK dependencies have landed so this should be mergeable ?

[AndrewFerr]

This works, and adds some docs to explain the extra configuration steps needed to get this working (namely Synapse experimental configs).

However, bridge startup sometimes fails with M_UNKNOWN_TOKEN, including when successfully setting up the bridge without encryption & updating configs to enable encryption. Either there are race conditions in startup / crypto registration, or I'm doing something wrong.

[AndrewFerr]

The M_UNKNOWN_TOKEN was a false positive, and was caused by fiddling with the homeserver DB & hookshot's encryption state (i.e. ./data/encryption and the redis instance). I added some brief notes to the docs to point out keeping them in sync.

The only remaining issue is that "greeting" messages sent by the bot may fail to be decryptable (and appear in Element Web with an error of OLM.UNKNOWN_MESSAGE_INDEX). I'll investigate whether this is an issue in the bot SDK, or is due to Hookshot sending messages earlier than it should.

[AndrewFerr]

Lastest push just rebases onto the tip of main to get the CI fixes.

[Half-Shot]

✔️ even though I can't review my own stuff!

[AndrewFerr]

Changes made by the latest force-push:

• Rebased onto current tip of the main branch.
• Replaced 2eb91ab with 9853b8c.
• Added all commits from 6b0e48b onwards.

Note that there remain some issues that can only be fixed in matrix-bot-sdk:

• Users who join an encrypted room that Hookshot is already in won't be able to decrypt Hookshot's future messages. Fixed by Get keys claim for all tracked users turt2live/matrix-bot-sdk#269.
• After encryption is enabled in a room, new Hookshot's messages that are sent before a user performs some action in that room won't be decryptable by that user. On track to be fixed by Query & claim needed keys before encrypting turt2live/matrix-bot-sdk#270.
• Encrypted messages sent by Hookshot after a user is invited to a room but before they join won't be decryptable. On track to be fixed by Handle pre-shared invite keys turt2live/matrix-bot-sdk#271.

[AndrewFerr]

Changes made by the latest (and hopefully last!) force-push:

• Once again rebased onto current tip of the main branch (putting this ahead of v2.5.0).
• Replaced 6bbaf1f with df5e4d8, which pulls in the new Element fork of the bot-sdk that includes the required fixes to crypto issues that I mentioned earlier: Add support for native e2ee #299 (comment)