From f346c77b8479c4e121b6f680b71e889ff360aa32 Mon Sep 17 00:00:00 2001 From: Dominik Henneke Date: Mon, 24 Jul 2023 19:58:58 +0200 Subject: [PATCH 01/17] Add an MSC for a new Widget API action to upload files into the media repository Signed-off-by: Dominik Henneke --- proposals/4039-widget-api-media.md | 142 +++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 proposals/4039-widget-api-media.md diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md new file mode 100644 index 00000000000..1cad9cebcbc --- /dev/null +++ b/proposals/4039-widget-api-media.md @@ -0,0 +1,142 @@ +# MSC4039: Access the Content repository with the Widget API + +[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) specifies a Widget API that +is able to receive events from and write events to the room that is loaded in the client. But beside +the room events, homeservers also manage media content for the users. While widgets can already +download media, given they acquired the URL of the homeserver, they can't upload new media into the +room. This would be useful to post images or attachments, but also to upload larger files that can't +be stored in a room event, like a whiteboard that can easily grow larger than 64kb. + +This proposal aims to bring the functionality of uploading media to the content repository into the +widget specification. It should provide the same features that the +[“Upload Content” endpoint](https://spec.matrix.org/v1.7/client-server-api/#post_matrixmediav3upload) +of the Client-Server API provides. + +## Proposal + +The widget API is extended with a new url parameter that can be requested and that will tell the +widget the URL of the homeserver API the user is currently using: + +* `matrix_base_url`: The URL of the homeserver API. + +The widget API is extended with a two new interface to upload files into the content repository. The +user must manually approve the following capability before the actions can be used: + +- `m.upload_file`: Let the widget access the user directory. + +### Get configuration + +To trigger the action to get the configuration, widgets will use a new `fromWidget` request with the +action `get_media_config` which takes the following shape: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "get_media_config", + "data": {} +} +``` + +If the widget did not get approved for the capability required to send the event, the client MUST +send an error response (as required currently by the capabilities system for widgets). + +If the event is successfully sent by the client, the client sends the following response: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "get_media_config", + "data": {}, + "response": { + "m.upload.size": 1000, + } +} +``` + +The `response` is are a mirrored representation of the original `/_matrix/media/v3/config` API. + +### Upload file + +To trigger the action to upload a file, widgets will use a new `fromWidget` request with the action +`upload_file` which takes the following shape: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "upload_file", + "data": { + "file": "some-content" + } +} +``` + +Under `data`, all keys are a mirrored representation of the original `/_matrix/client/v3/user_directory/search` +API. `file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API. + +If the widget did not get approved for the capability required to send the event, the client MUST +send an error response (as required currently by the capabilities system for widgets). + +The client SHOULD NOT modify the data of the request. + +If the event is successfully sent by the client, the client sends the following response: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "user_directory_search", + "data": { + "file": "some-content" + }, + "response": { + "content_uri": "mxc://..." + } +} +``` + +## Potential issues + +[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) plans to add authentication +for media access. This would require the Widget API to receive an additional extension to be able to +access stored media. It is assumed that a successor of the current widget api +(like https://github.com/matrix-org/matrix-spec-proposals/pull/3008) will replace this need in the +future. + +[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) splits the interface of the +media upload into a “create the mxc-id” and a “upload the file for the mxc-id” stage. It is assumed +that the new API can be transparently implemented in the client so the widget could still use the old +API definition. + +## Alternatives + +It would be preferable if the widget would not need to pass the actual data to the client over the +Widget API, but if it could acquire an authenticated upload URL that it can use to upload the file +directly to the homeserver without the need of an authentication token. +[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) goes into this direction, +however, it still requires that the upload is authenticated. + +## Security considerations + +The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) +apply. This feature will allow the widget to be able to receive upload data into the media repository. +This could potentially be used to upload malicious content. However, the access will only be possible +when the user accepts the capability and grant access if the widget is trusted by the user. + +## Unstable prefix + +While this MSC is not present in the spec, clients and widgets should: + +- Use `org.matrix.msc4039.` in place of `m.` in all new identifiers of this MSC. +- Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter. +- Use `org.matrix.msc4039.get_media_config` in place of `get_media_config` for the action type in the + `fromWidget` requests. +- Use `org.matrix.msc4039.upload_file` in place of `upload_file` for the action type in the + `fromWidget` requests. +- Only call/support the `action`s if an API version of `org.matrix.msc4039` is advertised. From 3e95735a098ca88d8c3e6252a15c1bb18b9aeac4 Mon Sep 17 00:00:00 2001 From: Dominik Henneke Date: Tue, 1 Aug 2023 09:26:43 +0200 Subject: [PATCH 02/17] Fix copy-paste errors Signed-off-by: Dominik Henneke --- proposals/4039-widget-api-media.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 1cad9cebcbc..338fdfac2bf 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -22,7 +22,7 @@ widget the URL of the homeserver API the user is currently using: The widget API is extended with a two new interface to upload files into the content repository. The user must manually approve the following capability before the actions can be used: -- `m.upload_file`: Let the widget access the user directory. +- `m.upload_file`: Let the widget upload files. ### Get configuration @@ -57,7 +57,7 @@ If the event is successfully sent by the client, the client sends the following } ``` -The `response` is are a mirrored representation of the original `/_matrix/media/v3/config` API. +The `response` is a mirrored representation of the original `/_matrix/media/v3/config` API. ### Upload file @@ -76,8 +76,7 @@ To trigger the action to upload a file, widgets will use a new `fromWidget` requ } ``` -Under `data`, all keys are a mirrored representation of the original `/_matrix/client/v3/user_directory/search` -API. `file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API. +`data.file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API. If the widget did not get approved for the capability required to send the event, the client MUST send an error response (as required currently by the capabilities system for widgets). @@ -91,7 +90,7 @@ If the event is successfully sent by the client, the client sends the following "api": "fromWidget", "widgetId": "20200827_WidgetExample", "requestid": "generated-id-1234", - "action": "user_directory_search", + "action": "upload_file", "data": { "file": "some-content" }, @@ -125,8 +124,8 @@ however, it still requires that the upload is authenticated. ## Security considerations The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) -apply. This feature will allow the widget to be able to receive upload data into the media repository. -This could potentially be used to upload malicious content. However, the access will only be possible +apply. This feature will allow the widget to be able to upload data into the media repository. This +could potentially be used to upload malicious content. However, the access will only be possible when the user accepts the capability and grant access if the widget is trusted by the user. ## Unstable prefix From 1adb2e3e70111be27f89cc7b7b499322f29c8ca5 Mon Sep 17 00:00:00 2001 From: Milton Moura Date: Thu, 27 Jun 2024 15:23:12 +0000 Subject: [PATCH 03/17] Fix new interface description Co-authored-by: Johannes Marbach --- proposals/4039-widget-api-media.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 338fdfac2bf..43e57af0125 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -19,7 +19,7 @@ widget the URL of the homeserver API the user is currently using: * `matrix_base_url`: The URL of the homeserver API. -The widget API is extended with a two new interface to upload files into the content repository. The +The widget API is extended with two new interfaces to upload files into the content repository. The user must manually approve the following capability before the actions can be used: - `m.upload_file`: Let the widget upload files. From 1fc1831a91795b91529a06fc5da0324a99da313f Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Wed, 7 Aug 2024 18:26:47 +0200 Subject: [PATCH 04/17] WIP: Update MSC4039 - added download API to handle MSC3916 "Authentication for media" - added e2ee aware variants for both upload and download Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 236 ++++++++++++++++++++++++----- 1 file changed, 199 insertions(+), 37 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 43e57af0125..a5f6a39c0f3 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -1,28 +1,53 @@ -# MSC4039: Access the Content repository with the Widget API - -[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) specifies a Widget API that -is able to receive events from and write events to the room that is loaded in the client. But beside -the room events, homeservers also manage media content for the users. While widgets can already -download media, given they acquired the URL of the homeserver, they can't upload new media into the -room. This would be useful to post images or attachments, but also to upload larger files that can't -be stored in a room event, like a whiteboard that can easily grow larger than 64kb. - -This proposal aims to bring the functionality of uploading media to the content repository into the -widget specification. It should provide the same features that the -[“Upload Content” endpoint](https://spec.matrix.org/v1.7/client-server-api/#post_matrixmediav3upload) -of the Client-Server API provides. +# MSC4039: Access the Content Repository With the Widget API + +[MSC1236][MSC1236], [MSC2762][MSC2762], etc. specify a Widget API that is able to receive events from and write events +to the room that is loaded in the client. Besides sending pure text events, it is possible to attach media and other +files to events. The two concepts are consciously separated: Files are stored in a "[content repository][content]" +(usually attached to the homeserver) and are addressable through a "[Matrix Content (`mxc://`) URI][mxc]". This URI can +in turn be used in an event to reference a file. + +While prior to [MSC3916][MSC3916] and deprecation in Matrix v1.11, widgets could already download media via the +unauthenticated download endpoint, provided they acquired the URL of the homeserver, the widget API does not currently +contain a way to authenticate with the upload and new download endpoints nor does it otherwise expose these functions +from the client hosting the widget to the widget itself. This would be useful to post images or attachments, but also to +upload larger data that can't easily be stored in a room event, like a whiteboard's content that can easily grow larger +than 64kb. + +This proposal aims to bring the functionality of reading and writing files from and to the content repository into the +widget specification. Specifically, it should provide the equivalents of the ["Upload Content"][upload] and ["Download +Content"][download] endpoints of the Client-Server API, while also having the client handle as much of the end-to-end +encryption as possible. ## Proposal +--- +TODO: remove? +TODO: remove from https://github.com/matrix-org/matrix-widget-api/blob/3c9543cbe4255869953d1b79d513de2eaf87033e/src/templating/url-template.ts#L48-L49 + The widget API is extended with a new url parameter that can be requested and that will tell the widget the URL of the homeserver API the user is currently using: * `matrix_base_url`: The URL of the homeserver API. +--- -The widget API is extended with two new interfaces to upload files into the content repository. The -user must manually approve the following capability before the actions can be used: +The widget API is extended with three new interfaces to access the content repository. The +user must manually approve the following capabilities before the actions can be used: - `m.upload_file`: Let the widget upload files. +- `m.download_file`: Let the widget download files. + +Uploading to and downloading from the content repository is separated from the events referencing these files as +explained above. This gives Matrix applications dealing with referenced files great flexibility, for example it would be +possible to reference multiple files from a single event. It is also not an issue for clients to deal with end-to-end +encrypted files, as they can easily reference the `key` and `iv` required for decryption through the +[`EncryptedFile`][encryptedfile] object in the referencing event. [MSC2762][MSC2762] defines the client as responsible +for handling encryption. + +As a consequence of the three factors separation, keeping to standard Matrix file encryption (i.e. storing +`EncryptedFile`), and implementing the en/decryption in the client follows, that the widget must be responsible for +attaching the `EncryptedFile` to any event it sends. For otherwise already standardised events, this means following +their [standardised scheme for encrypted variants][encryptedfile], but custom formats for custom events are possible, as +long as the widget can provide the required data again when requesting decryption from the client. ### Get configuration @@ -57,10 +82,24 @@ If the event is successfully sent by the client, the client sends the following } ``` -The `response` is a mirrored representation of the original `/_matrix/media/v3/config` API. +The `response` is a mirrored representation of the original [content repository configuration][config] API. ### Upload file +```mermaid +flowchart TD + A[Widget: upload] -->|fromWidget upload_file| B{Client: e2ee room?} + B -->|No| C[Client: upload file] + B -->|Yes| D[Client: encrypt file] + D -->|EncryptedFile| C + C -->|fromWidget upload_file| E[Widget: uploaded file] + E -->|fromWidget send_event| F{Client: e2ee room?} + F -->|No| G[Client: send event] + F -->|Yes| H[Client: encrypt event] + H --> G + G -->|fromWidget send_event| I[Widget: sent event] +``` + To trigger the action to upload a file, widgets will use a new `fromWidget` request with the action `upload_file` which takes the following shape: @@ -81,9 +120,13 @@ To trigger the action to upload a file, widgets will use a new `fromWidget` requ If the widget did not get approved for the capability required to send the event, the client MUST send an error response (as required currently by the capabilities system for widgets). -The client SHOULD NOT modify the data of the request. +TODO: error response if uploading fails? if encrypting fails? -If the event is successfully sent by the client, the client sends the following response: +#### Unencrypted + +The client SHOULD NOT modify the data of the request in unencrypted rooms. + +If the file is successfully uploaded by the client, the client sends the following response (mirroring the [Client-Server upload][upload] API): ```json { @@ -100,42 +143,161 @@ If the event is successfully sent by the client, the client sends the following } ``` +#### Encrypted + +If the room is end-to-end encrypted, the client SHOULD encrypt the data before uploading, generating the usual required +[`EncryptedFile`][encryptedfile] metadata. If sending is successful, the client adds the `EncryptedFile` to the +response, resulting in: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "upload_file", + "data": { + "file": "some-content" + }, + "response": { + "content_uri": "mxc://...", + "encryption": EncryptedFile + } +} +``` + +### Download file + +```mermaid +flowchart TD + N[Client: receive event] --> O{Client: e2ee event?} + O -->|No| P[Client: received event] + O -->|Yes| Q[Client: decrypt event] + Q --> P + P -->|toWidget send_event| R[Widget: received event] + R -->|fromWidget download_file| S[Client: download file] + S --> T{Client: widget sent EncryptedFile} + T -->|No| U[Client: downloaded file] + T -->|Yes| V[Client: decrypt file] + V --> U + U -->|toWidget download_file| W[Widget: downloaded file] +``` + +To trigger the action to download a file, widgets will use a new `fromWidget` request with the action `download_file`. + +#### Unencrypted (Request) + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "download_file", + "data": { + "content_uri": "mxc://..." + } +} +``` + +The client SHOULD NOT modify the received file from the content repository before responding to the widget in +unencrypted rooms. + +#### Encrypted (Request) + +It is the widget's responsibility to know if the file is encrypted, e.g. by checking if the +[`EncryptedFile`][encryptedfile] metadata is stored next to the `mxc:` URI. If so, it must be provided in the request to +the client alongside it. + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "download_file", + "data": { + "content_uri": "mxc://...", + "encryption": EncryptedFile + } +} +``` + +#### Common Response + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestid": "generated-id-1234", + "action": "download_file", + "data": , + "response": { + "file": "some-content" + } +} +``` + +`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API. + +If the widget did not get approved for the capability required to send the event, the client MUST +send an error response (as required currently by the capabilities system for widgets). + +TODO: error response if downloading fails? if decrypting fails? + ## Potential issues -[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) plans to add authentication -for media access. This would require the Widget API to receive an additional extension to be able to -access stored media. It is assumed that a successor of the current widget api -(like https://github.com/matrix-org/matrix-spec-proposals/pull/3008) will replace this need in the -future. +[MSC3916][MSC3916] plans to add authentication for media access. This would require the Widget API to receive an +additional extension to be able to access stored media. It is assumed that a successor of the current widget api (like +[MSC3008]) will replace this need in the future. -[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) splits the interface of the -media upload into a “create the mxc-id” and a “upload the file for the mxc-id” stage. It is assumed -that the new API can be transparently implemented in the client so the widget could still use the old -API definition. +[MSC2246][MSC2246] splits the interface of the media upload into a “create the mxc-id” and a “upload the file for the +mxc-id” stage. It is assumed that the new API can be transparently implemented in the client so the widget could still +use the old API definition. + +TODO: [MSC3911][MSC3911] linking media to events ## Alternatives -It would be preferable if the widget would not need to pass the actual data to the client over the -Widget API, but if it could acquire an authenticated upload URL that it can use to upload the file -directly to the homeserver without the need of an authentication token. -[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) goes into this direction, -however, it still requires that the upload is authenticated. +It would be preferable if the widget would not need to pass the actual data to the client over the Widget API, but if it +could acquire an authenticated upload URL that it can use to upload the file directly to the homeserver without the need +of an authentication token. [MSC2246][MSC2246] goes into this direction, however, it still requires that the upload is +authenticated. + +TODO: "proxy" the mxcs and have the client bookkeep the e2ee data? + +TODO: e2ee in the widget, contrary to [MSC2762][MSC2762] + +TODO: follow the C-S download API format for download_file. but our way alleviates the parsing from the widget to the +client which already can parse them. ## Security considerations -The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) -apply. This feature will allow the widget to be able to upload data into the media repository. This -could potentially be used to upload malicious content. However, the access will only be possible -when the user accepts the capability and grant access if the widget is trusted by the user. +The same considerations as in [MSC2762][MSC2762] apply. This feature will allow the widget to be able to upload data +into the media repository. This could potentially be used to upload malicious content. However, the access will only be +possible when the user accepts the capability and grant access if the widget is trusted by the user. ## Unstable prefix While this MSC is not present in the spec, clients and widgets should: - Use `org.matrix.msc4039.` in place of `m.` in all new identifiers of this MSC. -- Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter. +- TODO: remove? Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter. - Use `org.matrix.msc4039.get_media_config` in place of `get_media_config` for the action type in the `fromWidget` requests. - Use `org.matrix.msc4039.upload_file` in place of `upload_file` for the action type in the `fromWidget` requests. +- Use `org.matrix.msc4039.download_file` in place of `download_file` for the action type in the + `fromWidget` requests. - Only call/support the `action`s if an API version of `org.matrix.msc4039` is advertised. + +[MSC1236]: https://github.com/matrix-org/matrix-spec-proposals/pull/1236 +[MSC2246]: https://github.com/matrix-org/matrix-spec-proposals/pull/2246 +[MSC2762]: https://github.com/matrix-org/matrix-spec-proposals/pull/2762 +[MSC3911]: https://github.com/matrix-org/matrix-spec-proposals/pull/3911 +[MSC3916]: https://github.com/matrix-org/matrix-spec-proposals/pull/3916 +[content]: https://spec.matrix.org/v1.11/client-server-api/#content-repository +[mxc]: https://spec.matrix.org/v1.11/client-server-api/#matrix-content-mxc-uris +[config]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediaconfig +[upload]: https://spec.matrix.org/v1.11/client-server-api/#post_matrixmediav3upload +[download]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid +[encryptedfile]: https://spec.matrix.org/v1.11/client-server-api/#extensions-to-mroommessage-msgtypes + +TODO: review wrt. actual current implementation From c1c8f6400613888fc6c2b0c9bcd9b697bbc11da6 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Thu, 8 Aug 2024 16:18:53 +0200 Subject: [PATCH 05/17] define EncryptedFile object Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 31 ++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index a5f6a39c0f3..7f0d5138dd8 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -40,14 +40,33 @@ Uploading to and downloading from the content repository is separated from the e explained above. This gives Matrix applications dealing with referenced files great flexibility, for example it would be possible to reference multiple files from a single event. It is also not an issue for clients to deal with end-to-end encrypted files, as they can easily reference the `key` and `iv` required for decryption through the -[`EncryptedFile`][encryptedfile] object in the referencing event. [MSC2762][MSC2762] defines the client as responsible +[`EncryptedFile`][encryptedfile] metadata in the referencing event. [MSC2762][MSC2762] defines the client as responsible for handling encryption. -As a consequence of the three factors separation, keeping to standard Matrix file encryption (i.e. storing -`EncryptedFile`), and implementing the en/decryption in the client follows, that the widget must be responsible for -attaching the `EncryptedFile` to any event it sends. For otherwise already standardised events, this means following -their [standardised scheme for encrypted variants][encryptedfile], but custom formats for custom events are possible, as -long as the widget can provide the required data again when requesting decryption from the client. +As a consequence of the three factors separation, keeping to standard Matrix file encryption (i.e. `EncryptedFile`), and +implementing the en/decryption in the client follows, that the widget must be responsible for attaching the +`EncryptedFile` metadata to any event it sends. For otherwise already standardised events, this means following their +[standardised scheme for encrypted variants][encryptedfile], but custom formats for custom events are possible, as long +as the widget can provide the required data again when requesting decryption from the client. The `EncryptedFile` object +used in the definitions below is a JSON object simply containing all the information required for encrypted +attachments as already defined by the Matrix specification with their usual keys and values, i.e. for v1.11: + +```json +{ + "v": "v2", + "key": { + "alg": "A256CTR", + "ext": true, + "k": "aWF6-32KGYaC3A_FEUCk1Bt0JA37zP0wrStgmdCaW-0", + "key_ops": ["encrypt","decrypt"], + "kty": "oct" + }, + "iv": "w+sE15fzSc0AAAAAAAAAAA", + "hashes": { + "sha256": "fdSLu/YkRx3Wyh3KQabP3rd6+SFiKg5lsJZQHtkSAYA" + } +} +``` ### Get configuration From 1947d2d4ad5399a1e860ab249774bce461396ae5 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:28:30 +0200 Subject: [PATCH 06/17] restructure headings Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 7f0d5138dd8..b2cd485bd8d 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -141,7 +141,7 @@ send an error response (as required currently by the capabilities system for wid TODO: error response if uploading fails? if encrypting fails? -#### Unencrypted +#### Response (Unencrypted) The client SHOULD NOT modify the data of the request in unencrypted rooms. @@ -162,7 +162,7 @@ If the file is successfully uploaded by the client, the client sends the followi } ``` -#### Encrypted +#### Response (Encrypted) If the room is end-to-end encrypted, the client SHOULD encrypt the data before uploading, generating the usual required [`EncryptedFile`][encryptedfile] metadata. If sending is successful, the client adds the `EncryptedFile` to the @@ -203,7 +203,7 @@ flowchart TD To trigger the action to download a file, widgets will use a new `fromWidget` request with the action `download_file`. -#### Unencrypted (Request) +#### Request (Unencrypted) ```json { @@ -220,7 +220,7 @@ To trigger the action to download a file, widgets will use a new `fromWidget` re The client SHOULD NOT modify the received file from the content repository before responding to the widget in unencrypted rooms. -#### Encrypted (Request) +#### Request (Encrypted) It is the widget's responsibility to know if the file is encrypted, e.g. by checking if the [`EncryptedFile`][encryptedfile] metadata is stored next to the `mxc:` URI. If so, it must be provided in the request to @@ -239,7 +239,7 @@ the client alongside it. } ``` -#### Common Response +#### Response ```json { From 26b6a7bc9396c41f667f7f2bf6dffba2b7b56eed Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:29:52 +0200 Subject: [PATCH 07/17] address the "e2ee in widget" alternative Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index b2cd485bd8d..60f67c1b6d9 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -282,7 +282,10 @@ authenticated. TODO: "proxy" the mxcs and have the client bookkeep the e2ee data? -TODO: e2ee in the widget, contrary to [MSC2762][MSC2762] +As mentioned above, [MSC2762][MSC2762] defines the client to handle all end-to-end encryption tasks. Handling the +en/decryption steps directly in the widget would spare the back and forth of the encryption keys between widget and +client, however there would still need to be multiple requests for uploading (authentication and to determine the `mxc`) +and downloading (authentication) regardless. TODO: follow the C-S download API format for download_file. but our way alleviates the parsing from the widget to the client which already can parse them. From 92020518cb91efec6bb3d6be6d60d932a9d77da6 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:32:12 +0200 Subject: [PATCH 08/17] address "async uploads" Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 60f67c1b6d9..5cc839b70c7 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -139,7 +139,12 @@ To trigger the action to upload a file, widgets will use a new `fromWidget` requ If the widget did not get approved for the capability required to send the event, the client MUST send an error response (as required currently by the capabilities system for widgets). -TODO: error response if uploading fails? if encrypting fails? +Matrix v1.7 added asynchronous uploads through [MSC2246][MSC2246] by splitting the interface of the media upload into a +“create the mxc-id” and a “upload the file for the mxc-id” stage. It is left as an implementation detail for clients to +decide how to apply this to upload requests issued by widgets in detail, but it SHOULD be used for large files whose +upload can be expected to take longer than the widget API timeout. The widget API itself does not explicitly support +something mirroring the upload direction of asynchronous uploads because the assumption is that the communication +between widget and host client is instant. #### Response (Unencrypted) @@ -212,11 +217,19 @@ To trigger the action to download a file, widgets will use a new `fromWidget` re "requestid": "generated-id-1234", "action": "download_file", "data": { - "content_uri": "mxc://..." + "content_uri": "mxc://...", + "timeout_ms": 20000 } } ``` +It is possible that the (default) timeouts for downloading introduced with asynchronous uploads and the widget API +differ. Therefore, the widget MAY pass the `timeout_ms` parameter, whose default value follows and whose meaning mirrors +that of the [same field in the client-server API][download] but also implies that the timeout in the widget API is +adjusted accordingly for this single request. The widget is expected to be able to handle the +[`M_NOT_YET_UPLOADED`][download] error response in the usual way the client forwards client-server-API errors to +widgets. + The client SHOULD NOT modify the received file from the content repository before responding to the widget in unencrypted rooms. @@ -234,6 +247,7 @@ the client alongside it. "action": "download_file", "data": { "content_uri": "mxc://...", + "timeout_ms": 20000, "encryption": EncryptedFile } } @@ -267,10 +281,6 @@ TODO: error response if downloading fails? if decrypting fails? additional extension to be able to access stored media. It is assumed that a successor of the current widget api (like [MSC3008]) will replace this need in the future. -[MSC2246][MSC2246] splits the interface of the media upload into a “create the mxc-id” and a “upload the file for the -mxc-id” stage. It is assumed that the new API can be transparently implemented in the client so the widget could still -use the old API definition. - TODO: [MSC3911][MSC3911] linking media to events ## Alternatives From 4627fc052c3a1b95ba53fb37c442cdff27922e71 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:33:35 +0200 Subject: [PATCH 09/17] explain the choice of payload design Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 5cc839b70c7..ae6a6ef6dd2 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -223,6 +223,10 @@ To trigger the action to download a file, widgets will use a new `fromWidget` re } ``` +Instead of mirroring the client-server API's fields `mediaId` and `serverName`, the `download_file` action's `data` +field is structured to mirror the `upload_file` action both for consistency within the widget API and to leave parsing +the `mxc` URI to the client which already implements it for simplicity. + It is possible that the (default) timeouts for downloading introduced with asynchronous uploads and the widget API differ. Therefore, the widget MAY pass the `timeout_ms` parameter, whose default value follows and whose meaning mirrors that of the [same field in the client-server API][download] but also implies that the timeout in the widget API is @@ -297,8 +301,6 @@ en/decryption steps directly in the widget would spare the back and forth of the client, however there would still need to be multiple requests for uploading (authentication and to determine the `mxc`) and downloading (authentication) regardless. -TODO: follow the C-S download API format for download_file. but our way alleviates the parsing from the widget to the -client which already can parse them. ## Security considerations From 960ebc5084fa2c5851972d7fb79727c7fe59d817 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:34:46 +0200 Subject: [PATCH 10/17] remove authenticated media issue which has been addressed, reformat Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index ae6a6ef6dd2..775ec5d91f1 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -272,19 +272,14 @@ the client alongside it. } ``` -`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API. +`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a +data type in the `postMessage` API. If the widget did not get approved for the capability required to send the event, the client MUST send an error response (as required currently by the capabilities system for widgets). -TODO: error response if downloading fails? if decrypting fails? - ## Potential issues -[MSC3916][MSC3916] plans to add authentication for media access. This would require the Widget API to receive an -additional extension to be able to access stored media. It is assumed that a successor of the current widget api (like -[MSC3008]) will replace this need in the future. - TODO: [MSC3911][MSC3911] linking media to events ## Alternatives From 59386f37b4c133314783215bf2a73e17ef40cc02 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Mon, 12 Aug 2024 17:35:21 +0200 Subject: [PATCH 11/17] add "widget api 2.0" alternative Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 775ec5d91f1..d1a60add138 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -296,6 +296,9 @@ en/decryption steps directly in the widget would spare the back and forth of the client, however there would still need to be multiple requests for uploading (authentication and to determine the `mxc`) and downloading (authentication) regardless. +A completely new approach where widgets can use the client-server API directly with a scoped access token would also be +possible, but would touch on many parts of widget and client-server APIs including authentication and authorisation, +which means it would require a complete revamp of the widget API and is hence out of scope of this MSC. ## Security considerations From 09a62602e32269561b1e6a870db727a3bc4889ee Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Tue, 13 Aug 2024 16:53:34 +0200 Subject: [PATCH 12/17] remove matrix_base_url Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index d1a60add138..59c1abcd8c9 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -20,16 +20,6 @@ encryption as possible. ## Proposal ---- -TODO: remove? -TODO: remove from https://github.com/matrix-org/matrix-widget-api/blob/3c9543cbe4255869953d1b79d513de2eaf87033e/src/templating/url-template.ts#L48-L49 - -The widget API is extended with a new url parameter that can be requested and that will tell the -widget the URL of the homeserver API the user is currently using: - -* `matrix_base_url`: The URL of the homeserver API. ---- - The widget API is extended with three new interfaces to access the content repository. The user must manually approve the following capabilities before the actions can be used: @@ -311,7 +301,6 @@ possible when the user accepts the capability and grant access if the widget is While this MSC is not present in the spec, clients and widgets should: - Use `org.matrix.msc4039.` in place of `m.` in all new identifiers of this MSC. -- TODO: remove? Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter. - Use `org.matrix.msc4039.get_media_config` in place of `get_media_config` for the action type in the `fromWidget` requests. - Use `org.matrix.msc4039.upload_file` in place of `upload_file` for the action type in the From 58f899ee7ee2fe5047d63f673df9fea75b4e0ff3 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Tue, 13 Aug 2024 16:58:27 +0200 Subject: [PATCH 13/17] format diff Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 59c1abcd8c9..0999ae4cd83 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -274,11 +274,6 @@ TODO: [MSC3911][MSC3911] linking media to events ## Alternatives -It would be preferable if the widget would not need to pass the actual data to the client over the Widget API, but if it -could acquire an authenticated upload URL that it can use to upload the file directly to the homeserver without the need -of an authentication token. [MSC2246][MSC2246] goes into this direction, however, it still requires that the upload is -authenticated. - TODO: "proxy" the mxcs and have the client bookkeep the e2ee data? As mentioned above, [MSC2762][MSC2762] defines the client to handle all end-to-end encryption tasks. Handling the @@ -289,12 +284,19 @@ and downloading (authentication) regardless. A completely new approach where widgets can use the client-server API directly with a scoped access token would also be possible, but would touch on many parts of widget and client-server APIs including authentication and authorisation, which means it would require a complete revamp of the widget API and is hence out of scope of this MSC. +It may be preferable if the widget would not need to pass the actual data to the client over the +Widget API, but instead acquire an authenticated URL that it can use to upload/download the file +directly to/from the homeserver without the need of an authentication token. +Matrix v1.7 asynchronous uploads ([MSC2246][MSC2246]) go in this direction, +however they still require that the upload is authenticated. + ## Security considerations -The same considerations as in [MSC2762][MSC2762] apply. This feature will allow the widget to be able to upload data -into the media repository. This could potentially be used to upload malicious content. However, the access will only be -possible when the user accepts the capability and grant access if the widget is trusted by the user. +The same considerations as in [MSC2762][MSC2762] +apply. This feature will allow the widget to be able to upload data into the media repository. This +could potentially be used to upload malicious content. However, the access will only be possible +when the user accepts the capability and grant access if the widget is trusted by the user. ## Unstable prefix From a5e8065572f2daf18ec558a998cfe9c0ee7809da Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Tue, 13 Aug 2024 16:58:52 +0200 Subject: [PATCH 14/17] update alternatives Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 0999ae4cd83..4333967a48e 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -274,22 +274,31 @@ TODO: [MSC3911][MSC3911] linking media to events ## Alternatives -TODO: "proxy" the mxcs and have the client bookkeep the e2ee data? - As mentioned above, [MSC2762][MSC2762] defines the client to handle all end-to-end encryption tasks. Handling the en/decryption steps directly in the widget would spare the back and forth of the encryption keys between widget and client, however there would still need to be multiple requests for uploading (authentication and to determine the `mxc`) and downloading (authentication) regardless. -A completely new approach where widgets can use the client-server API directly with a scoped access token would also be -possible, but would touch on many parts of widget and client-server APIs including authentication and authorisation, -which means it would require a complete revamp of the widget API and is hence out of scope of this MSC. +In principle it would be preferable to be able to entirely rely on the client hosting the widget to handle end-to-end +encryption. It would be possible to design a different approach such as [MSC2881][MSC2881], where any event that wants +to reference multiple files cannot do so directly but instead uses event relations to refer to them "by proxy" of +another event that also keeps the `EncryptedFile` metadata, or [MSC3382][MSC3382], which simply allows multiple +attachments to one single event. The issue of multiple requests between widget and client remains for uploads due to +having to determine the `mxc`, which the widget needs to include in the (potentially custom) event body. For downloads +it would be possible for the client to automatically download, decrypt, and pass the file to the widget, but it takes +the control from the widget to determine which files are actually needed for the current view. We therefore don't see a +great advantage in this approach. + It may be preferable if the widget would not need to pass the actual data to the client over the Widget API, but instead acquire an authenticated URL that it can use to upload/download the file directly to/from the homeserver without the need of an authentication token. Matrix v1.7 asynchronous uploads ([MSC2246][MSC2246]) go in this direction, however they still require that the upload is authenticated. +A completely new approach such as [MSC3008][MSC3008] where widgets can use the client-server API directly with a scoped +access token would also be possible, but would touch on many parts of widget and client-server APIs including +authentication and authorisation, which means it would require a complete revamp of the widget API and is hence out of +scope of this MSC. ## Security considerations @@ -314,6 +323,9 @@ While this MSC is not present in the spec, clients and widgets should: [MSC1236]: https://github.com/matrix-org/matrix-spec-proposals/pull/1236 [MSC2246]: https://github.com/matrix-org/matrix-spec-proposals/pull/2246 [MSC2762]: https://github.com/matrix-org/matrix-spec-proposals/pull/2762 +[MSC2881]: https://github.com/matrix-org/matrix-spec-proposals/pull/2881 +[MSC3008]: https://github.com/matrix-org/matrix-spec-proposals/pull/3008 +[MSC3382]: https://github.com/matrix-org/matrix-spec-proposals/pull/3382 [MSC3911]: https://github.com/matrix-org/matrix-spec-proposals/pull/3911 [MSC3916]: https://github.com/matrix-org/matrix-spec-proposals/pull/3916 [content]: https://spec.matrix.org/v1.11/client-server-api/#content-repository From 569c85478fa06e74e8d89f59cc8d6441980d20d5 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Tue, 13 Aug 2024 18:26:32 +0200 Subject: [PATCH 15/17] consider MSC3911 Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 4333967a48e..f07dbaf6ca4 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -270,7 +270,13 @@ send an error response (as required currently by the capabilities system for wid ## Potential issues -TODO: [MSC3911][MSC3911] linking media to events +[MSC3911][MSC3911] proposes a way to link media (files) to events such that there is a 1:n relationship for event -> +file. Particularly, each file must be attached to exactly one event, and copied using the proposed copy API to attach it +again to another event. Since the client is unaware of event content sent by widgets per [MSC2762][MSC2762], it cannot +automatically handle copying any referenced media and the burden is hence on the widget, which would require a change to +the API described above to add the `copy` endpoint. Additionally, the event sending payload from MSC2762 will need to be +extended such that whenever a widget includes references to files in an event, it is required to provide them to the +client in the `send_event` request so the client knows to attach the files. ## Alternatives @@ -287,7 +293,8 @@ attachments to one single event. The issue of multiple requests between widget a having to determine the `mxc`, which the widget needs to include in the (potentially custom) event body. For downloads it would be possible for the client to automatically download, decrypt, and pass the file to the widget, but it takes the control from the widget to determine which files are actually needed for the current view. We therefore don't see a -great advantage in this approach. +great advantage in this approach. However, it might become more viable if [MSC3911][MSC3911] should land close to its +current form. It may be preferable if the widget would not need to pass the actual data to the client over the Widget API, but instead acquire an authenticated URL that it can use to upload/download the file From ff1692c15a08a7de120c2a1ed4bc1663ddc43bf7 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Tue, 13 Aug 2024 18:27:34 +0200 Subject: [PATCH 16/17] remove all TODOs Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index f07dbaf6ca4..3240492908c 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -341,5 +341,3 @@ While this MSC is not present in the spec, clients and widgets should: [upload]: https://spec.matrix.org/v1.11/client-server-api/#post_matrixmediav3upload [download]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid [encryptedfile]: https://spec.matrix.org/v1.11/client-server-api/#extensions-to-mroommessage-msgtypes - -TODO: review wrt. actual current implementation From 514e8beb698640d6b0072a8ebc7b01a1afd2be91 Mon Sep 17 00:00:00 2001 From: Kim Brose Date: Wed, 14 Aug 2024 16:12:49 +0200 Subject: [PATCH 17/17] replace flowchart with sequenceDiagram Signed-off-by: Kim Brose --- proposals/4039-widget-api-media.md | 63 +++++++++++++++++++----------- 1 file changed, 40 insertions(+), 23 deletions(-) diff --git a/proposals/4039-widget-api-media.md b/proposals/4039-widget-api-media.md index 3240492908c..318c366bd34 100644 --- a/proposals/4039-widget-api-media.md +++ b/proposals/4039-widget-api-media.md @@ -95,18 +95,27 @@ The `response` is a mirrored representation of the original [content repository ### Upload file +The following diagram shows the (optionally encrypted) upload file flow. + ```mermaid -flowchart TD - A[Widget: upload] -->|fromWidget upload_file| B{Client: e2ee room?} - B -->|No| C[Client: upload file] - B -->|Yes| D[Client: encrypt file] - D -->|EncryptedFile| C - C -->|fromWidget upload_file| E[Widget: uploaded file] - E -->|fromWidget send_event| F{Client: e2ee room?} - F -->|No| G[Client: send event] - F -->|Yes| H[Client: encrypt event] - H --> G - G -->|fromWidget send_event| I[Widget: sent event] +sequenceDiagram + participant W as widget + participant C as client + participant H as homeserver + W->>C: fromWidget upload_file + alt e2ee room + C->>C: encrypt file + end + C->>H: upload file + H->>C: mxc + C->>W: fromWidget upload_file (mxc, EncryptedFile) + W->>C: fromWidget send_event + alt e2ee room + C->>C: encrypt event + end + C->>H: send event + H->>C: event ID + C->>W: fromWidget send_event (event ID) ``` To trigger the action to upload a file, widgets will use a new `fromWidget` request with the action @@ -181,19 +190,27 @@ response, resulting in: ### Download file +The following diagram shows the (optionally encrypted) download file flow. + ```mermaid -flowchart TD - N[Client: receive event] --> O{Client: e2ee event?} - O -->|No| P[Client: received event] - O -->|Yes| Q[Client: decrypt event] - Q --> P - P -->|toWidget send_event| R[Widget: received event] - R -->|fromWidget download_file| S[Client: download file] - S --> T{Client: widget sent EncryptedFile} - T -->|No| U[Client: downloaded file] - T -->|Yes| V[Client: decrypt file] - V --> U - U -->|toWidget download_file| W[Widget: downloaded file] +sequenceDiagram + participant W as widget + participant C as client + participant H as homeserver + H->>C: sync receives event + alt e2ee room + C->>C: decrypt event + end + C->>W: toWidget send_event + W->>W: extract mxc + W->>W: look up EncryptedFile + W->>C: fromWidget download_file (mxc, EncryptedFile) + C->>H: download (mxc) + H->>C: file + alt EncryptedFile provided + C->>C: decrypt file + end + C->>W: fromWidget download_file ``` To trigger the action to download a file, widgets will use a new `fromWidget` request with the action `download_file`.