Update MSC4039

proposals/4039-widget-api-media.md

@@ -1,28 +1,62 @@
-# MSC4039: Access the Content repository with the Widget API
-
-[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) specifies a Widget API that
-is able to receive events from and write events to the room that is loaded in the client. But beside
-the room events, homeservers also manage media content for the users. While widgets can already
-download media, given they acquired the URL of the homeserver, they can't upload new media into the
-room. This would be useful to post images or attachments, but also to upload larger files that can't
-be stored in a room event, like a whiteboard that can easily grow larger than 64kb.
-
-This proposal aims to bring the functionality of uploading media to the content repository into the
-widget specification. It should provide the same features that the
-[“Upload Content” endpoint](https://spec.matrix.org/v1.7/client-server-api/#post_matrixmediav3upload)
-of the Client-Server API provides. 
+# MSC4039: Access the Content Repository With the Widget API
+
+[MSC1236][MSC1236], [MSC2762][MSC2762], etc. specify a Widget API that is able to receive events from and write events
+to the room that is loaded in the client. Besides sending pure text events, it is possible to attach media and other
+files to events. The two concepts are consciously separated: Files are stored in a "[content repository][content]"
+(usually attached to the homeserver) and are addressable through a "[Matrix Content (`mxc://`) URI][mxc]". This URI can
+in turn be used in an event to reference a file.
+
+While prior to [MSC3916][MSC3916] and deprecation in Matrix v1.11, widgets could already download media via the
+unauthenticated download endpoint, provided they acquired the URL of the homeserver, the widget API does not currently
+contain a way to authenticate with the upload and new download endpoints nor does it otherwise expose these functions
+from the client hosting the widget to the widget itself. This would be useful to post images or attachments, but also to
+upload larger data that can't easily be stored in a room event, like a whiteboard's content that can easily grow larger
+than 64kb.
+
+This proposal aims to bring the functionality of reading and writing files from and to the content repository into the
+widget specification. Specifically, it should provide the equivalents of the ["Upload Content"][upload] and ["Download
+Content"][download] endpoints of the Client-Server API, while also having the client handle as much of the end-to-end
+encryption as possible.
 
 ## Proposal
 
-The widget API is extended with a new url parameter that can be requested and that will tell the
-widget the URL of the homeserver API the user is currently using:
-
-* `matrix_base_url`: The URL of the homeserver API.
-
-The widget API is extended with two new interfaces to upload files into the content repository. The
-user must manually approve the following capability before the actions can be used:
+The widget API is extended with three new interfaces to access the content repository. The
+user must manually approve the following capabilities before the actions can be used:
 
 - `m.upload_file`: Let the widget upload files.
+- `m.download_file`: Let the widget download files.
+
+Uploading to and downloading from the content repository is separated from the events referencing these files as
+explained above. This gives Matrix applications dealing with referenced files great flexibility, for example it would be
+possible to reference multiple files from a single event. It is also not an issue for clients to deal with end-to-end
+encrypted files, as they can easily reference the `key` and `iv` required for decryption through the
+[`EncryptedFile`][encryptedfile] metadata in the referencing event. [MSC2762][MSC2762] defines the client as responsible
+for handling encryption.
+
+As a consequence of the three factors separation, keeping to standard Matrix file encryption (i.e. `EncryptedFile`), and
+implementing the en/decryption in the client follows, that the widget must be responsible for attaching the
+`EncryptedFile` metadata to any event it sends. For otherwise already standardised events, this means following their
+[standardised scheme for encrypted variants][encryptedfile], but custom formats for custom events are possible, as long
+as the widget can provide the required data again when requesting decryption from the client. The `EncryptedFile` object
+used in the definitions below is a JSON object simply containing all the information required for encrypted
+attachments as already defined by the Matrix specification with their usual keys and values, i.e. for v1.11:
+
+```json
+{
+  "v": "v2",
+  "key": {
+    "alg": "A256CTR",
+    "ext": true,
+    "k": "aWF6-32KGYaC3A_FEUCk1Bt0JA37zP0wrStgmdCaW-0",
+    "key_ops": ["encrypt","decrypt"],
+    "kty": "oct"
+  },
+  "iv": "w+sE15fzSc0AAAAAAAAAAA",
+  "hashes": {
+    "sha256": "fdSLu/YkRx3Wyh3KQabP3rd6+SFiKg5lsJZQHtkSAYA"
+  }
+}
+```
 
 ### Get configuration
 
@@ -57,10 +91,33 @@ If the event is successfully sent by the client, the client sends the following
 }
 ```
 
-The `response` is a mirrored representation of the original `/_matrix/media/v3/config` API.
+The `response` is a mirrored representation of the original [content repository configuration][config] API.
 
 ### Upload file
 
+The following diagram shows the (optionally encrypted) upload file flow.
+
+```mermaid
+sequenceDiagram
+    participant W as widget
+    participant C as client
+    participant H as homeserver
+    W->>C: fromWidget upload_file
+    alt e2ee room
+    C->>C: encrypt file
+    end
+    C->>H: upload file
+    H->>C: mxc
+    C->>W: fromWidget upload_file (mxc, EncryptedFile)
+    W->>C: fromWidget send_event
+    alt e2ee room
+    C->>C: encrypt event
+    end
+    C->>H: send event
+    H->>C: event ID
+    C->>W: fromWidget send_event (event ID)
+```
+
 To trigger the action to upload a file, widgets will use a new `fromWidget` request with the action
 `upload_file` which takes the following shape:
 
@@ -81,9 +138,18 @@ To trigger the action to upload a file, widgets will use a new `fromWidget` requ
 If the widget did not get approved for the capability required to send the event, the client MUST
 send an error response (as required currently by the capabilities system for widgets).
 
-The client SHOULD NOT modify the data of the request.
+Matrix v1.7 added asynchronous uploads through [MSC2246][MSC2246] by splitting the interface of the media upload into a
+“create the mxc-id” and a “upload the file for the mxc-id” stage. It is left as an implementation detail for clients to
+decide how to apply this to upload requests issued by widgets in detail, but it SHOULD be used for large files whose
+upload can be expected to take longer than the widget API timeout. The widget API itself does not explicitly support
+something mirroring the upload direction of asynchronous uploads because the assumption is that the communication
+between widget and host client is instant.
 
-If the event is successfully sent by the client, the client sends the following response:
+#### Response (Unencrypted)
+
+The client SHOULD NOT modify the data of the request in unencrypted rooms.
+
+If the file is successfully uploaded by the client, the client sends the following response (mirroring the [Client-Server upload][upload] API):
 
 ```json
 {
@@ -100,30 +166,167 @@ If the event is successfully sent by the client, the client sends the following
 }
 ```
 
-## Potential issues
+#### Response (Encrypted)
+
+If the room is end-to-end encrypted, the client SHOULD encrypt the data before uploading, generating the usual required
+[`EncryptedFile`][encryptedfile] metadata. If sending is successful, the client adds the `EncryptedFile` to the
+response, resulting in:
+
+```json
+{
+  "api": "fromWidget",
+  "widgetId": "20200827_WidgetExample",
+  "requestid": "generated-id-1234",
+  "action": "upload_file",
+  "data": {
+    "file": "some-content"
+  },
+  "response": {
+    "content_uri": "mxc://...",
+    "encryption": EncryptedFile
+  }
+}
+```
+
+### Download file
+
+The following diagram shows the (optionally encrypted) download file flow.
+
+```mermaid
+sequenceDiagram
+    participant W as widget
+    participant C as client
+    participant H as homeserver
+    H->>C: sync receives event
+    alt e2ee room
+    C->>C: decrypt event
+    end
+    C->>W: toWidget send_event
+    W->>W: extract mxc
+    W->>W: look up EncryptedFile
+    W->>C: fromWidget download_file (mxc, EncryptedFile)
+    C->>H: download (mxc)
+    H->>C: file
+    alt EncryptedFile provided
+    C->>C: decrypt file
+    end
+    C->>W: fromWidget download_file
+```
+
+To trigger the action to download a file, widgets will use a new `fromWidget` request with the action `download_file`.
+
+#### Request (Unencrypted)
+
+```json
+{
+  "api": "fromWidget",
+  "widgetId": "20200827_WidgetExample",
+  "requestid": "generated-id-1234",
+  "action": "download_file",
+  "data": {
+    "content_uri": "mxc://...",
+    "timeout_ms": 20000
+  }
+}
+```
+
+Instead of mirroring the client-server API's fields `mediaId` and `serverName`, the `download_file` action's `data`
+field is structured to mirror the `upload_file` action both for consistency within the widget API and to leave parsing
+the `mxc` URI to the client which already implements it for simplicity.
+
+It is possible that the (default) timeouts for downloading introduced with asynchronous uploads and the widget API
+differ. Therefore, the widget MAY pass the `timeout_ms` parameter, whose default value follows and whose meaning mirrors
+that of the [same field in the client-server API][download] but also implies that the timeout in the widget API is
+adjusted accordingly for this single request. The widget is expected to be able to handle the
+[`M_NOT_YET_UPLOADED`][download] error response in the usual way the client forwards client-server-API errors to
+widgets.
+
+The client SHOULD NOT modify the received file from the content repository before responding to the widget in
+unencrypted rooms.
+
+#### Request (Encrypted)
+
+It is the widget's responsibility to know if the file is encrypted, e.g. by checking if the
+[`EncryptedFile`][encryptedfile] metadata is stored next to the `mxc:` URI. If so, it must be provided in the request to
+the client alongside it.
+
+```json
+{
+  "api": "fromWidget",
+  "widgetId": "20200827_WidgetExample",
+  "requestid": "generated-id-1234",
+  "action": "download_file",
+  "data": {
+    "content_uri": "mxc://...",
+    "timeout_ms": 20000,
+    "encryption": EncryptedFile
+  }
+}
+```
+
+#### Response
+
+```json
+{
+  "api": "fromWidget",
+  "widgetId": "20200827_WidgetExample",
+  "requestid": "generated-id-1234",
+  "action": "download_file",
+  "data": <see above>,
+  "response": {
+    "file": "some-content"
+  }
+}
+```
+
+`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a
+data type in the `postMessage` API.
+
+If the widget did not get approved for the capability required to send the event, the client MUST
+send an error response (as required currently by the capabilities system for widgets).
 
-[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) plans to add authentication
-for media access. This would require the Widget API to receive an additional extension to be able to
-access stored media. It is assumed that a successor of the current widget api
-(like https://github.com/matrix-org/matrix-spec-proposals/pull/3008) will replace this need in the
-future.
+## Potential issues
 
-[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) splits the interface of the
-media upload into a “create the mxc-id” and a “upload the file for the mxc-id” stage. It is assumed
-that the new API can be transparently implemented in the client so the widget could still use the old
-API definition.
+[MSC3911][MSC3911] proposes a way to link media (files) to events such that there is a 1:n relationship for event ->
+file. Particularly, each file must be attached to exactly one event, and copied using the proposed copy API to attach it
+again to another event. Since the client is unaware of event content sent by widgets per [MSC2762][MSC2762], it cannot
+automatically handle copying any referenced media and the burden is hence on the widget, which would require a change to
+the API described above to add the `copy` endpoint. Additionally, the event sending payload from MSC2762 will need to be
+extended such that whenever a widget includes references to files in an event, it is required to provide them to the
+client in the `send_event` request so the client knows to attach the files.
 
 ## Alternatives
 
-It would be preferable if the widget would not need to pass the actual data to the client over the
-Widget API, but if it could acquire an authenticated upload URL that it can use to upload the file
-directly to the homeserver without the need of an authentication token. 
-[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) goes into this direction,
-however, it still requires that the upload is authenticated.
+As mentioned above, [MSC2762][MSC2762] defines the client to handle all end-to-end encryption tasks. Handling the
+en/decryption steps directly in the widget would spare the back and forth of the encryption keys between widget and
+client, however there would still need to be multiple requests for uploading (authentication and to determine the `mxc`)
+and downloading (authentication) regardless.
+
+In principle it would be preferable to be able to entirely rely on the client hosting the widget to handle end-to-end
+encryption. It would be possible to design a different approach such as [MSC2881][MSC2881], where any event that wants
+to reference multiple files cannot do so directly but instead uses event relations to refer to them "by proxy" of
+another event that also keeps the `EncryptedFile` metadata, or [MSC3382][MSC3382], which simply allows multiple
+attachments to one single event. The issue of multiple requests between widget and client remains for uploads due to
+having to determine the `mxc`, which the widget needs to include in the (potentially custom) event body. For downloads
+it would be possible for the client to automatically download, decrypt, and pass the file to the widget, but it takes
+the control from the widget to determine which files are actually needed for the current view. We therefore don't see a
+great advantage in this approach. However, it might become more viable if [MSC3911][MSC3911] should land close to its
+current form.
+
+It may be preferable if the widget would not need to pass the actual data to the client over the
+Widget API, but instead acquire an authenticated URL that it can use to upload/download the file
+directly to/from the homeserver without the need of an authentication token.
+Matrix v1.7 asynchronous uploads ([MSC2246][MSC2246]) go in this direction,
+however they still require that the upload is authenticated.
+
+A completely new approach such as [MSC3008][MSC3008] where widgets can use the client-server API directly with a scoped
+access token would also be possible, but would touch on many parts of widget and client-server APIs including
+authentication and authorisation, which means it would require a complete revamp of the widget API and is hence out of
+scope of this MSC.
 
 ## Security considerations
 
-The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762)
+The same considerations as in [MSC2762][MSC2762]
 apply. This feature will allow the widget to be able to upload data into the media repository. This
 could potentially be used to upload malicious content. However, the access will only be possible
 when the user accepts the capability and grant access if the widget is trusted by the user.
@@ -133,9 +336,25 @@ when the user accepts the capability and grant access if the widget is trusted b
 While this MSC is not present in the spec, clients and widgets should:
 
 - Use `org.matrix.msc4039.` in place of `m.` in all new identifiers of this MSC.
-- Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter.
 - Use `org.matrix.msc4039.get_media_config` in place of `get_media_config` for the action type in the
   `fromWidget` requests.
 - Use `org.matrix.msc4039.upload_file` in place of `upload_file` for the action type in the
   `fromWidget` requests.
+- Use `org.matrix.msc4039.download_file` in place of `download_file` for the action type in the
+  `fromWidget` requests.
 - Only call/support the `action`s if an API version of `org.matrix.msc4039` is advertised.
+
+[MSC1236]: https://github.com/matrix-org/matrix-spec-proposals/pull/1236
+[MSC2246]: https://github.com/matrix-org/matrix-spec-proposals/pull/2246
+[MSC2762]: https://github.com/matrix-org/matrix-spec-proposals/pull/2762
+[MSC2881]: https://github.com/matrix-org/matrix-spec-proposals/pull/2881
+[MSC3008]: https://github.com/matrix-org/matrix-spec-proposals/pull/3008
+[MSC3382]: https://github.com/matrix-org/matrix-spec-proposals/pull/3382
+[MSC3911]: https://github.com/matrix-org/matrix-spec-proposals/pull/3911
+[MSC3916]: https://github.com/matrix-org/matrix-spec-proposals/pull/3916
+[content]: https://spec.matrix.org/v1.11/client-server-api/#content-repository
+[mxc]: https://spec.matrix.org/v1.11/client-server-api/#matrix-content-mxc-uris
+[config]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediaconfig
+[upload]: https://spec.matrix.org/v1.11/client-server-api/#post_matrixmediav3upload
+[download]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid
+[encryptedfile]: https://spec.matrix.org/v1.11/client-server-api/#extensions-to-mroommessage-msgtypes