Skip to content

Commit

Permalink
WIP: Update MSC4039
Browse files Browse the repository at this point in the history 
- added download API to handle MSC3916 "Authentication for media"
- added e2ee aware variants for both upload and download

Signed-off-by: Kim Brose <kim.brose@nordeck.net>
  • Loading branch information
@HarHarLinks
HarHarLinks committed Aug 7, 2024
1 parent 1adb2e3 commit 096ef1c
Showing 1 changed file with 195 additions and 37 deletions.
232 changes: 195 additions & 37 deletions proposals/4039-widget-api-media.md
View file
Original file line number Diff line number Diff line change
@@ -1,28 +1,53 @@
# MSC4039: Access the Content repository with the Widget API

[MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762) specifies a Widget API that
is able to receive events from and write events to the room that is loaded in the client. But beside
the room events, homeservers also manage media content for the users. While widgets can already
download media, given they acquired the URL of the homeserver, they can't upload new media into the
room. This would be useful to post images or attachments, but also to upload larger files that can't
be stored in a room event, like a whiteboard that can easily grow larger than 64kb.

This proposal aims to bring the functionality of uploading media to the content repository into the
widget specification. It should provide the same features that the
[“Upload Content” endpoint](https://spec.matrix.org/v1.7/client-server-api/#post_matrixmediav3upload)
of the Client-Server API provides.
# MSC4039: Access the Content Repository With the Widget API

[MSC1236][MSC1236], [MSC2762][MSC2762], etc. specify a Widget API that is able to receive events from and write events
to the room that is loaded in the client. Besides sending pure text events, it is possible to attach media and other
files to events. The two concepts are consciously separated: Files are stored in a "[content repository][content]"
(usually attached to the homeserver) and are addressable through a "[Matrix Content (`mxc://`) URI][mxc]". This URI can
in turn be used in an event to reference a file.

While prior to [MSC3916][MSC3916] and deprecation in Matrix v1.11, widgets could already download media via the
unauthenticated download endpoint, provided they acquired the URL of the homeserver, the widget API does not currently
contain a way to authenticate with the upload and new download endpoints nor does it otherwise expose these functions
from the client hosting the widget to the widget itself. This would be useful to post images or attachments, but also to
upload larger data that can't easily be stored in a room event, like a whiteboard's content that can easily grow larger
than 64kb.

This proposal aims to bring the functionality of reading and writing files from and to the content repository into the
widget specification. Specifically, it should provide the equivalents of the ["Upload Content"][upload] and ["Download
Content"][download] endpoints of the Client-Server API, while also having the client handle as much of the end-to-end
encryption as possible.

## Proposal

---
TODO: remove?
TODO: remove from https://github.com/matrix-org/matrix-widget-api/blob/3c9543cbe4255869953d1b79d513de2eaf87033e/src/templating/url-template.ts#L48-L49

The widget API is extended with a new url parameter that can be requested and that will tell the
widget the URL of the homeserver API the user is currently using:

* `matrix_base_url`: The URL of the homeserver API.
---

The widget API is extended with two new interfaces to upload files into the content repository. The
user must manually approve the following capability before the actions can be used:
The widget API is extended with three new interfaces to access the content repository. The
user must manually approve the following capabilities before the actions can be used:

- `m.upload_file`: Let the widget upload files.
- `m.download_file`: Let the widget download files.

Uploading to and downloading from the content repository is separated from the events referencing these files as
explained above. This gives Matrix applications dealing with referenced files great flexibility, for example it would be
possible to reference multiple files from a single event. It is also not an issue for clients to deal with end-to-end
encrypted files, as they can easily reference the `key` and `iv` required for decryption through the
[`EncryptedFile`][encryptedfile] object in the referencing event. [MSC2762][MSC2762] defines the client as responsible
for handling encryption.

As a consequence of the three factors separation, keeping to standard Matrix file encryption (i.e. storing
`EncryptedFile`), and implementing the en/decryption in the client follows, that the widget must be responsible for
attaching the `EncryptedFile` to any event it sends. For otherwise already standardised events, this means following
their [standardised scheme for encrypted variants][encryptedfile], but custom formats for custom events are possible, as
long as the widget can provide the required data again when requesting decryption from the client.

### Get configuration

Expand Down Expand Up @@ -57,10 +82,24 @@ If the event is successfully sent by the client, the client sends the following
}
```

The `response` is a mirrored representation of the original `/_matrix/media/v3/config` API.
The `response` is a mirrored representation of the original [content repository configuration][config] API.

### Upload file

```mermaid
flowchart TD
A[Widget: upload] -->|fromWidget upload_file| B{Client: e2ee room?}
B -->|No| C[Client: upload file]
B -->|Yes| D[Client: encrypt file]
D -->|EncryptedFile| C
C -->|fromWidget upload_file| E[Widget: uploaded file]
E -->|fromWidget send_event| F{Client: e2ee room?}
F -->|No| G[Client: send event]
F -->|Yes| H[Client: encrypt message]
H --> G
G -->|fromWidget send_event| I[Widget: sent event]
```

To trigger the action to upload a file, widgets will use a new `fromWidget` request with the action
`upload_file` which takes the following shape:

Expand All @@ -81,9 +120,34 @@ To trigger the action to upload a file, widgets will use a new `fromWidget` requ
If the widget did not get approved for the capability required to send the event, the client MUST
send an error response (as required currently by the capabilities system for widgets).

The client SHOULD NOT modify the data of the request.
TODO: error response if uploading fails? if encrypting fails?

If the event is successfully sent by the client, the client sends the following response:
#### Unencrypted

The client SHOULD NOT modify the data of the request in unencrypted rooms.

If the file is successfully uploaded by the client, the client sends the following response (mirroring the [Client-Server upload][upload] API):

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "upload_file",
"data": {
"file": "some-content"
},
"response": {
"content_uri": "mxc://..."
}
}
```

#### Encrypted

If the room is end-to-end encrypted, the client SHOULD encrypt the data before uploading, generating the usual required
[`EncryptedFile`][encryptedfile] metadata. If sending is successful, the client adds the `EncryptedFile` to the
response, resulting in:

```json
{
Expand All @@ -95,47 +159,141 @@ If the event is successfully sent by the client, the client sends the following
"file": "some-content"
},
"response": {
"content_uri": "mxc://...",
"encryption": EncryptedFile
}
}
```

### Download file

```mermaid
flowchart TD
N[Client: receive event] --> O{Client: e2ee event?}
O -->|No| P[Client: received event]
O -->|Yes| Q[Client: decrypt event]
Q --> P
P -->|toWidget send_event| R[Widget: received event]
R -->|fromWidget download_file| S[Client: download file]
S --> T{Client: widget sent EncryptedFile}
T -->|No| U[Client: downloaded file]
T -->|Yes| V[Client: decrypt file]
V --> U
U -->|toWidget download_file| W[Widget: downloaded file]
```

To trigger the action to download a file, widgets will use a new `fromWidget` request with the action `download_file`.

#### Unencrypted (Request)

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "download_file",
"data": {
"content_uri": "mxc://..."
}
}
```

The client SHOULD NOT modify the received file from the content repository before responding to the widget in
unencrypted rooms.

#### Encrypted (Request)

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "download_file",
"data": {
"content_uri": "mxc://...",
"encryption": EncryptedFile
}
}
```

#### Common Response

```json
{
"api": "fromWidget",
"widgetId": "20200827_WidgetExample",
"requestid": "generated-id-1234",
"action": "download_file",
"data": <see above>,
"response": {
"file": "some-content"
}
}
```

`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.

If the widget did not get approved for the capability required to send the event, the client MUST
send an error response (as required currently by the capabilities system for widgets).

TODO: error response if downloading fails? if decrypting fails?

## Potential issues

[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) plans to add authentication
for media access. This would require the Widget API to receive an additional extension to be able to
access stored media. It is assumed that a successor of the current widget api
(like https://github.com/matrix-org/matrix-spec-proposals/pull/3008) will replace this need in the
future.
[MSC3916][MSC3916] plans to add authentication for media access. This would require the Widget API to receive an
additional extension to be able to access stored media. It is assumed that a successor of the current widget api (like
[MSC3008]) will replace this need in the future.

[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) splits the interface of the
media upload into a “create the mxc-id” and a “upload the file for the mxc-id” stage. It is assumed
that the new API can be transparently implemented in the client so the widget could still use the old
API definition.
[MSC2246][MSC2246] splits the interface of the media upload into a “create the mxc-id” and a “upload the file for the
mxc-id” stage. It is assumed that the new API can be transparently implemented in the client so the widget could still
use the old API definition.

TODO: [MSC3911][MSC3911] linking media to events

## Alternatives

It would be preferable if the widget would not need to pass the actual data to the client over the
Widget API, but if it could acquire an authenticated upload URL that it can use to upload the file
directly to the homeserver without the need of an authentication token.
[MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) goes into this direction,
however, it still requires that the upload is authenticated.
It would be preferable if the widget would not need to pass the actual data to the client over the Widget API, but if it
could acquire an authenticated upload URL that it can use to upload the file directly to the homeserver without the need
of an authentication token. [MSC2246][MSC2246] goes into this direction, however, it still requires that the upload is
authenticated.

TODO: "proxy" the mxcs and have the client bookkeep the e2ee data?

TODO: e2ee in the widget, contrary to [MSC2762][MSC2762]

TODO: follow the C-S download API format for download_file. but our way alleviates the parsing from the widget to the
client which already can parse them.

## Security considerations

The same considerations as in [MSC2762](https://github.com/matrix-org/matrix-spec-proposals/pull/2762)
apply. This feature will allow the widget to be able to upload data into the media repository. This
could potentially be used to upload malicious content. However, the access will only be possible
when the user accepts the capability and grant access if the widget is trusted by the user.
The same considerations as in [MSC2762][MSC2762] apply. This feature will allow the widget to be able to upload data
into the media repository. This could potentially be used to upload malicious content. However, the access will only be
possible when the user accepts the capability and grant access if the widget is trusted by the user.

## Unstable prefix

While this MSC is not present in the spec, clients and widgets should:

- Use `org.matrix.msc4039.` in place of `m.` in all new identifiers of this MSC.
- Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter.
- TODO: remove? Use `org.matrix.msc4039.matrix_base_url` in place of `matrix_base_url` for the url parameter.
- Use `org.matrix.msc4039.get_media_config` in place of `get_media_config` for the action type in the
`fromWidget` requests.
- Use `org.matrix.msc4039.upload_file` in place of `upload_file` for the action type in the
`fromWidget` requests.
- Use `org.matrix.msc4039.download_file` in place of `download_file` for the action type in the
`fromWidget` requests.
- Only call/support the `action`s if an API version of `org.matrix.msc4039` is advertised.

[MSC1236]: https://github.com/matrix-org/matrix-spec-proposals/pull/1236
[MSC2246]: https://github.com/matrix-org/matrix-spec-proposals/pull/2246
[MSC2762]: https://github.com/matrix-org/matrix-spec-proposals/pull/2762
[MSC3911]: https://github.com/matrix-org/matrix-spec-proposals/pull/3911
[MSC3916]: https://github.com/matrix-org/matrix-spec-proposals/pull/3916
[content]: https://spec.matrix.org/v1.11/client-server-api/#content-repository
[mxc]: https://spec.matrix.org/v1.11/client-server-api/#matrix-content-mxc-uris
[config]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediaconfig
[upload]: https://spec.matrix.org/v1.11/client-server-api/#post_matrixmediav3upload
[download]: https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid
[encryptedfile]: https://spec.matrix.org/v1.11/client-server-api/#extensions-to-mroommessage-msgtypes

TODO: review wrt. actual current implementation

0 comments on commit 096ef1c

Please sign in to comment.