From e0e713320ba62525ce960dffc77d1bd929515b1f Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 11:17:32 +0300 Subject: [PATCH 01/39] Added proposial "Message Attachments" --- proposals/XXXX-message-attachments.md | 123 ++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 proposals/XXXX-message-attachments.md diff --git a/proposals/XXXX-message-attachments.md b/proposals/XXXX-message-attachments.md new file mode 100644 index 00000000000..843cc1956a5 --- /dev/null +++ b/proposals/XXXX-message-attachments.md @@ -0,0 +1,123 @@ +# MSCXXXX: Message Attachments + +*This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* + +In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to his text message when he is composing it. + +And now the user can send images only before the message (or after it) as a separate message, but he can't attach images during the composing process to send them when the text is finished, together with the text message in one event. + +On the display side, when the user sends multiple images, the problem is that each image is displayed alone, as separate event with full width in timeline, not linked to the message, and not grouped to the gallery. + +## Proposal + +To solve the described problem, I propose to extend `m.room.message` event with `m.attachments` field, that contains the array of message attachments. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. + +Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: + +```json +{ + "type": "m.room.message", + "content": { + "msgtype": "m.text", + "body": "Here is my photos and videos from yesterday event", + "m.attachments": [ + { + "msgtype": "m.image", + "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg", + "body": "Image 1.jpg", + "info": { + "mimetype": "image/jpg", + "size": 1153501, + "w": 963, + "h": 734, + "thumbnail_url": "mxc://example.com/0f4f88220b7c9a83d122ca8f9f11faacfc93cd18", + "thumbnail_info": { + "mimetype": "image/jpg", + "size": 575468, + "w": 787, + "h": 600 + } + } + }, + { + "msgtype": "m.video", + "url": "mxc://example.com/KUAQOe1GECk2TgdtedkftISg", + "body": "Video 2.mp4", + "info": { + "mimetype": "video/mp4", + "size": 6615304, + "w": 1280, + "h": 720, + "thumbnail_url": "mxc://example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", + "thumbnail_info": { + "mimetype": "image/jpeg", + "size": 2459, + "w": 800, + "h": 450 + }, + } + } + ] + } +} +``` + +## Client support + +### Compose recommendations: + +In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more files. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* + +On "message send" action, Matrix client must upload each attached file to server, get `mxc` of it, and attach `mxc` to message contents. + +If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message. + +### Display recommendations: + +On the client site, attachments must be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. + +If the message contains only one attachment, it can be displayed as full-width thumbnail, like current `m.image` and `m.video` messages. + +### Fallback display + +For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: + +```html +

Here is my photos and videos from yesterday event

+
+

Attachments:

+ +
+``` + +and JSON of `content` field: + +```json +"content": { + "msgtype": "m.text", + "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com//0f4f88120bfc9183d122ca8f9f11faacfc93cd18", + "format": "org.matrix.custom.html", + "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" + } +``` + +## Server support + +This MSC does not need any changes on server side. + +## Potential issues + +The main issue is fallback display for old clients. Providing the list of links to each attachment into the formatted body is suitable workaround, and clients, which render attachments on their own, can easily remove this block via cutting `
` tag. + +## Alternatives + +1. An alternative can be posting media messages as separate events, as it was done earlier, and aggregating them into event via `m.relates_to` field, but clients must do a hide of those events, when aggregating event will be added to the room (like editions), but this will be harder to implement. +2. Other alternative is embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. +3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. + +## Future considerations + +In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). From 96fef8b4b327d7450a6d3d964e489c3757f05e26 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 11:25:12 +0300 Subject: [PATCH 02/39] Assign 2881 number to MSC and some typos --- ...X-message-attachments.md => 2881-message-attachments.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename proposals/{XXXX-message-attachments.md => 2881-message-attachments.md} (97%) diff --git a/proposals/XXXX-message-attachments.md b/proposals/2881-message-attachments.md similarity index 97% rename from proposals/XXXX-message-attachments.md rename to proposals/2881-message-attachments.md index 843cc1956a5..e86a39c7f0b 100644 --- a/proposals/XXXX-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -1,4 +1,4 @@ -# MSCXXXX: Message Attachments +# MSC2881: Message Attachments *This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* @@ -98,9 +98,9 @@ and JSON of `content` field: ```json "content": { "msgtype": "m.text", - "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com//0f4f88120bfc9183d122ca8f9f11faacfc93cd18", + "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", "format": "org.matrix.custom.html", - "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" + "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" } ``` From 8065221043bb445dbbae15b66d0ca28115fe7de7 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 21:31:00 +0300 Subject: [PATCH 03/39] Added mentions of similar MSC #2530 and #2529 --- proposals/2881-message-attachments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index e86a39c7f0b..d2d6e30d0a1 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -117,6 +117,7 @@ The main issue is fallback display for old clients. Providing the list of links 1. An alternative can be posting media messages as separate events, as it was done earlier, and aggregating them into event via `m.relates_to` field, but clients must do a hide of those events, when aggregating event will be added to the room (like editions), but this will be harder to implement. 2. Other alternative is embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. 3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. +4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images #2529](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. ## Future considerations From 0de8e47e8c8b3407ae410c18fef33c04a6d6b6c4 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 21:47:48 +0300 Subject: [PATCH 04/39] Direct mxc links and improved alternative --- proposals/2881-message-attachments.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index d2d6e30d0a1..58e84ab1497 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -104,6 +104,8 @@ and JSON of `content` field: } ``` +If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. + ## Server support This MSC does not need any changes on server side. @@ -114,10 +116,13 @@ The main issue is fallback display for old clients. Providing the list of links ## Alternatives -1. An alternative can be posting media messages as separate events, as it was done earlier, and aggregating them into event via `m.relates_to` field, but clients must do a hide of those events, when aggregating event will be added to the room (like editions), but this will be harder to implement. +1. Main alternative can be posting media messages as separate events, as it was done earlier, and aggregating them into event via `m.relates_to` field, but clients must do a hide of those events, when aggregating event will be added to the room (like edits). This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one message with 20 attachments - in room will be 21 events instead of 1). + 2. Other alternative is embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. + 3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. -4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images #2529](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. + +4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. ## Future considerations From 45c91fd50d2210f05cebb21b6ed926cc2ae3920a Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 21:53:13 +0300 Subject: [PATCH 05/39] Improved description of 1'st alternative --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 58e84ab1497..dd4834831c0 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -116,7 +116,7 @@ The main issue is fallback display for old clients. Providing the list of links ## Alternatives -1. Main alternative can be posting media messages as separate events, as it was done earlier, and aggregating them into event via `m.relates_to` field, but clients must do a hide of those events, when aggregating event will be added to the room (like edits). This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one message with 20 attachments - in room will be 21 events instead of 1). +1. Main alternative is posting media messages as separate events, as it was done earlier, and aggregating them later into one visual place via event with `m.relates_to` field. So modern clients must do a hide of those events, when aggregating event will be added to the room (like edits do now). This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one text message with 20 attachments - in room will generate 21 events instead of 1). For exclude showing those events in modern clients before grouping event added, we can also extend separate media events via adding into them some "marker" field like `show: false` or `attached_to_message: true`. 2. Other alternative is embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. From a2261420aae5fe37095a9f61a0dce5abca3a867a Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:08:42 +0300 Subject: [PATCH 06/39] Added second implementation --- proposals/2881-message-attachments.md | 42 +++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index dd4834831c0..42d6d6c062a 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -10,7 +10,11 @@ On the display side, when the user sends multiple images, the problem is that ea ## Proposal -To solve the described problem, I propose to extend `m.room.message` event with `m.attachments` field, that contains the array of message attachments. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. +To solve the described problem, I propose to extend `m.room.message` event with `m.attachments` field, that contains the array of message attachments. I see two ways of implementation, can't decide which is better: + +### Implementation 1: One event with direct links to all attached media + +This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: @@ -61,6 +65,34 @@ Each element of `m.attachments` array has a structure like a message with media } } ``` +### Implementation 2: Aggregating event to group several previously sent media events + +In this implementation - client will send all attached media as separate events to room (how it is done now) before sending message, and after - send message an aggregating event with `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), pointing to all those events, to group them into one gallery. + +This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one text message with 20 attachments - in room will generate 21 events instead of 1). + +For exclude showing those events in modern clients before grouping event added, we can also extend separate media events via adding into them some "marker" field like `is_attachment: true`. + +Here is example of this implementation: +```json +{ + "type": "m.room.message", + "content": { + "msgtype": "m.text", + "body": "Here is my photos and videos from yesterday event", + "m.relates_to": [ + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_1" + {, + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_2" + { + ] + } +} +``` ## Client support @@ -116,13 +148,11 @@ The main issue is fallback display for old clients. Providing the list of links ## Alternatives -1. Main alternative is posting media messages as separate events, as it was done earlier, and aggregating them later into one visual place via event with `m.relates_to` field. So modern clients must do a hide of those events, when aggregating event will be added to the room (like edits do now). This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one text message with 20 attachments - in room will generate 21 events instead of 1). For exclude showing those events in modern clients before grouping event added, we can also extend separate media events via adding into them some "marker" field like `show: false` or `attached_to_message: true`. - -2. Other alternative is embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. +1. Alternative can be embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. -3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. +2. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. -4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. +3. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. ## Future considerations From 616dcbadde5bb3f36f0de29d3b062d52258602a8 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:13:12 +0300 Subject: [PATCH 07/39] Added unstable prefixes, fixed typo --- proposals/2881-message-attachments.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 42d6d6c062a..2f3eb691399 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -84,11 +84,11 @@ Here is example of this implementation: { "rel_type": "m.attachment", "event_id": "$id_of_previosly_send_media_event_1" - {, + }, { "rel_type": "m.attachment", "event_id": "$id_of_previosly_send_media_event_2" - { + } ] } } @@ -157,3 +157,6 @@ The main issue is fallback display for old clients. Providing the list of links ## Future considerations In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). + + ## Unstable prefix +Clients should use `org.matrix.msc2881.m.attachments` and `org.matrix.msc2881.m.attachment` field names, while this MSC has not been included in a spec release. From 6b5ccbe2f37f4d4f151f7e3789b0d6861c31ce38 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:14:41 +0300 Subject: [PATCH 08/39] Added unstable to class to --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 2f3eb691399..56019a19334 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -159,4 +159,4 @@ The main issue is fallback display for old clients. Providing the list of links In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). ## Unstable prefix -Clients should use `org.matrix.msc2881.m.attachments` and `org.matrix.msc2881.m.attachment` field names, while this MSC has not been included in a spec release. +Clients should use `org.matrix.msc2881.m.attachments`, `org.matrix.msc2881.m.attachment` and `org-matrix-msc2881-mx-attachments` strings instead of proposed, while this MSC has not been included in a spec release. From e343893fded3fb1a7bb6ad67d371241955b32d8f Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:16:29 +0300 Subject: [PATCH 09/39] Moved fallback dispay to implementation 1 --- proposals/2881-message-attachments.md | 58 ++++++++++++++------------- 1 file changed, 30 insertions(+), 28 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 56019a19334..a2224888550 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -65,6 +65,36 @@ Each element of `m.attachments` array has a structure like a message with media } } ``` + +#### Fallback display + +For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: + +```html +

Here is my photos and videos from yesterday event

+
+

Attachments:

+ +
+``` + +and JSON of `content` field: + +```json +"content": { + "msgtype": "m.text", + "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", + "format": "org.matrix.custom.html", + "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" + } +``` + +If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. + + ### Implementation 2: Aggregating event to group several previously sent media events In this implementation - client will send all attached media as separate events to room (how it is done now) before sending message, and after - send message an aggregating event with `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), pointing to all those events, to group them into one gallery. @@ -110,34 +140,6 @@ On the client site, attachments must be displayed as grid of clickable thumbnail If the message contains only one attachment, it can be displayed as full-width thumbnail, like current `m.image` and `m.video` messages. -### Fallback display - -For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: - -```html -

Here is my photos and videos from yesterday event

-
-

Attachments:

- -
-``` - -and JSON of `content` field: - -```json -"content": { - "msgtype": "m.text", - "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", - "format": "org.matrix.custom.html", - "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" - } -``` - -If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. - ## Server support This MSC does not need any changes on server side. From d448559fcf3669e4d113de40021768fa1c90786a Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:21:15 +0300 Subject: [PATCH 10/39] Added example of media event to implentation 2 --- proposals/2881-message-attachments.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index a2224888550..8c0de963e5f 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -103,7 +103,21 @@ This way give better fallback, but will generate more unecessary events instead For exclude showing those events in modern clients before grouping event added, we can also extend separate media events via adding into them some "marker" field like `is_attachment: true`. -Here is example of this implementation: +Here is example of this implementation - media events, to sent before aggregating event: +```json +{ + "msgtype": "m.image", + "body": "Image 1.jpg", + "info": { + "mimetype": "image/jpg", + "size": 1153501, + "w": 963, + "h": 734, + }, + "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" +}, +``` +And aggregating event: ```json { "type": "m.room.message", From c33ef1ac2efce02570f9057c3e57e4e240da703b Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 22:36:00 +0300 Subject: [PATCH 11/39] Move second implementation to main place --- proposals/2881-message-attachments.md | 152 ++++++++++++-------------- 1 file changed, 69 insertions(+), 83 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 8c0de963e5f..8e8cad61f4b 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -10,11 +10,74 @@ On the display side, when the user sends multiple images, the problem is that ea ## Proposal -To solve the described problem, I propose to extend `m.room.message` event with `m.attachments` field, that contains the array of message attachments. I see two ways of implementation, can't decide which is better: +Matrix client allow users to attach media (images, video, files) to message without instant sending of them to room, and send together with text message. -### Implementation 1: One event with direct links to all attached media +When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery. -This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. +For exclude showing those events in modern clients before grouping event added, I propose extend separate media events via adding "marker" field `is_attachment: true`, if clients got this value - they must exclude showing this media in timeline, and shows them only in gallery with grouping event. + +Example of media event, that send before grouping event: +```json +{ + "msgtype": "m.image", + "body": "Image 1.jpg", + "info": { + "mimetype": "image/jpg", + "size": 1153501, + "w": 963, + "h": 734, + }, + "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" +}, +``` +And aggregating event, to send after all message attachments: +```json +{ + "type": "m.room.message", + "content": { + "msgtype": "m.text", + "body": "Here is my photos and videos from yesterday event", + "m.relates_to": [ + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_1" + }, + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_2" + } + ] + } +} +``` + +## Client support + +### Compose recommendations: + +In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more files. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* + +On "message send" action, Matrix client must upload each attached file to server, get `mxc` of it, and attach `mxc` to message contents. + +If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message. + +### Display recommendations: + +On the client site, attachments must be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. + +If the message contains only one attachment, it can be displayed as full-width thumbnail, like current `m.image` and `m.video` messages. + +## Server support + +This MSC does not need any changes on server side. + +## Potential issues + +The main issue is fallback display for old clients. Providing the list of links to each attachment into the formatted body is suitable workaround, and clients, which render attachments on their own, can easily remove this block via cutting `
` tag. + +## Alternatives + +1. Alternative implementation can be sending one event with direct links to all attached media, instead of sending separate event for each attachment. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: @@ -65,11 +128,7 @@ Each element of `m.attachments` array has a structure like a message with media } } ``` - -#### Fallback display - For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: - ```html

Here is my photos and videos from yesterday event

@@ -80,9 +139,7 @@ For fallback display of attachments in old Matrix clients, we can attach them di
``` - and JSON of `content` field: - ```json "content": { "msgtype": "m.text", @@ -91,84 +148,13 @@ and JSON of `content` field: "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" } ``` - If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. +2. Alternative can be embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. -### Implementation 2: Aggregating event to group several previously sent media events - -In this implementation - client will send all attached media as separate events to room (how it is done now) before sending message, and after - send message an aggregating event with `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), pointing to all those events, to group them into one gallery. - -This way give better fallback, but will generate more unecessary events instead of one event with group of medias (eg when user send one text message with 20 attachments - in room will generate 21 events instead of 1). - -For exclude showing those events in modern clients before grouping event added, we can also extend separate media events via adding into them some "marker" field like `is_attachment: true`. - -Here is example of this implementation - media events, to sent before aggregating event: -```json -{ - "msgtype": "m.image", - "body": "Image 1.jpg", - "info": { - "mimetype": "image/jpg", - "size": 1153501, - "w": 963, - "h": 734, - }, - "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" -}, -``` -And aggregating event: -```json -{ - "type": "m.room.message", - "content": { - "msgtype": "m.text", - "body": "Here is my photos and videos from yesterday event", - "m.relates_to": [ - { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_1" - }, - { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_2" - } - ] - } -} -``` - -## Client support - -### Compose recommendations: - -In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more files. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* - -On "message send" action, Matrix client must upload each attached file to server, get `mxc` of it, and attach `mxc` to message contents. - -If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message. - -### Display recommendations: - -On the client site, attachments must be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. - -If the message contains only one attachment, it can be displayed as full-width thumbnail, like current `m.image` and `m.video` messages. - -## Server support - -This MSC does not need any changes on server side. - -## Potential issues - -The main issue is fallback display for old clients. Providing the list of links to each attachment into the formatted body is suitable workaround, and clients, which render attachments on their own, can easily remove this block via cutting `
` tag. - -## Alternatives - -1. Alternative can be embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. - -2. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. +3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. -3. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. +4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. ## Future considerations From ad7f2fa87bf3a9c7ae3377cfefb73d05623586c4 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 27 Nov 2020 23:22:30 +0300 Subject: [PATCH 12/39] Paraphrasing --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 8e8cad61f4b..037d1a9ea29 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -1,6 +1,6 @@ # MSC2881: Message Attachments -*This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* +*This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, and in future - external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to his text message when he is composing it. From dd423aa431bf608a27a32491dcd3c57db8e270e0 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 09:00:47 +0300 Subject: [PATCH 13/39] Described edits and improved description --- proposals/2881-message-attachments.md | 38 +++++++++++++++++++++------ 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 037d1a9ea29..3026f8ef785 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -27,6 +27,7 @@ Example of media event, that send before grouping event: "w": 963, "h": 734, }, + "is_attachment": true, "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" }, ``` @@ -50,22 +51,39 @@ And aggregating event, to send after all message attachments: } } ``` +For edits of "message with attachments" we can reuse same "m.relates_to" array via simply adding `"rel_type": "m.replace"` item to it, here is example: +```json + "m.relates_to": [ + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_1" + }, + { + "rel_type": "m.replace", + "event_id": "$id_of_original event" + }, + { + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_2" + } + ] +``` ## Client support ### Compose recommendations: -In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more files. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* +In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* -On "message send" action, Matrix client must upload each attached file to server, get `mxc` of it, and attach `mxc` to message contents. +On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array. -If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message. +If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. ### Display recommendations: -On the client site, attachments must be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. +On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. -If the message contains only one attachment, it can be displayed as full-width thumbnail, like current `m.image` and `m.video` messages. +If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. ## Server support @@ -73,11 +91,13 @@ This MSC does not need any changes on server side. ## Potential issues -The main issue is fallback display for old clients. Providing the list of links to each attachment into the formatted body is suitable workaround, and clients, which render attachments on their own, can easily remove this block via cutting `
` tag. +1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered. + +2. Individual media event, to which `m.message` refers, can be deleted after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. ## Alternatives -1. Alternative implementation can be sending one event with direct links to all attached media, instead of sending separate event for each attachment. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. +1. Main alternative implementation (my fist proposal in this MSC) is sending only one event with direct links to all attached media, instead of sending separate event for each attachment. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: @@ -150,7 +170,9 @@ and JSON of `content` field: ``` If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. -2. Alternative can be embedding images (and other media types) into message body via html tags, but this will make extracting and stylizing of the attachments harder. +This way will give less "spam" for room, because when user sends message with 20 attachments, it will send only one event to room, instead of 21 like in main implementation. But it have worse fallback, than main implementation. + +2. Second alternative can be embedding images (and other media types) into message body via html tags to "body" field, but this will make extracting and stylizing of the attachments harder. 3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. From 76e0522ec7f3b15b396e884ebdac6da47b40fed3 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 09:05:25 +0300 Subject: [PATCH 14/39] Added fallback description --- proposals/2881-message-attachments.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 3026f8ef785..ee28f9fc119 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -68,6 +68,9 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v } ] ``` +### Fallback: + +I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment before sending main message. ## Client support From 9a7f694832181704b4bbabec12ca82c943beccfe Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 09:09:09 +0300 Subject: [PATCH 15/39] Added edits composer description --- proposals/2881-message-attachments.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index ee28f9fc119..a5a03004391 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -82,6 +82,8 @@ On "message send" action, Matrix client must upload each attached media to serve If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. +Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can remove one of current attachments (that will remove its line from array of `m.relates_to`), add new attachment (that will upload it as new event, and refer to it in edited message `m.relates_to` array). + ### Display recommendations: On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. From 910be82cdfccad26bcc1074608d231ece8462668 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 09:49:52 +0300 Subject: [PATCH 16/39] Added link to extensible events suggestion --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index a5a03004391..5e825189b26 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -179,7 +179,7 @@ This way will give less "spam" for room, because when user sends message with 20 2. Second alternative can be embedding images (and other media types) into message body via html tags to "body" field, but this will make extracting and stylizing of the attachments harder. -3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. Maybe, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message. +3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. So, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message, [here](https://github.com/matrix-org/matrix-doc/pull/1767/files#r532373829) is my comment with this suggestion. 4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. From abc7edfb31cc91e8c933493215177169c4c239f0 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 09:56:58 +0300 Subject: [PATCH 17/39] Added alternative with attachments after message --- proposals/2881-message-attachments.md | 42 ++++++++++++++++++++------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 5e825189b26..0e9b80aa9e0 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -40,12 +40,12 @@ And aggregating event, to send after all message attachments: "body": "Here is my photos and videos from yesterday event", "m.relates_to": [ { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_1" + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_1" }, { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_2" + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_2" } ] } @@ -55,16 +55,16 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v ```json "m.relates_to": [ { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_1" + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_1" }, { - "rel_type": "m.replace", - "event_id": "$id_of_original event" + "rel_type": "m.replace", + "event_id": "$id_of_original event" }, { - "rel_type": "m.attachment", - "event_id": "$id_of_previosly_send_media_event_2" + "rel_type": "m.attachment", + "event_id": "$id_of_previosly_send_media_event_2" } ] ``` @@ -183,6 +183,28 @@ This way will give less "spam" for room, because when user sends message with 20 4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. +5. Other alternative can be posting `m.message` event at first, and link all attachments to it later via `m.relates_to` field, something like this: +```json +{ + "msgtype": "m.image", + "body": "Image 1.jpg", + "info": { + "mimetype": "image/jpg", + "size": 1153501, + "w": 963, + "h": 734, + }, + "m.relates_to": [ + { + "rel_type": "m.attachment_to", + "event_id": "$id_of_main_message" + }, + "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" +}, +``` +But this way will give harder way to render of main message event, because Matrix clients must be search all attached events manually in timeline. + + ## Future considerations In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). From 108ffa7a62ea4922807e2e0525802338d4eb9eb1 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 10:00:47 +0300 Subject: [PATCH 18/39] Added redact action on remove media --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 0e9b80aa9e0..6260bcd92fc 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -82,7 +82,7 @@ On "message send" action, Matrix client must upload each attached media to serve If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. -Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can remove one of current attachments (that will remove its line from array of `m.relates_to`), add new attachment (that will upload it as new event, and refer to it in edited message `m.relates_to` array). +Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media), add new attachment (that will upload it as new event, and refer to it in edited message `m.relates_to` array). ### Display recommendations: From 2be176c5e28e28107ebadd617db8bda99383de43 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 10:17:58 +0300 Subject: [PATCH 19/39] Fixed typo --- proposals/2881-message-attachments.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 6260bcd92fc..07a2b198dc3 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -198,7 +198,8 @@ This way will give less "spam" for room, because when user sends message with 20 { "rel_type": "m.attachment_to", "event_id": "$id_of_main_message" - }, + } + ] "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" }, ``` From eea1ba0892172912e8b62ab68c7dec283304eeef Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 10:18:27 +0300 Subject: [PATCH 20/39] Fix typo again --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 07a2b198dc3..c8a4baa6767 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -199,7 +199,7 @@ This way will give less "spam" for room, because when user sends message with 20 "rel_type": "m.attachment_to", "event_id": "$id_of_main_message" } - ] + ], "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" }, ``` From 87bfff25cd6bc3381f859f7f70b01785772d9a0f Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:03:24 +0300 Subject: [PATCH 21/39] Added delete (redact action) --- proposals/2881-message-attachments.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index c8a4baa6767..6ef191cef4a 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -68,6 +68,8 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v } ] ``` +For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment too. + ### Fallback: I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment before sending main message. From ba196936dce848467f228d29cab3099cbbc21a3f Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:11:49 +0300 Subject: [PATCH 22/39] Added potential issue and hiding recommendation --- proposals/2881-message-attachments.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 6ef191cef4a..918c751b2db 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -8,6 +8,7 @@ And now the user can send images only before the message (or after it) as a sepa On the display side, when the user sends multiple images, the problem is that each image is displayed alone, as separate event with full width in timeline, not linked to the message, and not grouped to the gallery. + ## Proposal Matrix client allow users to attach media (images, video, files) to message without instant sending of them to room, and send together with text message. @@ -74,6 +75,7 @@ For delete (redact action) message with attachments, we must also apply `redact` I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment before sending main message. + ## Client support ### Compose recommendations: @@ -86,21 +88,29 @@ If the user uploads only one media and leaves the message text empty, media can Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media), add new attachment (that will upload it as new event, and refer to it in edited message `m.relates_to` array). + ### Display recommendations: On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. +For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments, for speed-up display of them in gallery. + + ## Server support This MSC does not need any changes on server side. + ## Potential issues 1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered. -2. Individual media event, to which `m.message` refers, can be deleted after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. +2. Individual media event, to which `m.message` refers, can be deleted (redacted) after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. + +3. There are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server. + ## Alternatives @@ -212,5 +222,6 @@ But this way will give harder way to render of main message event, because Matri In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). - ## Unstable prefix + +## Unstable prefix Clients should use `org.matrix.msc2881.m.attachments`, `org.matrix.msc2881.m.attachment` and `org-matrix-msc2881-mx-attachments` strings instead of proposed, while this MSC has not been included in a spec release. From 6147bd629a8266f0a870c9328b540dea6227658a Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:16:44 +0300 Subject: [PATCH 23/39] Added link to MSC2278 --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 918c751b2db..81bb56bc76d 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -80,7 +80,7 @@ I see no serious problems with fallback display of attachments. For Matrix clien ### Compose recommendations: -In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API, so server will store each file, even if it is not attached to the message.* +In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API yet ([MSC2278: Deleting attachments for expired and redacted messages](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md), so server will store each file, even if it is not attached to the message.* On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array. From 6da8d3a1b269aa499515433df762a5e34b647f7d Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:24:05 +0300 Subject: [PATCH 24/39] Fix typos and added link to MSC2675 --- proposals/2881-message-attachments.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 81bb56bc76d..cd30810def0 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -52,6 +52,7 @@ And aggregating event, to send after all message attachments: } } ``` + For edits of "message with attachments" we can reuse same "m.relates_to" array via simply adding `"rel_type": "m.replace"` item to it, here is example: ```json "m.relates_to": [ @@ -69,6 +70,7 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v } ] ``` + For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment too. ### Fallback: @@ -97,6 +99,7 @@ If the message contains only one attachment, it can be displayed as full-width t For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments, for speed-up display of them in gallery. +Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. ## Server support @@ -215,7 +218,7 @@ This way will give less "spam" for room, because when user sends message with 20 "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" }, ``` -But this way will give harder way to render of main message event, because Matrix clients must be search all attached events manually in timeline. +But this way will give harder way to render of main message event, because Matrix clients must do the search of all attached events manually in timeline, and server will can't aggregate them to main message. ## Future considerations From fa6f59b7f3f219078b647a2ff2d2ddbad8141cb2 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:25:29 +0300 Subject: [PATCH 25/39] Fix typo --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index cd30810def0..8fab885d9d3 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -97,7 +97,7 @@ On the client site, attachments can be displayed as grid of clickable thumbnails If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. -For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments, for speed-up display of them in gallery. +For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery. Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. From 2e8536eadc5cc462b07d8ea3538f991c427e051d Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:29:43 +0300 Subject: [PATCH 26/39] Added proposal for replacement to #2530 / #2529 --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 8fab885d9d3..7be9821b3bf 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -196,7 +196,7 @@ This way will give less "spam" for room, because when user sends message with 20 3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. So, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message, [here](https://github.com/matrix-org/matrix-doc/pull/1767/files#r532373829) is my comment with this suggestion. -4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. +4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. And also this MSC can be replacement of #2530 / #2529, when user send text + only one media item. 5. Other alternative can be posting `m.message` event at first, and link all attachments to it later via `m.relates_to` field, something like this: ```json From 0eb7773186d9a3f58f1217e18b4eeb25c2f5530c Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:42:01 +0300 Subject: [PATCH 27/39] Added links to implementation examples --- proposals/2881-message-attachments.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 7be9821b3bf..4838a398f58 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -2,12 +2,13 @@ *This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, and in future - external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* -In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to his text message when he is composing it. +In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases even one media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to his text message when he is composing it, not after or before it. And now the user can send images only before the message (or after it) as a separate message, but he can't attach images during the composing process to send them when the text is finished, together with the text message in one event. On the display side, when the user sends multiple images, the problem is that each image is displayed alone, as separate event with full width in timeline, not linked to the message, and not grouped to the gallery. +Messages with multiple attachments now already implemented in many messengers, for example - in Skype, Slack, VK Messenger. And Matrix, because lack of support, now have problems with bridging those messages to Matrix rooms. ## Proposal @@ -93,10 +94,12 @@ Editing interface can be represented exactly like the composer interface, where ### Display recommendations: -On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. +On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. +Example of composer interface implementation we can lookup in [Slack](https://slack.com/), [VK Messenger](https://vk.com/messenger), [Skype](https://skype.com). + For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery. Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. From 00ade35c1ca8611152d94bf37bff4c6a406a3919 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 11:51:52 +0300 Subject: [PATCH 28/39] Improved description --- proposals/2881-message-attachments.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 4838a398f58..7a23f1173d2 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -10,15 +10,18 @@ On the display side, when the user sends multiple images, the problem is that ea Messages with multiple attachments now already implemented in many messengers, for example - in Skype, Slack, VK Messenger. And Matrix, because lack of support, now have problems with bridging those messages to Matrix rooms. + ## Proposal -Matrix client allow users to attach media (images, video, files) to message without instant sending of them to room, and send together with text message. +For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message. + +With having this feature, Matrix client should allow users to attach one or multiple media (images, video, files) to message on client side, without instant sending of them to room, and send them together with text message. When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery. For exclude showing those events in modern clients before grouping event added, I propose extend separate media events via adding "marker" field `is_attachment: true`, if clients got this value - they must exclude showing this media in timeline, and shows them only in gallery with grouping event. -Example of media event, that send before grouping event: +Example of media event, that send before aggregating event: ```json { "msgtype": "m.image", @@ -72,11 +75,11 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v ] ``` -For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment too. +For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment event too. ### Fallback: -I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment before sending main message. +I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. ## Client support @@ -117,6 +120,7 @@ This MSC does not need any changes on server side. 3. There are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server. +4. If client attach too much media to one message, he can got rate limiting problem on server side. This can be solved via splitting and delaying send of attachments, to match server rate limits. ## Alternatives From 5df15b6d2b1e7bafb41fb4f593de4753f172db67 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 30 Nov 2020 18:04:39 +0300 Subject: [PATCH 29/39] Description for implementation together with #2530 --- proposals/2881-message-attachments.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 7a23f1173d2..e647cf02231 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -203,7 +203,9 @@ This way will give less "spam" for room, because when user sends message with 20 3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. So, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message, [here](https://github.com/matrix-org/matrix-doc/pull/1767/files#r532373829) is my comment with this suggestion. -4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough. And also this MSC can be replacement of #2530 / #2529, when user send text + only one media item. +4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough for most cases, also this MSC can be the replacement of #2530 / #2529, when user send text + only one media item. + +Implementing both things together (MSC2881 as one text for all attachments + separate media caption text for each attachment via [MSC2530](https://github.com/matrix-org/matrix-doc/pull/2530)) is possble, but will give very complex UI for manage this in Matrix clients, so, I think, Matrix don't need so complex feature and only one text for all attachments will be enouth, for manage full-featured "Photos Albums" with descriptions and comments we already have more suitable tools. 5. Other alternative can be posting `m.message` event at first, and link all attachments to it later via `m.relates_to` field, something like this: ```json From a470fbab4a6c654e91fe07418d8a5766b52cf05c Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Wed, 16 Dec 2020 21:36:46 +0300 Subject: [PATCH 30/39] improve grammar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Erkin Alp Güney --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index e647cf02231..d21cf2ad978 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -227,7 +227,7 @@ Implementing both things together (MSC2881 as one text for all attachments + sep "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg" }, ``` -But this way will give harder way to render of main message event, because Matrix clients must do the search of all attached events manually in timeline, and server will can't aggregate them to main message. +But this way will give harder way to render of main message event, because Matrix clients must do the search of all attached events manually in timeline, and server will be unable to aggregate them to main message. ## Future considerations From 39c08c4d1cc5785ddfa5d6a1b1d264a07291a98f Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Wed, 16 Dec 2020 22:14:40 +0300 Subject: [PATCH 31/39] Move back first alternative as option two --- proposals/2881-message-attachments.md | 121 ++++++++++++-------------- 1 file changed, 58 insertions(+), 63 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index d21cf2ad978..a25a74ee7ed 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -13,6 +13,8 @@ Messages with multiple attachments now already implemented in many messengers, f ## Proposal +### Option one + For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message. With having this feature, Matrix client should allow users to attach one or multiple media (images, video, files) to message on client side, without instant sending of them to room, and send them together with text message. @@ -77,54 +79,15 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment event too. -### Fallback: +#### Fallback: I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. +### Option two -## Client support - -### Compose recommendations: - -In the message composer, on "paste file" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API yet ([MSC2278: Deleting attachments for expired and redacted messages](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md), so server will store each file, even if it is not attached to the message.* - -On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array. - -If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. - -Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media), add new attachment (that will upload it as new event, and refer to it in edited message `m.relates_to` array). - - -### Display recommendations: - -On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. - -If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. - -Example of composer interface implementation we can lookup in [Slack](https://slack.com/), [VK Messenger](https://vk.com/messenger), [Skype](https://skype.com). - -For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery. - -Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. - -## Server support - -This MSC does not need any changes on server side. - +As lite alternative to option one, we can send only one event with direct links to all attached media, instead of sending separate event for each attachment, because it will give less "spam" for room. Eg when user is sending message with 20 attachments - it will send only one event to room, instead of 21 like in option one implementation. But the main problem with this option is *fallback*. -## Potential issues - -1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered. - -2. Individual media event, to which `m.message` refers, can be deleted (redacted) after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. - -3. There are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server. - -4. If client attach too much media to one message, he can got rate limiting problem on server side. This can be solved via splitting and delaying send of attachments, to match server rate limits. - -## Alternatives - -1. Main alternative implementation (my fist proposal in this MSC) is sending only one event with direct links to all attached media, instead of sending separate event for each attachment. This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. +This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: @@ -144,13 +107,6 @@ Each element of `m.attachments` array has a structure like a message with media "size": 1153501, "w": 963, "h": 734, - "thumbnail_url": "mxc://example.com/0f4f88220b7c9a83d122ca8f9f11faacfc93cd18", - "thumbnail_info": { - "mimetype": "image/jpg", - "size": 575468, - "w": 787, - "h": 600 - } } }, { @@ -162,19 +118,15 @@ Each element of `m.attachments` array has a structure like a message with media "size": 6615304, "w": 1280, "h": 720, - "thumbnail_url": "mxc://example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", - "thumbnail_info": { - "mimetype": "image/jpeg", - "size": 2459, - "w": 800, - "h": 450 - }, } } ] } } ``` + +#### Fallback + For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: ```html

Here is my photos and videos from yesterday event

@@ -197,17 +149,60 @@ and JSON of `content` field: ``` If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. -This way will give less "spam" for room, because when user sends message with 20 attachments, it will send only one event to room, instead of 21 like in main implementation. But it have worse fallback, than main implementation. +**If we will come up with better fallback display, maybe bring this option as main suggestion?** + + +## Client support + +### Compose recommendations: + +In the message composer, on "upload file" or "paste file from clipboard" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API yet ([MSC2278: Deleting attachments for expired and redacted messages](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md), so server will store each file, even if it is not attached to the message.* + +On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array (option one); or collect all `mxc` urls in `m.attachments` array on option two. + +If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. + +Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can rearrange attachments, remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media in option one, or remove item from `m.attachments` in option two; and delete media file using [MSC2278](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md)), add new attachment (that will upload it as new event with refer to it in edited message `m.relates_to` array in option one / added to `m.attachments` in option two). + + +### Display recommendations: + +On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. + +If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. + +Example of composer interface implementation with multiple attachments we can lookup in [Slack](https://slack.com/), [VK Messenger](https://vk.com/messenger), [Skype](https://skype.com). + +For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery. + +Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. + +## Server support + +This MSC does not need any changes on server side. + + +## Potential issues + +1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered. + +2. In option one - individual media event, to which `m.message` refers, can be deleted (redacted) after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. + +3. In option one - there are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server. + +4. If client attach too much media to one message, he can got rate limiting problem on server side. This can be solved via splitting and delaying send of attachments, to match server rate limits. + +## Alternatives -2. Second alternative can be embedding images (and other media types) into message body via html tags to "body" field, but this will make extracting and stylizing of the attachments harder. +1. An alternative can be embedding images (and other media types) directrly into message body via html tags to "body" field (`` or ``), but this will make composing, extracting and stylizing of the attachments harder. -3. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. So, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message, [here](https://github.com/matrix-org/matrix-doc/pull/1767/files#r532373829) is my comment with this suggestion. +2. Next alternative is reuse [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) for attaching and grouping media attachments, but in current state it requires only one unique type of content per message, so we can't attach, for example, two `m.image` items into one message. So, instead of separate current issue, we can extend [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) via converting `content` to array, to allow adding several items of same type to one message, [here](https://github.com/matrix-org/matrix-doc/pull/1767/files#r532373829) is my comment with this suggestion. -4. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough for most cases, also this MSC can be the replacement of #2530 / #2529, when user send text + only one media item. +3. There are also [MSC2530: Body field as media caption](https://github.com/matrix-org/matrix-doc/pull/2530) but it describes only text description for one media, not several media items, and very similar [MSC2529: Proposal to use existing events as captions for images](https://github.com/matrix-org/matrix-doc/pull/2529) that implement same thing, but via separate event. But if we send several medias grouped as gallery, usually one text description is enough for most cases, also this MSC can be the replacement of #2530 / #2529, when user send text + only one media item. And the main problem with those MSC, that in most cases, it is the image that is the comment to the text message, and not vice versa, as implied in the phrase "image caption" from those MSC, [here is my comment about this in that MSC](https://github.com/matrix-org/matrix-doc/pull/2529#issuecomment-736638196). -Implementing both things together (MSC2881 as one text for all attachments + separate media caption text for each attachment via [MSC2530](https://github.com/matrix-org/matrix-doc/pull/2530)) is possble, but will give very complex UI for manage this in Matrix clients, so, I think, Matrix don't need so complex feature and only one text for all attachments will be enouth, for manage full-featured "Photos Albums" with descriptions and comments we already have more suitable tools. +We still can implement both things together (my *MSC2881* as one main text with as attachments + separate short *caption texts* for each attachment via [MSC2530](https://github.com/matrix-org/matrix-doc/pull/2530)), but this will give very complex UI for manage attachments in Matrix clients, so, I think, Matrix don't need so complex feature and only one text for all attachments will be enouth, for manage full-featured "Photos Albums" with descriptions and comments we already have more suitable tools. -5. Other alternative can be posting `m.message` event at first, and link all attachments to it later via `m.relates_to` field, something like this: +4. Other alternative can be posting `m.message` event at first, and link all attachments to it later via `m.relates_to` field, something like this: ```json { "msgtype": "m.image", From 12c0a3fe5694e7240476c88c1691d58e6976b95e Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Wed, 16 Dec 2020 22:16:50 +0300 Subject: [PATCH 32/39] Fix typo --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index a25a74ee7ed..ed569593b1f 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -144,7 +144,7 @@ and JSON of `content` field: "msgtype": "m.text", "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", "format": "org.matrix.custom.html", - "formatted_body": "

Here is my photos and videos from yesterday event

\n
" + "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" } ``` If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. From 21f14c243b5878fbb311008228fbdb6341183fb1 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Wed, 16 Dec 2020 22:20:07 +0300 Subject: [PATCH 33/39] Described replacing fallback to rich representation --- proposals/2881-message-attachments.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index ed569593b1f..3693dbb204d 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -147,6 +147,8 @@ and JSON of `content` field: "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" } ``` +And modern clients, that have support of the attachments feature, will cut the `div.mx-attachment` tag and replace it to rich gallery block with thumbnails of attachments. + If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. **If we will come up with better fallback display, maybe bring this option as main suggestion?** From cfb8f557809241538c11de249d4d566e91098bca Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Thu, 17 Dec 2020 15:05:22 +0300 Subject: [PATCH 34/39] Name options as primary-secondary --- proposals/2881-message-attachments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 3693dbb204d..e0822f5aec5 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -13,7 +13,7 @@ Messages with multiple attachments now already implemented in many messengers, f ## Proposal -### Option one +### Option one (primary) For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message. @@ -83,7 +83,7 @@ For delete (redact action) message with attachments, we must also apply `redact` I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. -### Option two +### Option two (secondary) As lite alternative to option one, we can send only one event with direct links to all attached media, instead of sending separate event for each attachment, because it will give less "spam" for room. Eg when user is sending message with 20 attachments - it will send only one event to room, instead of 21 like in option one implementation. But the main problem with this option is *fallback*. From 55db2fe0c3bbdbdc741ec5d177b8d05e36310be2 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Fri, 26 Feb 2021 18:39:06 +0300 Subject: [PATCH 35/39] Named options --- proposals/2881-message-attachments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index e0822f5aec5..69032bca886 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -13,7 +13,7 @@ Messages with multiple attachments now already implemented in many messengers, f ## Proposal -### Option one (primary) +### Option one "Best fallback" For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message. @@ -83,7 +83,7 @@ For delete (redact action) message with attachments, we must also apply `redact` I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. -### Option two (secondary) +### Option two "Best implementation" As lite alternative to option one, we can send only one event with direct links to all attached media, instead of sending separate event for each attachment, because it will give less "spam" for room. Eg when user is sending message with 20 attachments - it will send only one event to room, instead of 21 like in option one implementation. But the main problem with this option is *fallback*. From 9ebce4ad236fcc60cb960cb4a84581b9bdf5bb25 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 12 Apr 2021 18:08:43 +0300 Subject: [PATCH 36/39] Grammar fixes Co-authored-by: Kevin Cox --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 69032bca886..4efdf94a7d1 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -2,7 +2,7 @@ *This MSC is especially for media image attachments to message, but I try to make it extendable for multiple attachment types (files, videos, and in future - external URLs, links to other Matrix events, etc). So, in most of examples, I am using "image", but it means, that, instead of image, there may be another attachment type.* -In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases even one media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to his text message when he is composing it, not after or before it. +In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases even one media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to their text message when composing it, not after or before it. And now the user can send images only before the message (or after it) as a separate message, but he can't attach images during the composing process to send them when the text is finished, together with the text message in one event. From 633160a78736929e95cdf832ec9c50073b23f5e4 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 12 Apr 2021 18:08:54 +0300 Subject: [PATCH 37/39] Grammar fixes Co-authored-by: Kevin Cox --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 4efdf94a7d1..695bcc0ac5b 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -4,7 +4,7 @@ In the current implementation each media (file, image, video) can be sent only via a separate event to the room. But in most cases even one media is not sent alone, it must be commented via some text by the sender. So the user often wants to attach some images (one or several) directly to their text message when composing it, not after or before it. -And now the user can send images only before the message (or after it) as a separate message, but he can't attach images during the composing process to send them when the text is finished, together with the text message in one event. +And now the user can send images only before the message (or after it) as a separate message, but they can't attach images during the composing process to send them when the text is finished, together with the text message in one event. On the display side, when the user sends multiple images, the problem is that each image is displayed alone, as separate event with full width in timeline, not linked to the message, and not grouped to the gallery. From d77ba1ef3c8db4ca6a88d584a296ef31aaa606ce Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Mon, 12 Apr 2021 18:09:48 +0300 Subject: [PATCH 38/39] Grammar fixes Co-authored-by: Kevin Cox --- proposals/2881-message-attachments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 695bcc0ac5b..801444898f9 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -169,7 +169,7 @@ Editing interface can be represented exactly like the composer interface, where ### Display recommendations: -On the client site, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. +On the client side, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. From 7d4542f742443d80799e3ecaf8c49633fba8efa4 Mon Sep 17 00:00:00 2001 From: Alexey Murz Korepov Date: Tue, 7 Sep 2021 10:09:40 +0300 Subject: [PATCH 39/39] Link to MSC3051 and separate alternative version to 3382 Option two is moved to https://github.com/matrix-org/matrix-doc/pull/3382 Also added link to https://github.com/matrix-org/matrix-doc/pull/3051 with array implementation of relations. --- proposals/2881-message-attachments.md | 130 ++------------------------ 1 file changed, 6 insertions(+), 124 deletions(-) diff --git a/proposals/2881-message-attachments.md b/proposals/2881-message-attachments.md index 801444898f9..efcabc97309 100644 --- a/proposals/2881-message-attachments.md +++ b/proposals/2881-message-attachments.md @@ -10,16 +10,15 @@ On the display side, when the user sends multiple images, the problem is that ea Messages with multiple attachments now already implemented in many messengers, for example - in Skype, Slack, VK Messenger. And Matrix, because lack of support, now have problems with bridging those messages to Matrix rooms. +This is a "best fallback" version of multiple attachments implementation, more optimal implementation is moved to separate [MSC3382: Inline message Attachments](https://github.com/matrix-org/matrix-doc/pull/3382). ## Proposal -### Option one "Best fallback" - For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message. With having this feature, Matrix client should allow users to attach one or multiple media (images, video, files) to message on client side, without instant sending of them to room, and send them together with text message. -When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery. +When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery. Currently the `m.relates_to` is object that is bad for extensability, but here is [MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-doc/pull/3051) that fixes this. For exclude showing those events in modern clients before grouping event added, I propose extend separate media events via adding "marker" field `is_attachment: true`, if clients got this value - they must exclude showing this media in timeline, and shows them only in gallery with grouping event. @@ -79,122 +78,7 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment event too. -#### Fallback: - -I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. - -### Option two "Best implementation" - -As lite alternative to option one, we can send only one event with direct links to all attached media, instead of sending separate event for each attachment, because it will give less "spam" for room. Eg when user is sending message with 20 attachments - it will send only one event to room, instead of 21 like in option one implementation. But the main problem with this option is *fallback*. - -This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements. - -Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field: - -```json -{ - "type": "m.room.message", - "content": { - "msgtype": "m.text", - "body": "Here is my photos and videos from yesterday event", - "m.attachments": [ - { - "msgtype": "m.image", - "url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg", - "body": "Image 1.jpg", - "info": { - "mimetype": "image/jpg", - "size": 1153501, - "w": 963, - "h": 734, - } - }, - { - "msgtype": "m.video", - "url": "mxc://example.com/KUAQOe1GECk2TgdtedkftISg", - "body": "Video 2.mp4", - "info": { - "mimetype": "video/mp4", - "size": 6615304, - "w": 1280, - "h": 720, - } - } - ] - } -} -``` - -#### Fallback - -For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation: -```html -

Here is my photos and videos from yesterday event

-
-

Attachments:

- -
-``` -and JSON of `content` field: -```json -"content": { - "msgtype": "m.text", - "body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18", - "format": "org.matrix.custom.html", - "formatted_body": "

Here is my photos and videos from yesterday event

\n

Attachments:

\n
" - } -``` -And modern clients, that have support of the attachments feature, will cut the `div.mx-attachment` tag and replace it to rich gallery block with thumbnails of attachments. - -If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions. - -**If we will come up with better fallback display, maybe bring this option as main suggestion?** - - -## Client support - -### Compose recommendations: - -In the message composer, on "upload file" or "paste file from clipboard" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API yet ([MSC2278: Deleting attachments for expired and redacted messages](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md), so server will store each file, even if it is not attached to the message.* - -On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array (option one); or collect all `mxc` urls in `m.attachments` array on option two. - -If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation. - -Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can rearrange attachments, remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media in option one, or remove item from `m.attachments` in option two; and delete media file using [MSC2278](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md)), add new attachment (that will upload it as new event with refer to it in edited message `m.relates_to` array in option one / added to `m.attachments` in option two). - - -### Display recommendations: - -On the client side, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid. - -If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages. - -Example of composer interface implementation with multiple attachments we can lookup in [Slack](https://slack.com/), [VK Messenger](https://vk.com/messenger), [Skype](https://skype.com). - -For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery. - -Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side. - -## Server support - -This MSC does not need any changes on server side. - - -## Potential issues - -1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered. - -2. In option one - individual media event, to which `m.message` refers, can be deleted (redacted) after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display. - -3. In option one - there are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server. - -4. If client attach too much media to one message, he can got rate limiting problem on server side. This can be solved via splitting and delaying send of attachments, to match server rate limits. - -## Alternatives + ## Alternatives 1. An alternative can be embedding images (and other media types) directrly into message body via html tags to "body" field (`` or ``), but this will make composing, extracting and stylizing of the attachments harder. @@ -226,11 +110,9 @@ We still can implement both things together (my *MSC2881* as one main text with ``` But this way will give harder way to render of main message event, because Matrix clients must do the search of all attached events manually in timeline, and server will be unable to aggregate them to main message. +#### Fallback: -## Future considerations - -In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment). - +I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message. ## Unstable prefix -Clients should use `org.matrix.msc2881.m.attachments`, `org.matrix.msc2881.m.attachment` and `org-matrix-msc2881-mx-attachments` strings instead of proposed, while this MSC has not been included in a spec release. +Clients should use `org.matrix.msc2881.m.attachment` string instead of proposed, while this MSC has not been included in a spec release.