Sometime, when I open a mailbox, most entries are ommited

[chibenwa]

Description

mailto-2022-03-14_09.58.30.mp4

Opening my JIRA mailbox only showed me one old mail...

I had to set + unset a filter to force it displaying newest entries.

It's ok to do Email/query + Email/get for the first time I open a mailbox.

[Arsnael]

I just detected that issue on the ios app as well on my phone

[hoangdat]

hi, I found a weird problem in dev. I filter notKeyword: $seen, but some the responses have the keyword $seen: true.

Request

{
    "using": [
        "urn:ietf:params:jmap:core",
        "urn:ietf:params:jmap:mail"
    ],
    "methodCalls": [
        [
            "Email/query",
            {
                "accountId": "{{accountId}}",
                "limit": 20,
                "filter": {
                    "inMailbox": "{{Sent}}",
                    "notKeyword": "$seen"
                },
                "sort": [{
                  "property":"sentAt",
                  "isAscending": false
                }],
                "position": 0
            },
            "c2"
        ],
        [
            "Email/get",
            {
                "accountId": "{{accountId}}",
                "properties": [
                    "subject",
                    "from",
                    "receivedAt",
                    "sentAt",
                    "preview",
                    "hasAttachment",
                    "mailboxIds",
                    "keywords"
                ],
                "#ids": {
                    "resultOf": "c2",
                    "name": "Email/query",
                    "path": "ids/*"
                }
            },
            "c3"
        ]
    ]
}

Response

{
    "sessionState": "2c9f1b12-b35a-43e6-9af2-0106fb53a943",
    "methodResponses": [
        [
            "Email/query",
            {
                "accountId": "3ce33c876a726662c627746eb9537a1d13c2338193ef27bd051a3ce5c0fe5b12",
                "queryState": "11dc2cfd",
                "canCalculateChanges": false,
                "ids": [
                    "c331d3c0-90ac-11ec-984e-e3f8b83572b4",
                    "36128e70-32e8-11ec-a759-2fef1ee78d9e",
                    "4b0db860-32e6-11ec-a759-2fef1ee78d9e",
                    "4f769140-2cc6-11ec-a759-2fef1ee78d9e",
                    "3b21cb60-2cc6-11ec-a759-2fef1ee78d9e",
                    "254999d0-2cc6-11ec-a759-2fef1ee78d9e",
                    "19707570-2cc6-11ec-a759-2fef1ee78d9e",
                    "0e877d70-2cc6-11ec-a759-2fef1ee78d9e",
                    "8b5a5280-2cc3-11ec-a759-2fef1ee78d9e",
                    "826ba3e0-2cc3-11ec-a759-2fef1ee78d9e",
                    "5f771f90-2cc3-11ec-a759-2fef1ee78d9e"
                ],
                "position": 0
            },
            "c2"
        ],
        [
            "Email/get",
            {
                "accountId": "3ce33c876a726662c627746eb9537a1d13c2338193ef27bd051a3ce5c0fe5b12",
                "notFound": [],
                "state": "c55b33d0-a52e-11ec-90ba-e169393c493e",
                "list": [
                    {
                        "preview": "------- Forwarded message ------- Subject: [linshare-mobile-flutter-app-android-release] - Build [success] Date: October 8, 2021 12:50 PM From: dphamhoang@linagora.com To: laisam114@gmail.com <laisam114@gmail.com>, userb@qa.open-paas.org <userb@qa.open-paa",
                        "hasAttachment": false,
                        "keywords": {
                            "$seen": true
                        },
                        "subject": "Fwd: [linshare-mobile-flutter-app-android-release] - Build [success]",
                        "mailboxIds": {
                            "abb99c10-18d9-11eb-a677-2990b970028d": true
                        },
                        "from": [
                            {
                                "name": "Lê Nguyễn User B",
                                "email": "userb@qa.open-paas.org"
                            }
                        ],
                        "sentAt": "2021-10-14T07:51:15Z",
                        "id": "826ba3e0-2cc3-11ec-a759-2fef1ee78d9e",
                        "receivedAt": "2021-10-14T07:51:15Z"
                    },
 ....

Note: this mailbox is Sent

[hoangdat]

Maybe I am wrong in something? Please help me? I debugged all the day :)

[chibenwa]

Ahah you got a james bug!

Email/query relies on elasticSearch while Email/get relies on cassandra. Getting those two in sync can be an issue and you can have inconsistencies across the 2 data-stores. Now the indexing is done asynchronously which means that if two actions happens veeeery quickly (say read then move emails or vice versa) we can get into a data race in elasticsearch where we do the operation in the wrong order which could result into what you actually see.

Moving emails to sent box does just that: mark as seen and move in one request.

I got an idea of a fix. Basically the key problem is that we use a mutable id as an indexing key which results in poor, non transactional behaviour if indexing requests get reordered. Of course it demands significant redesign of ES layers... And might bring additional concerns!

That will take time. In the mean time we might need the mobile app to account for this.

Now I don't think it relates to the bug described here... neither me nor René pointed this out it in sent mailbox.

[chibenwa]

https://issues.apache.org/jira/browse/JAMES-3728

linagora/james-project#4494

[lecaotunglam]

Passed on CI platform.

bandicam.2022-04-08.18-40-34-191.mp4