[Upgrade assistant]Upgrade assistant doesn't format the date format exception and doesn't show remediation steps

[bhavyarm]

Kibana version: 8.18.0 Snapshot

Elasticsearch version: 8.18.0 Snapshot

Browser version: Chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from snapshots

Describe the bug: If user has indices with older compat format fields, upgrade assistant shows critical deprecation issue for this and reindex fails (requires manual script correction(https://www.elastic.co/blog/locale-changes-elasticsearch-8-16-jdk-23) - but doesn't show any remediation steps on the assistant itself.

Steps to reproduce:

1. Ingest data with old data format in 8.16.x

PUT books
{
  "mappings": {
    "properties": {
      "release_date": {
        "type":   "date",
        "format": "EEEE dd MMMM yyyy GGG"
      },
      "release_date_2": {
        "type":   "date",
        "format": "EEEE dd MMMM yyyy h:mm B"
      },
      "release_date_3": {
        "type":   "date",
        "format": "qqqq yyyy"
      },
      "release_date_de": {
        "type":   "date",
        "format": "EEE dd MMM yyyy GGGGG",
        "locale": "de"
      },
      "release_date_ja": {
        "type":   "date",
        "format": "dd MMM yyyy",
        "locale": "ja"
      }
    }
  }
}

Ingest some sample documents:

POST /_bulk
{"index":{"_index":"books"}}
{"name":"Revelation Space","author":"Alastair Reynolds","release_date":"Tuesday 17 September 2024 AD","release_date_2":"Tuesday 17 September 2024 7:30 AM","release_date_3":"3 2024","release_date_de":"Di 17 Sep 2024 A","release_date_ja":"17 9 2024","page_count":585}
{"index":{"_index":"books"}}
{"name":"1984","author":"George Orwell","release_date":"Wednesday 18 September 2024 AD","release_date_2":"Wednesday 18 September 2024 8:30 AM","release_date_3":"1 2024","release_date_de":"Mi 18 Sep 2024 A","release_date_ja":"18 9 2024","page_count":328}
{"index":{"_index":"books"}}
{"name":"Fahrenheit 451","author":"Ray Bradbury","release_date":"Monday 16 September 2024 AD","release_date_2":"Monday 16 September 2024 8:30 AM","release_date_3":"2 2024","release_date_de":"Mo 16 Sep 2024 A","release_date_ja":"16 9 2024","page_count":227}
{"index":{"_index":"books"}}
{"name":"Brave New World","author":"Aldous Huxley","release_date":"Sunday 15 September 2024 AD","release_date_2":"Sunday 15 September 2024 5:30 AM","release_date_3":"2 2024","release_date_de":"So 16 Sep 2024 A","release_date_ja":"15 9 2024","page_count":268}
{"index":{"_index":"books"}}
{"name":"The Handmaids Tale","author":"Margaret Atwood","release_date":"Saturday 14 September 2024 AD","release_date_2":"Saturday 14 September 2024 4:30 AM","release_date_3":"4 2024","release_date_de":"Sa 16 Sep 2024 A","release_date_ja":"14 9 2024","page_count":311}

2. Upgrade to 8.18.x
3. The upgrade assistant shows books index needs to be reindexed and if you click on reindex fails with an error which isn't formatted and no remediation steps.

Error:

Reindexing error

Error: Reindexing failed: {"completed":true,"task":{"node":"IYA5ReTVTKKo38sGA452lw","id":745759,"type":"transport","action":"indices:data/write/reindex","status":{"total":5,"updated":0,"created":0,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0},"description":"reindex from [books] to [reindexed-v8-books]","start_time_in_millis":1734453342305,"running_time_in_nanos":12691250,"cancellable":true,"cancelled":false,"headers":{"X-elastic-product-origin":"kibana","trace.id":"d8ab0425ed518bf6b78e54b58d03d1f5","X-Opaque-Id":"7ffc4a30-e5e3-43eb-b491-c8c6952defc8;kibana:application:management:"}},"response":{"took":12,"timed_out":false,"total":5,"updated":0,"created":0,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled":"0s","throttled_millis":0,"requests_per_second":-1,"throttled_until":"0s","throttled_until_millis":0,"failures":[{"index":"reindexed-v8-books","id":"uKrwC5IBJd_B63XbLGr-","cause":{"type":"document_parsing_exception","reason":"[1:173] failed to parse field [release_date_3] of type [date] in document with id 'uKrwC5IBJd_B63XbLGr-'. Preview of field's value: '3 2024'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [3 2024] with format [qqqq yyyy]","caused_by":{"type":"date_time_parse_exception","reason":"Text '3 2024' could not be parsed at index 0"}}},"status":400},{"index":"reindexed-v8-books","id":"uarwC5IBJd_B63XbLGr-","cause":{"type":"document_parsing_exception","reason":"[1:161] failed to parse field [release_date_3] of type [date] in document with id 'uarwC5IBJd_B63XbLGr-'. Preview of field's value: '1 2024'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [1 2024] with format [qqqq yyyy]","caused_by":{"type":"date_time_parse_exception","reason":"Text '1 2024' could not be parsed at index 0"}}},"status":400},{"index":"reindexed-v8-books","id":"uqrwC5IBJd_B63XbLGr-","cause":{"type":"document_parsing_exception","reason":"[1:164] failed to parse field [release_date_3] of type [date] in document with id 'uqrwC5IBJd_B63XbLGr-'. Preview of field's value: '2 2024'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2 2024] with format [qqqq yyyy]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2 2024' could not be parsed at index 0"}}},"status":400},{"index":"reindexed-v8-books","id":"u6rwC5IBJd_B63XbLGr-","cause":{"type":"document_parsing_exception","reason":"[1:166] failed to parse field [release_date_3] of type [date] in document with id 'u6rwC5IBJd_B63XbLGr-'. Preview of field's value: '2 2024'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2 2024] with format [qqqq yyyy]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2 2024' could not be parsed at index 0"}}},"status":400},{"index":"reindexed-v8-books","id":"vKrwC5IBJd_B63XbLGr-","cause":{"type":"document_parsing_exception","reason":"[1:175] failed to parse field [release_date_3] of type [date] in document with id 'vKrwC5IBJd_B63XbLGr-'. Preview of field's value: '4 2024'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [4 2024] with format [qqqq yyyy]","caused_by":{"type":"date_time_parse_exception","reason":"Text '4 2024' could not be parsed at index 0"}}},"status":400}]}}

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Given that the dates need to be reindexed with a custom script, ES should not raise a deprecation for the index compatibility format but instead raise a deprecation for the custom date formatting. That way a user will be prompted the fix their data and in the process also resolve the index compatibility version

[rudolf]

I can reproduce the problem described in the issue, but what's strange is like in the screenshot, there is no deprecation warning from Elasticsearch for the incompatible date format specifiers.

I've reached out to the Elasticsearch team for clarification.

If we want to solve this bug we'd need upstream changes to Elasticsearch:

1. Elasticsearch should raise a deprecation for incompatible date format specifiers
2. Elasticsearch should not raise an "incompatible index version" deprecation for indices with incompatible date format specifiers

These two things will ensure that upgrade assistant guides users to reindex with a script that solves the date format specifier problem and so at the same time address the incompatible index version.

[rjernst]

Extracting out the underlying reindex failure:

{"type":"illegal_argument_exception","reason":"failed to parse date field [3 2024] with format [qqqq yyyy]"

Regarding the setup:

1. Ingest data with old data format in 8.16.x

Are you sure the documents causing reindexing to fail didn't exist prior to 8.16 in this scenario? 8.16 uses JDK 23 and CLDR. There should be no problem reindexing, since nothing has changed then between 8.16 and 8.18.

Note that the need for reindex was not driven by the COMPAT->CLDR change; that is independent, and migration assistant doesn't know anything about it. The change to CLDR was forced upon ES by a change in the JDK, that's why it happened in a minor version. The reindex message you see should be because the index was created in 7.x. If the index was created in 8.16, upgrade assistant should not suggest reindexing is required.

Since the CLDR change occurred in a minor release, there's not a whole lot we can do here. The deprecation logging was put in place to help where it could, in the specific case of 8.15 where COMPAT was still used.

Elasticsearch should raise a deprecation for incompatible date format specifiers

The date format specifiers in question are not incompatible, per se. They are valid specifiers, and can continue to be used.

Elasticsearch should not raise an "incompatible index version" deprecation for indices with incompatible date format specifiers

The index is incompatible though, it needs to be reindexed. In this case it needs manual reindexing, but the warning is still valid and should be shown to the user.

Could the migration assistant have a custom warning for this case? We could tweak the incompatible index version deprecation to check for problematic fields and emit the different "deprecation".

[rudolf]

If the index was created in 8.16, upgrade assistant should not suggest reindexing is required.

Yes, I had to ingest the documents in 7.17 when reproducing this.

The index is incompatible though, it needs to be reindexed. In this case it needs manual reindexing, but the warning is still valid and should be shown to the user.
Could the migration assistant have a custom warning for this case? We could tweak the incompatible index version deprecation to check for problematic fields and emit the different "deprecation".

Agreed. Upgrade assistant renders the deprecation, details and url from the ES response. For 8.18, if ES tweaked the copy in these fields similar to what ML does for transform destination indices elastic/elasticsearch#122192 then that would go a long way to help users understand what's wrong and how to fix it.