From 93758c21e58843ca79e6e16c5b1d938a6ff0730f Mon Sep 17 00:00:00 2001 From: Luke Elmers Date: Tue, 22 Jun 2021 14:59:54 -0600 Subject: [PATCH 1/3] Update corrupt SO docs to include removal of index write blocks. --- .../setup/upgrade/upgrade-migrations.asciidoc | 47 +++++++++++++++---- 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/setup/upgrade/upgrade-migrations.asciidoc b/docs/setup/upgrade/upgrade-migrations.asciidoc index fdcd71791ad3a..396c69a426a8a 100644 --- a/docs/setup/upgrade/upgrade-migrations.asciidoc +++ b/docs/setup/upgrade/upgrade-migrations.asciidoc @@ -55,22 +55,53 @@ This section highlights common causes of {kib} upgrade failures and how to preve There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index. This can cause Kibana to log errors like: -> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] -> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] + +[source,sh] +-------------------------------------------- +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] + +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] +-------------------------------------------- See https://github.com/elastic/kibana/issues/95321 for instructions to work around this issue. [float] ===== Corrupt saved objects -We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed. +We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. + +Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed. For example, given the following error message: -> Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. -The following steps must be followed to allow the upgrade migration to succeed. -Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` will no more be available: -1. Delete the corrupt document with `DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275` -2. Restart {kib} +[source,sh] +-------------------------------------------- +Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. +-------------------------------------------- + +The following steps must be followed to allow the upgrade migration to succeed: + +. Remove the write block which the migration system has placed on the index: ++ +[source,sh] +-------------------------------------------- +PUT .kibana_7.12.1_001/_settings +{ + "index": { + "blocks.write": false + } +} +-------------------------------------------- + +. Delete the corrupt document: ++ +[source,sh] +-------------------------------------------- +DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 +-------------------------------------------- + +. Restart {kib}. + +Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` **will no longer be available**. [float] ===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings From 7e90e1e180bede62a6cb44e433f8622a974a7f48 Mon Sep 17 00:00:00 2001 From: Luke Elmers Date: Wed, 23 Jun 2021 17:16:32 -0600 Subject: [PATCH 2/3] Update based on initial feedback. Co-authored-by: Christiane (Tina) Heiligers --- docs/setup/upgrade/upgrade-migrations.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup/upgrade/upgrade-migrations.asciidoc b/docs/setup/upgrade/upgrade-migrations.asciidoc index 396c69a426a8a..245e660d24b30 100644 --- a/docs/setup/upgrade/upgrade-migrations.asciidoc +++ b/docs/setup/upgrade/upgrade-migrations.asciidoc @@ -101,7 +101,7 @@ DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab . Restart {kib}. -Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` **will no longer be available**. +Please be aware the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**. [float] ===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings From b497d5a7057c83db4cb3c81baae953ffe600cf43 Mon Sep 17 00:00:00 2001 From: Luke Elmers Date: Thu, 24 Jun 2021 16:39:06 -0600 Subject: [PATCH 3/3] Update based on feedback. --- docs/setup/upgrade/upgrade-migrations.asciidoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/setup/upgrade/upgrade-migrations.asciidoc b/docs/setup/upgrade/upgrade-migrations.asciidoc index 245e660d24b30..947043b21ef50 100644 --- a/docs/setup/upgrade/upgrade-migrations.asciidoc +++ b/docs/setup/upgrade/upgrade-migrations.asciidoc @@ -78,9 +78,9 @@ For example, given the following error message: Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. -------------------------------------------- -The following steps must be followed to allow the upgrade migration to succeed: +The following steps must be followed to delete the document that is causing the migration to fail: -. Remove the write block which the migration system has placed on the index: +. Remove the write block which the migration system has placed on the previous index: + [source,sh] -------------------------------------------- @@ -101,7 +101,9 @@ DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab . Restart {kib}. -Please be aware the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**. +In this example, the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**. + +Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying. [float] ===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings