Add important warning note to backup .kibana index before upgrading

[ppf2]

Kibana version: 6.5

Given that we are doing some major migration/rewrite of the .kibana index automatically (https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html#upgrade-migrations-old-indices) when upgrading to 6.5 and we are not automatically backing up the original .kibana index for troubleshooting or unexpected failures, it is important for the end users to backup the .kibana index before performing the upgrade.

"When changes are necessary, a new incremental .kibana_N index is created with updated mappings, then the saved objects are loaded in batches from the existing index, transformed to whatever extent necessary, and added to this new index."

"Saved objects are stored in an index named .kibana_N, where N is a number that increments over time as Kibana is upgraded."

While we keep kibana_N around, the above statements suggest that .kibana_1 is already a modified version of the original so isn't a good rollback index to the original.

Currently in our upgrade guide, it simply says "Backup your data using the Elasticsearch snapshots feature. "
https://www.elastic.co/guide/en/kibana/current/upgrade.html#upgrade

We should explicitly call out taking a snapshot of the .kibana index before upgrading (not just the end user's data indices).

[ppf2]

@AlonaNadler

[jpcarey]

I don't feel that just documenting a snapshot backup (not all users have this setup) is enough, it should be automatic.
#20674

[ppf2]

If .kibana_1 is actually the "original" backup (without any modifications), let's clarify that in the documentation :)

[AlonaNadler]

Adding @elastic/kibana-operations which is now working on the upgrade assistant

[epixa]

.kibana_1 is a straight reindex (no migrations) from .kibana, so it's the backup. We created it that way intentionally. So folks shouldn't need to manually back anything up, though doing so before upgrades is and has always been a good idea.

We can clarify docs about this. There is a note in https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html about the initial migration, but it's probably worth clarifying a bit.

[gchaps]

@jbudz Can you please provide some guidance on how to handle this issue in the docs?

[jbudz]

Probably something at the end of the first paragraph to the tune of what court `mentioned.

It's always recommended to have backups on hand prior to upgrading (something about Kibana upgrades automatically when starting a new version). https://www.elastic.co/guide/en/kibana/current/snapshot-repositories.html can be used to backup Kibana's data by targeting .kibana* indices.

I don't think we need to add other scenarios, they may help, but the end result should be to always have backups on hand.

Unrelated but related I think the general approach in the document is at odds with user friendly automatic upgrades. I think the troubleshooting section should match exact errors with resolution steps, and the internals description should be removed.