Session key backup for minimal downtime migration - outage

[kucharskim]

Is there anyway to export (backup) and then restore (import) session keys in case of urgent node fail-over of Polkadot or Kusama validator?

We are developing disaster recovery, rapid failover mechanisms and we would like to have a passive Polkadot / Kusama instance which naturally syncs with the chain and is not a validator, but when a real validator node is affected by an outage, we would like to stop active validator, reconfigure standby node to re-use it's own local database directory (.../db/), but re-import session key and start validation again.

Is that scenario possible?

[bkchr]

The session keys are also just stored in the file system. So, you are able to backup them. If you don't set an explicit keystore path, check the base path for the keystore.

[kucharskim]

Remote signer is also a solution, which I didn't mention here, as I wanted to focus on import / export of session keys. What would be also practical if import of session keys is easy, could session keys be generated in advance and then later distributes into the node(s), according to our disaster recovery procedures?

[rphmeier]

Import is relatively easy. You'd just copy over the keystore. But note that double-signing penalties are high. You'll want to build a system that restricts access of a set of keys to 1 node at a time

[kucharskim]

Yes, @rphmeier I'm aware of that. Is there a way to generate those session keys by hand?

[burdges]

Just fyi, this is dangerous right now, but it'll become much safer once paritytech/substrate#7398 happens. I should do a clearer write up I guess..