[DPE-4656] add TLS CA rotation routine

[reneradoi]

Issue

When a new TLS certificate authority (CA) certificate is issued, the opensearch-operator should add this new CA to all its units and request new certificates. The new certificates (including the CA certificate) should be distributed to all OpenSearch nodes in a rolling restart manner, without downtime to the entire cluster.

Due to limitations on the self-signed-certificates operator it is not possible to:

• get a notice if a CA certificate is about to expire
• request a new CA when the current one is about to or has expired
• request an intermediate CA and sign future certificates with it

There is currently no support for renewing a root / CA certificate on the self-signed-certificates operator. A new root / CA certificate will only be generated and issued if the common_name of the CA changes.

We have decided to implement the logic in that way that we check each certificate if it includes a new CA. If so, we store the new CA and initiate the CA rotation workflow on OpenSearch.

Solution

This PR implements the following workflow:

• check each CertificateAvailableEvent if it includes a new CA
• add the new CA to the truststore
• add a notice tls_ca_renewing to the unit's peer data
• initiate a restart of OpenSearch (using the locking mechanism to coordinate cluster availability during the restart)
• after restarting, add a notice tls_ca_renewed to the unit's peer data
• when the restart is done on all of the cluster nodes, request new TLS certificates and apply them to the node

During the phase of renewing the CA, all incoming CertificateAvailableEvents will be deferred in order to avoid incompatibilites in communication between the nodes.

Please also see the flow of events and actions that has been documented here: https://github.com/canonical/opensearch-operator/wiki/TLS-CA-rotation-flow

Notes

• There is a dependency to [DPE-4575][DPE-4886][DPE-4983] Add voting exclusions management #367 because during the rolling restart when the CA is rotated it is very likely that the voting exclusion issue shows up (at least in 3-node-clusters). Therefore the integration test is currently running only with two nodes. Once the voting exclusions issue is resolved, this can be updated to the usual three nodes.
• Due to an upstream bug with JDK it is necessary to use TLS v1.2 (more details see [Bug] SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment opensearch-project/security#3299).
• This PR introduces a method to append configuration to the jvm options file of OpenSearch (used to set TLS config to v1.2).

[juditnovak]

Unittests are added :-)

I'm waiting for the ongoing issue resolved (the one that @skourta have identified on certificate-available event being forever defer()-red). Since that's the very same part of the code where the double app-admin certificate fetch is happening (see the issue highlighted in the unittests), it may be a good idea to fix that part of the workflow as well.

[skourta]

I met with @reneradoi and explained the issue and its source to him. He's on it.

[juditnovak]

Wonderful work, congrats @reneradoi :-)

I'm checking locally if #417 may still hold (I suspect it should not as the code went through related changes since the issue was raised).

Regardless, this is a big chunk of work, strongly tested in numerous ways (incl. a lot of manual testging, against 2 TLS charms).

High time to merge and if any bugs are to be fixed those can be addressed in smaller (clearer :-) ) tasks.

[juditnovak]

Important to mention withing the highlights of earlier discussions.
With the new unittests, instead of the original 55% coverage we've ended up at a 62% unittes coverage :-) :-) :-)