CLOUDSTACK-9993: Securing Agents Communications #2239
Conversation
|
Pinging for review @nvazquez @borisstoyanov @DaanHoogland @wido @marcaurele @GabrielBrascher @rafaelweingartner @ustcweizhou @pyr and others @blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-961 |
d9cac4e to
2fc5234
Compare
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-962 |
78c740c to
2a67c3c
Compare
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-963 |
|
@blueorangutan test |
|
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
Trillian test result (tid-1380)
|
|
@blueorangutan test centos6 vmware-55u3 |
|
@rhtyd a Trillian-Jenkins test job (centos6 mgmt + vmware-55u3) has been kicked to run smoke tests |
|
Trillian test result (tid-1381)
|
|
The one test failing The failure was caused due to; |
|
What I am missing is that it's enforced/mandatory for all Agents, right? What if somebody doesn't want to enable this and simply wants to run unencrypted. Why shouldn't we make a opt-out somewhere? |
|
@wido in current implementation -- all agent-mgmt server connections are encrypted and SSL enabled, based on a random cert that the mgmt server creates, stores and uses from With this new work/PR -- for existing environments, after upgrade the auth strictness enforcement will be |
|
Ok, good! Because I know many (like us) do not use SSH from the mgmt server to add a new KVM host, but simply generate a agent.properties and add it to the mgmt server. |
|
@wido okay, so for you environment just keep |
2a67c3c to
f1ad7ef
Compare
4d421ce to
30b576d
Compare
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1012 |
|
@blueorangutan test centos7 xenserver-65sp1 |
|
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + xenserver-65sp1) has been kicked to run smoke tests |
| throw new CloudRuntimeException("Failed to find default configured CA provider plugin"); | ||
| } | ||
| return configuredCaProvider; | ||
| } |
There was a problem hiding this comment.
What about refactoring method a little bit to something like
if (configuredCaProvider != null) return configuredCaProvider
else if (caProviderMap.containsKey(..) && caProviderMap.get(..) != null) return caProviderMap.get(..);
else throw new CloudRuntimeException("..")
There was a problem hiding this comment.
@nvazquez sure, however if the second else if, we want to set and then return, it would be the same kind of code again.
There was a problem hiding this comment.
You are right, sorry I ommited set line, then makes no sense to refactor in my opinion
There was a problem hiding this comment.
@nvazquez thanks, anyway I made a change
6712edb to
d6c7a34
Compare
|
Thanks @DaanHoogland @nvazquez for reviewing |
|
Thanks @borisstoyanov , I'll keep this open for now until we receive the xenserver smoke test results. I'll merge this on satisfactory results. |
|
Trillian test result (tid-1420)
|
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1023 |
|
@blueorangutan test centos7 xenserver-65sp1 |
|
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + xenserver-65sp1) has been kicked to run smoke tests |
|
@mlsorensen yes if @wido wants to use the new secure system he can (1) create a keystore at /etc/cloudstack/agent/cloud.jks with a 'cloud' alias and password at agent.properties file, and (2) create a CSR using the keystore, (3) use the CSR to issue a certificate from management server and save the certificate and ca-certs and import them into the keystore, (4) add/start the agent to a zone/cluster. I'll document the usage of these scripts, in admin docs or somewhere, briefly here's how these scripts work: (the command name and options in <> brackets)
After the certificate (ca+client) are created, they are stored in the keystore file using Running the import-script would save the certs in the keystore file |
|
Thanks. I realize the initial question was more along the lines of "will this break how we do things now?", but it's good to know that it can be incorporated into that mode of operation as well if people choose. |
|
Trillian test result (tid-1424)
|
ACS CI BVT RunSumarry: Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0 Failed tests:
Skipped tests: Passed test suits: |
This introduces a new certificate authority framework that allows pluggable CA provider implementations to handle certificate operations around issuance, revocation and propagation. The framework injects itself to `NioServer` to handle agent connections securely. The framework adds assumptions in `NioClient` that a keystore if available with known name `cloud.jks` will be used for SSL negotiations and handshake. This includes a default 'root' CA provider plugin which creates its own self-signed root certificate authority on first run and uses it for issuance and provisioning of certificate to CloudStack agents such as the KVM, CPVM and SSVM agents and also for the management server for peer clustering. Additional changes and notes: - Comma separate list of management server IPs can be set to the 'host' global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get radomized comma separated list to which they will attempt connection or reconnection in provided order. This removes need of a TCP LB on port 8250 (default) of the management server(s). - All fresh deployment will enforce two-way SSL authentication where connecting agents will be required to present certificates issued by the 'root' CA plugin. - Existing environment on upgrade will continue to use one-way SSL authentication and connecting agents will not be required to present certificates. - A script `keystore-setup` is responsible for initial keystore setup and CSR generation on the agent/hosts. - A script `keystore-cert-import` is responsible for import provided certificate payload to the java keystore file. - Agent security (keystore, certificates etc) are setup initially using SSH, and later provisioning is handled via an existing agent connection using command-answers. The supported clients and agents are limited to CPVM, SSVM, and KVM agents, and clustered management server (peering). - Certificate revocation does not revoke an existing agent-mgmt server connection, however rejects a revoked certificate used during SSL handshake. - Older `cloudstackmanagement.keystore` is deprecated and will no longer be used by mgmt server(s) for SSL negotiations and handshake. New keystores will be named `cloud.jks`, any additional SSL certificates should not be imported in it for use with tomcat etc. The `cloud.jks` keystore is stricly used for agent-server communications. - Management server keystore are validated and renewed on start up only, the validity of them are same as the CA certificates. New APIs: - listCaProviders: lists all available CA provider plugins - listCaCertificate: lists the CA certificate(s) - issueCertificate: issues X509 client certificate with/without a CSR - provisionCertificate: provisions certificate to a host - revokeCertificate: revokes a client certificate using its serial Global settings for the CA framework: - ca.framework.provider.plugin: The configured CA provider plugin - ca.framework.cert.keysize: The key size for certificate generation - ca.framework.cert.signature.algorithm: The certificate signature algorithm - ca.framework.cert.validity.period: Certificate validity in days - ca.framework.cert.automatic.renewal: Certificate auto-renewal setting - ca.framework.background.task.delay: CA background task delay/interval - ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates Global settings for the default 'root' CA provider: - ca.plugin.root.private.key: (hidden/encrypted) CA private key - ca.plugin.root.public.key: (hidden/encrypted) CA public key - ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate - ca.plugin.root.issuer.dn: The CA issue distinguished name - ca.plugin.root.auth.strictness: Are clients required to present certificates - ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed UI changes: - Button to download/save the CA certificates. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Upgrades bountycastle version and uses newer classes - Refactors SAMLUtil to use new CertUtils Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
d6c7a34 to
f12b00e
Compare
|
The XenServer env related failure was caused by centos7 based packging/dependencies issues where pyopessl.crypto may not have a === TestName: test_issue_certificate_with_csr | Status : SUCCESS === === TestName: test_issue_certificate_without_csr | Status : SUCCESS === === TestName: test_list_ca_certificate | Status : SUCCESS === === TestName: test_list_ca_providers | Status : SUCCESS === === TestName: test_provision_certificate | Status : SUCCESS === === TestName: test_revoke_certificate | Status : SUCCESS === With enough test results and code reviews, this is ready for merging. I'll wait for Travis to go green and then merge this feature. |
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to
NioServerto handle agent connections securely. Theframework adds assumptions in
NioClientthat a keystore if availablewith known name
cloud.jkswill be used for SSL negotiations andhandshake.
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+Agent+Communications
This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.
Additional changes and notes:
global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
radomized comma separated list to which they will attempt connection
or reconnection in provided order. This removes need of a TCP LB on
port 8250 (default) of the management server(s).
connecting agents will be required to present certificates issued
by the 'root' CA plugin.
authentication and connecting agents will not be required to present
certificates.
keystore-setupis responsible for initial keystore setupand CSR generation on the agent/hosts.
keystore-cert-importis responsible for import providedcertificate payload to the java keystore file.
SSH, and later provisioning is handled via an existing agent connection
using command-answers. The supported clients and agents are limited to
CPVM, SSVM, and KVM agents, and clustered management server (peering).
connection, however rejects a revoked certificate used during SSL
handshake.
cloudstackmanagement.keystoreis deprecated and will no longerbe used by mgmt server(s) for SSL negotiations and handshake. New
keystores will be named
cloud.jks, any additional SSL certificatesshould not be imported in it for use with tomcat etc. The
cloud.jkskeystore is stricly used for agent-server communications.
the validity of them are same as the CA certificates.
New APIs:
Global settings for the CA framework:
Global settings for the default 'root' CA provider:
UI changes: