[Question]: Agent installation using "Production" mode of security on a Cloud build.

[amolnater-qasource]

Description

We have successfully installed an agent using "Production" mode of security on a self managed environment as per ticket https://github.com/elastic/observability-test-plans/issues/218 and guide available at https://www.elastic.co/guide/en/fleet/current/secure-connections.html

• [For a self managed environment]: In this mode we attempt to run certutil commands to create required certificates from the extracted elasticsearch\bin folder.

Question?
We have observed that when we add Fleet Server integration to a policy, "Production" mode of security option is also available on Cloud build under Add Agent flyout.

Do we have any other steps for agent installation using "Production" mode on cloud builds as we don't have elasticsearch\bin location for cloud builds?

• We are not able to create certificates for a cloud build, hence we won't be able to run agent installation command.

Collaboration

• The docs team will lead producing the content
• The product team will provide the initial content and the docs team will edit / review
• The docs team will define with product team the structure and location, and the product team will provide the initial content
• Other (please describe)

Contact Person:

(We need to have a contact person in the product/development team to provide information about how the item to be documented works.)
@amolnater-qasource @mostlyjason

Suggested Target Release

TBD

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & assigned to @andresrc

CC: @EricDavisX

[ruflin]

Thanks for raising this. We should likely offer a better experience on this. I suggest we move this to the Kibana repo and ping the Fleet team on it as I don't think the Elastic Agent can do anything here. @mostlyjason @jen-huang

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mostlyjason]

@amolnater-qasource what did you select as your agent policy in this example? You should not be able to choose Elastic Cloud agent policy in the dropdown box. Also, what do you mean by "cloud build"?

[nchaulet]

Just tested with a cloud deployment I had around and you cannot select the default cloud policy from the add agent Flyout You will have this issue if you add fleet server to another policy and select that policy.

[amolnater-qasource]

Hi @mostlyjason

what did you select as your agent policy in this example?

I added Fleet Server integration to a New Policy[User Created] and selected that policy.

Also, what do you mean by "cloud build"?

Cloud build refer to the builds we deploy from: https://staging.found.no/login?redirectTo=%2Fhome
Similarly builds deployed from production and cloud-qa. We don't refer self-managed environments here.

You will have this issue if you add fleet server to another policy and select that policy.

Yes, @nchaulet we refer to this only.

Thanks

[mostlyjason]

It seems like a near term fix is to update our docs to include use cases where the cluster is running Elastic Cloud and the user adds a self-managed Fleet server. This could instruct the user to download the binaries. I think some other language on the page needs to be updated such as the title which says "Encrypt traffic in a self-managed cluster". It should probably say "Encrypt traffic with a self-managed Fleet Server" and it should discuss both self-managed and cloud-managed clusters. I'll add this to our docs tracking board.

Long term, we are looking at ways to simplify this so users don't need to add certs manually https://github.com/elastic/beats/issues/25705

[ollyhowell]

<ignore my comments - was just testing the add to project workflow>

[amolnater-qasource]

Hi @mostlyjason
We have re-attempted to setup fleet-server using production mode of security on 7.16.0 Snapshot cloud-staging environment.

As per guides it is suggest to run the certs command from elasticsearch\bin folder.
So we followed the below steps:

1. Downloaded and extracted the elasticsearch for 7.16.0 Snapshot.
2. Run cert commands and provide dns name and ip for fleet-server certs command.
3. We successfully created all the certs and run agent install command and got below output:

PS C:\Program Files\elastic-agent-7.16.0-SNAPSHOT-windows-x86_64> .\elastic-agent.exe install --url=https://54.163.214.80:8220 `
>> -f `
>> --fleet-server-es=https://b1d13ca868f244f9b2cab2241f4125dd.us-central1.gcp.foundit.no:443 `
>> --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzQyODY0MDAyNTI6YzVhU0ZTS3JSeDJrNFFrMlprUW81Zw `
>> --fleet-server-policy=policy-elastic-agent-on-cloud `
>> --certificate-authorities=C:\elk\elasticsearch-7.16.0-SNAPSHOT\ca\ca.crt `
>> --fleet-server-es-ca=C:\elk\elasticsearch-7.16.0-SNAPSHOT\new\cert.crt `
>> --fleet-server-cert=C:\elk\elasticsearch-7.16.0-SNAPSHOT\fleet-server\fleet-server.crt `
>> --fleet-server-cert-key=C:\elk\elasticsearch-7.16.0-SNAPSHOT\fleet-server\fleet-server.key
2021-10-15T08:36:40.282Z        INFO    cmd/enroll_cmd.go:674   Waiting for Elastic Agent to start
2021-10-15T08:36:41.499Z        INFO    cmd/enroll_cmd.go:691   Waiting for Elastic Agent to start Fleet Server
2021-10-15T08:36:42.598Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Starting
2021-10-15T08:36:44.616Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:36:50.645Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:36:54.713Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:36:55.722Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Starting
2021-10-15T08:36:56.745Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:02.800Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:06.846Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:37:07.863Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Starting
2021-10-15T08:37:08.867Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:14.936Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:18.973Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:37:19.974Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:26.032Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:30.076Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:37:32.099Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:38.129Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:42.170Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:37:43.177Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Starting
2021-10-15T08:37:44.185Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:50.258Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:37:54.271Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:37:55.273Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:01.327Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:05.366Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:38:07.387Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:13.457Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:17.514Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:38:18.527Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Starting
2021-10-15T08:38:19.529Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:25.586Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:29.627Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Restarting
2021-10-15T08:38:30.640Z        INFO    cmd/enroll_cmd.go:724   Fleet Server - Error - x509: certificate signed by unknown authority
2021-10-15T08:38:36.693Z        INFO    cmd/enroll_cmd.go:729   Fleet Server - Error - x509: certificate signed by unknown authority
Error: fleet-server never started by elastic-agent daemon: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.16/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.16/fleet-troubleshooting.html

Note:

• We have tried this with --insecure and without --insecure, however got no success.
• We haven't done any changes in elasticsearch.yml to generate certificates.

Please let us know if we are missing anything, as we are still not able to install fleet-server on cloud-staging kibana using production mode of security.

cc: @EricDavisX

Thanks

[EricDavisX]

The issue here, @amolnater-qasource will point to the other needed setup items for production usage of security:
elastic/beats#28007

• see the requirement that... You will also need to set ssl.verification_mode: none in the Output settings in Fleet and Integrations UI.
  I also think we can move this to a testing ticket elsewhere and report back here on just on the Docs side - questions for Agent engineers won't be seen here necessarily.