Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #17687 to 7.x: [Heartbeat] Add Additional ECS tls.* fields #18029

Merged
merged 4 commits into from
Apr 29, 2020

Conversation

andrewvc
Copy link
Contributor

@andrewvc andrewvc commented Apr 27, 2020

Cherry-pick of PR #17687 to 7.x branch. Original message:

Work in support of elastic/uptime#161

This patch adds additional ECS TLS and x509 fields. Note that we are blocked on the x509 fields which are not yet merged into ECS.

Sample output of the tls.* fields with this patch is below. Note the somewhat strange nesting of data in issuer and subject. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS x509 type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. I did break out the CN into its own field in x509.subject/issuer.common_name. However, if we do want to aggregate on issuer in the future it's good to have the full distinguished name to do that on.

This PR also refactors some libbeat code around parsing TLS versions and adds test coverage there as well.

{
	"tls": {
		"certificate_not_valid_after": "2020-07-16T03:15:39Z",
		"certificate_not_valid_before": "2019-08-16T01:40:25Z",
		"server": {
			"hash": {
				"sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1",
				"sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d"
			},
			"x509": {
				"issuer": {
					"common_name": "GlobalSign CloudSSL CA - SHA256 - G3",
					"distinguished_name": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE"
				},
				"not_after": "2020-07-16T03:15:39Z",
				"not_before": "2019-08-16T01:40:25Z",
				"public_key_algorithm": "RSA",
				"public_key_size": 2048,
				"serial_number": "26610543540289562361990401194",
				"signature_algorithm": "SHA256-RSA",
				"subject": {
					"common_name": "r2.shared.global.fastly.net",
					"distinguished_name": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US"
				}
			}
		}
	}
}

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Verify that the fields here match the ECS spec

How to test this PR locally

Run against TLS/Non-TLS endpoints

Work in support of elastic/uptime#161

This patch adds additional ECS [TLS](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html) and [x509](elastic/ecs#762) fields. Note that we are blocked on the x509 fields which are not yet merged into ECS.

Sample output of the `tls.*` fields with this patch is below. Note the somewhat strange nesting of data in `issuer` and `subject`. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS `x509` type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. I did break out the CN into its own field in `x509.subject/issuer.common_name`. However, if we do want to aggregate on issuer in the future it's good to have the full distinguished name to do that on.

This PR also refactors some `libbeat` code around parsing TLS versions and adds test coverage there as well.

```json
{
	"tls": {
		"certificate_not_valid_after": "2020-07-16T03:15:39Z",
		"certificate_not_valid_before": "2019-08-16T01:40:25Z",
		"server": {
			"hash": {
				"sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1",
				"sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d"
			},
			"x509": {
				"issuer": {
					"common_name": "GlobalSign CloudSSL CA - SHA256 - G3",
					"distinguished_name": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE"
				},
				"not_after": "2020-07-16T03:15:39Z",
				"not_before": "2019-08-16T01:40:25Z",
				"public_key_algorithm": "RSA",
				"public_key_size": 2048,
				"serial_number": "26610543540289562361990401194",
				"signature_algorithm": "SHA256-RSA",
				"subject": {
					"common_name": "r2.shared.global.fastly.net",
					"distinguished_name": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US"
				}
			}
		}
	}
}
```

## How to test this PR locally

Run against TLS/Non-TLS endpoints

(cherry picked from commit eb2dc26)
@andrewvc andrewvc requested a review from a team as a code owner April 27, 2020 21:12
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 27, 2020
@andresrc andresrc added [zube]: Inbox and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 28, 2020
@andrewvc andrewvc added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Apr 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@andrewvc
Copy link
Contributor Author

Jenkins, retest this please

@andrewvc
Copy link
Contributor Author

This includes #18066 (comment) since wildcard type was removed from 7.x ES

@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 28, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-04-28T23:55:28.612+0000

  • Duration: 93 min 38 sec (5557743)

  • Commit: 9a72695

Test stats 🧪

Test Results
Failed 0
Passed 5822
Skipped 902
Total 6724

Steps errors

Expand to view the steps failures

  • Name: Make -C generator/_templates/metricbeat test
    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 3 min 15 sec<

    • Start Time: 2020-04-29T00:40:41.242+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T01:21:05.887Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T01:21:05.887Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T01:21:05.887Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T01:22:36.823Z] 
Creating appsearch_7918a246f6f8_appsearch_1      ... done
.Stopping appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T01:22:36.823Z] Stopping appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T01:22:36.823Z] 
Stopping appsearch_7918a246f6f8_appsearch_1      ... done

Stopping appsearch_7918a246f6f8_elasticsearch_1  ... done
Removing appsearch_7918a246f6f8_appsearch_1      ... 
[2020-04-29T01:22:36.823Z] Removing appsearch_7918a246f6f8_elasticsearch_1  ... 
[2020-04-29T01:22:37.963Z] 
Removing appsearch_7918a246f6f8_elasticsearch_1  ... done

Removing appsearch_7918a246f6f8_appsearch_1      ... done
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T01:22:48.038Z] 
Creating cockroachdb_7918a246f6f8_cockroachdb_1  ... done
.Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T01:22:48.302Z] 
Stopping cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... 
[2020-04-29T01:22:49.264Z] 
Removing cockroachdb_7918a246f6f8_cockroachdb_1  ... done
Creating coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T01:22:58.797Z] 
Creating coredns_7918a246f6f8_coredns_1          ... done
.Stopping coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T01:22:59.104Z] 
Stopping coredns_7918a246f6f8_coredns_1          ... done
Removing coredns_7918a246f6f8_coredns_1          ... 
[2020-04-29T01:23:00.089Z] 
Removing coredns_7918a246f6f8_coredns_1          ... done
Creating ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T01:23:28.601Z] 
Creating ibmmq_7918a246f6f8_ibmmq_1              ... done
.Stopping ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T01:23:28.601Z] 
Stopping ibmmq_7918a246f6f8_ibmmq_1              ... done
Removing ibmmq_7918a246f6f8_ibmmq_1              ... 
[2020-04-29T01:23:29.171Z] 
Removing ibmmq_7918a246f6f8_ibmmq_1              ... done
Creating mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T01:23:49.210Z] 
Creating mssql_7918a246f6f8_mssql_1              ... done
..Stopping mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T01:23:59.466Z] 
Stopping mssql_7918a246f6f8_mssql_1              ... done
Removing mssql_7918a246f6f8_mssql_1              ... 
[2020-04-29T01:24:04.833Z] 
Removing mssql_7918a246f6f8_mssql_1              ... done
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T01:24:11.050Z] 
Creating openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
.Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T01:24:11.625Z] 
Stopping openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... 
[2020-04-29T01:24:13.007Z] 
Removing openmetrics_7918a246f6f8_openmetrics-node_exporter_1 ... done
Creating redisenterprise_3010fd099e8366db_redisenterprise_1   ... 
[2020-04-29T01:26:53.451Z] 
Creating redisenterprise_3010fd099e8366db_redisenterprise_1   ... done
..Stopping redisenterprise_3010fd099e8366db_redisenterprise_1   ... 
[2020-04-29T01:26:57.723Z] 
Stopping redisenterprise_3010fd099e8366db_redisenterprise_1   ... done
Removing redisenterprise_3010fd099e8366db_redisenterprise_1   ... 
[2020-04-29T01:26:58.938Z] 
Removing redisenterprise_3010fd099e8366db_redisenterprise_1   ... done
Creating sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T01:27:18.982Z] 
Creating sql_7918a246f6f8_mysql_1                             ... done
.Stopping sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T01:27:18.982Z] 
Stopping sql_7918a246f6f8_mysql_1                             ... done
Removing sql_7918a246f6f8_mysql_1                             ... 
[2020-04-29T01:27:19.562Z] 
Removing sql_7918a246f6f8_mysql_1                             ... done
Creating stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T01:27:33.861Z] 
Creating stan_7918a246f6f8_stan_1                             ... done
...Stopping stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T01:27:43.904Z] 
Stopping stan_7918a246f6f8_stan_1                             ... done
Removing stan_7918a246f6f8_stan_1                             ... 
[2020-04-29T01:27:50.499Z] 
Removing stan_7918a246f6f8_stan_1                             ... done
.
[2020-04-29T01:27:50.499Z] [success] 59.46% test_xpack_base.Test.test_dashboards: 92.4142s
[2020-04-29T01:27:50.499Z] [success] 4.87% test_activemq.ActiveMqTest_0.test_broker_metrics_collected: 7.5764s
[2020-04-29T01:27:50.499Z] [success] 3.99% test_activemq.ActiveMqTest_1.test_queue_metrics_collected: 6.1993s
[2020-04-29T01:27:50.499Z] [success] 3.96% test_statsd.Test.test_server: 6.1475s
[2020-04-29T01:27:50.499Z] [success] 3.47% test_cockroachdb.Test.test_status: 5.3926s
[2020-04-29T01:27:50.499Z] [success] 2.68% test_xpack_base.Test.test_migration: 4.1690s
[2020-04-29T01:27:50.499Z] [success] 2.09% test_sql.Test.test_query: 3.2518s
[2020-04-29T01:27:50.499Z] [success] 1.81% test_openmetrics.Test.test_openmetrics: 2.8106s
[2020-04-29T01:27:50.499Z] [success] 1.77% test_xpack_base.Test.test_template: 2.7561s
[2020-04-29T01:27:50.499Z] [success] 1.72% test_stan.TestStan.test_metricset_2_subscriptions: 2.6676s
[2020-04-29T01:27:50.499Z] [success] 1.49% test_redisenterprise.Test_0.test_metricset_1_proxy: 2.3137s
[2020-04-29T01:27:50.499Z] [success] 1.46% test_stan.TestStan.test_metricset_1_channels: 2.2663s
[2020-04-29T01:27:50.499Z] [success] 1.42% test_appsearch.Test.test_stats: 2.2121s
[2020-04-29T01:27:50.499Z] [success] 1.07% test_activemq.ActiveMqTest_0.test_topic_metrics_collected: 1.6621s
[2020-04-29T01:27:50.500Z] [success] 1.05% test_redisenterprise.Test_0.test_metricset_0_node: 1.6252s
[2020-04-29T01:27:50.500Z] [success] 0.98% test_activemq.ActiveMqTest_0.test_queue_metrics_collected: 1.5262s
[2020-04-29T01:27:50.500Z] [success] 0.88% test_activemq.ActiveMqTest_1.test_topic_metrics_collected: 1.3738s
[2020-04-29T01:27:50.500Z] [success] 0.87% test_mssql.Test.test_status: 1.3569s
[2020-04-29T01:27:50.500Z] [success] 0.87% test_ibmmq.Test.test_qmgr: 1.3485s
[2020-04-29T01:27:50.500Z] [success] 0.87% test_activemq.ActiveMqTest_1.test_broker_metrics_collected: 1.3455s
[2020-04-29T01:27:50.500Z] [success] 0.86% test_coredns.Test.test_stats: 1.3432s
[2020-04-29T01:27:50.500Z] [success] 0.82% test_stan.TestStan.test_metricset_0_stats: 1.2781s
[2020-04-29T01:27:50.500Z] [success] 0.79% test_mssql.Test.test_performance: 1.2336s
[2020-04-29T01:27:50.500Z] [success] 0.75% test_xpack_base.Test.test_start_stop: 1.1639s
[2020-04-29T01:27:50.500Z] ----------------------------------------------------------------------
[2020-04-29T01:27:50.500Z] Ran 24 tests in 884.920s
[2020-04-29T01:27:50.500Z] 
[2020-04-29T01:27:50.500Z] OK
[2020-04-29T01:27:50.500Z] >> python test: Integration Testing Complete
[2020-04-29T01:27:51.088Z] >> Stopping Docker test environment...
[2020-04-29T01:27:56.426Z] Recording test results
[2020-04-29T01:28:01.780Z] Archiving artifacts
[2020-04-29T01:28:03.111Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T01:28:03.690Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T01:28:03.690Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T01:28:03.690Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T01:28:05.639Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18029
[2020-04-29T01:28:05.854Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T01:28:05.916Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T01:28:06.639Z] + chmod 755 generate-build-data.sh
[2020-04-29T01:28:06.639Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18029/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18029/runs/5 FAILURE 5557743
[2020-04-29T01:28:07.190Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18029/runs/5/steps/?limit=10000 -o steps-info.json
[2020-04-29T01:28:08.101Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18029/runs/5/tests/?status=FAILED -o tests-errors.json
[2020-04-29T01:28:08.652Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18029/runs/5/log/ -o pipeline-log.txt

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@andrewvc andrewvc merged commit f724a6f into elastic:7.x Apr 29, 2020
@andrewvc andrewvc deleted the backport_17687_7.x branch April 29, 2020 01:57
@zube zube bot added [zube]: Done and removed [zube]: Inbox labels Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants