[receiver/splunkhecreceiver] make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour #20873

omrozowicz-splunk · 2023-04-12T10:37:20Z

Description: Make Splunk HEC endpoint mimic the real one. After applying changes from this PR the response from the Splunk HEC health endpoint of Splunk HEC receiver will be:

curl -X GET http://0.0.0.0:8004/services/collector/health
{"text": "HEC is healthy", "code": 17}%

Which is the same for the real Splunk HEC:

curl -k -X GET https://X.X.X.X:8088/services/collector/health    
{"text":"HEC is healthy","code":17}%

The same for services/collector/health/1.0 endpoint.

For /1.0 support I just concatenate healthPath + "/1.0", not sure if it is a correct approach when user sets up its own healthPath endpoint. An alternative here is to create another variable or to create /1.0 route only when healthPath is the default one.

Also this endpoint returns a proper json, not the plain text response like the rest of our endpoints (for ex. "{\"text\":\"No data\",\"code\":5}")

I limited this endpoint to accept only GET method as this is how it works in the real Splunk HEC.

Link to tracking Issue: 20871

Testing: Unit tests, manual tests

Documentation: Nothing new added

runforesight · 2023-04-12T10:39:31Z

Foresight Summary

Major Impacts

build-and-test-windows duration(3 seconds) has decreased 30 minutes 31 seconds compared to main branch avg(30 minutes 34 seconds).

View More Details

✅ check-links workflow has finished in 54 seconds and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
changed files	- 🔗	N/A	See Details
check-links	- 🔗	N/A	See Details

✅ telemetrygen workflow has finished in 1 minute 1 second and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
build-dev	- 🔗	N/A	See Details
publish-latest	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

✅ prometheus-compliance-tests workflow has finished in 3 minutes 5 seconds (3 minutes 19 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
prometheus-compliance-tests	- 🔗	N/A	See Details

✅ e2e-tests workflow has finished in 14 minutes 17 seconds and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
kubernetes-test (v1.26.0)	- 🔗	N/A	See Details
kubernetes-test (v1.25.3)	- 🔗	N/A	See Details
kubernetes-test (v1.24.7)	- 🔗	N/A	See Details
kubernetes-test (v1.23.13)	- 🔗	N/A	See Details

✅ load-tests workflow has finished in 16 minutes 1 second (⚠️ 5 minutes 28 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
setup-environment	- 🔗	N/A	See Details
loadtest (TestIdleMode)	- 🔗	N/A	See Details
loadtest (TestBallastMemory\|TestLog10kDPS)	- 🔗	N/A	See Details
loadtest (TestMetric10kDPS\|TestMetricsFromFile)	- 🔗	N/A	See Details
loadtest (TestMetricResourceProcessor\|TestTrace10kSPS)	- 🔗	N/A	See Details
loadtest (TestTraceNoBackend10kSPS\|TestTrace1kSPSWithAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceBallast1kSPSWithAttrs\|TestTraceBallast1kSPSAddAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceAttributesProcessor)	- 🔗	N/A	See Details

✅ build-and-test workflow has finished in 35 minutes 26 seconds (11 minutes 16 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
govulncheck	- 🔗	N/A	See Details
setup-environment	- 🔗	N/A	See Details
checks	- 🔗	N/A	See Details
check-collector-module-version	- 🔗	N/A	See Details
lint-matrix (receiver-0)	- 🔗	N/A	See Details
lint-matrix (receiver-1)	- 🔗	N/A	See Details
lint-matrix (processor)	- 🔗	N/A	See Details
lint-matrix (exporter)	- 🔗	N/A	See Details
lint-matrix (extension)	- 🔗	N/A	See Details
lint-matrix (connector)	- 🔗	N/A	See Details
lint-matrix (internal)	- 🔗	N/A	See Details
lint-matrix (other)	- 🔗	N/A	See Details
build-examples	- 🔗	N/A	See Details
check-codeowners	- 🔗	N/A	See Details
correctness-metrics	- 🔗	N/A	See Details
integration-tests	- 🔗	N/A	See Details
correctness-traces	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.20, processor)	- 🔗	N/A	See Details
unittest-matrix (1.20, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.20, extension)	- 🔗	N/A	See Details
unittest-matrix (1.20, connector)	- 🔗	N/A	See Details
unittest-matrix (1.20, internal)	- 🔗	N/A	See Details
unittest-matrix (1.20, other)	- 🔗	N/A	See Details
unittest-matrix (1.19, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.19, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.19, processor)	- 🔗	N/A	See Details
unittest-matrix (1.19, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.19, extension)	- 🔗	N/A	See Details
unittest-matrix (1.19, connector)	- 🔗	N/A	See Details
unittest-matrix (1.19, internal)	- 🔗	N/A	See Details
unittest-matrix (1.19, other)	- 🔗	N/A	See Details
lint	- 🔗	N/A	See Details
unittest (1.20)	- 🔗	N/A	See Details
unittest (1.19)	- 🔗	N/A	See Details
cross-compile (darwin, amd64)	- 🔗	N/A	See Details
cross-compile (darwin, arm64)	- 🔗	N/A	See Details
cross-compile (linux, 386)	- 🔗	N/A	See Details
cross-compile (linux, amd64)	- 🔗	N/A	See Details
cross-compile (linux, arm)	- 🔗	N/A	See Details
cross-compile (linux, arm64)	- 🔗	N/A	See Details
cross-compile (linux, ppc64le)	- 🔗	N/A	See Details
cross-compile (windows, 386)	- 🔗	N/A	See Details
cross-compile (windows, amd64)	- 🔗	N/A	See Details
build-package (deb)	- 🔗	N/A	See Details
build-package (rpm)	- 🔗	N/A	See Details
windows-msi	- 🔗	N/A	See Details
publish-check	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details
publish-dev	- 🔗	N/A	See Details
rotate-milestone	- 🔗	N/A	See Details

⭕ build-and-test-windows workflow has finished in 3 seconds (30 minutes 31 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
windows-unittest-matrix	- 🔗	N/A	See Details
windows-unittest	- 🔗	N/A	See Details

✅ changelog workflow has finished in 2 minutes 11 seconds and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
changelog	- 🔗	N/A	See Details

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

…eck-fix

TylerHelmuth · 2023-04-12T14:29:06Z

@omrozowicz-splunk the 2 responses in the description look the same to me except for some spacing. Is the spacing the issue or is there a typo in the description?

receiver/splunkhecreceiver/receiver.go

TylerHelmuth · 2023-04-12T14:34:06Z

receiver/splunkhecreceiver/receiver.go

+	writer.Header().Add("Content-Type", "application/json")
+	writer.WriteHeader(http.StatusOK)
+	_, err := writer.Write([]byte(responseHecHealthy))
+	if err != nil {


Since we aren't doing anything with this error I think we can ignore its return from the Write call with a _.

Yes, although my IDE shows "Unhandled error" warnings, I think It doesn't make sense as the value is always the same and hardcoded. I replace this line with bare writer.Write([]byte(responseHecHealthy))

Ok, linter failed, you were right. Now it is _, _ = writer.Write([]byte(responseHecHealthy))

atoulme · 2023-04-12T16:18:57Z

receiver/splunkhecreceiver/receiver_test.go

@@ -1042,12 +1042,12 @@ func Test_splunkhecreceiver_handleHealthPath(t *testing.T) {
 		assert.NoError(t, r.Shutdown(context.Background()))
 	}()
 	w := httptest.NewRecorder()
-	r.handleHealthReq(w, httptest.NewRequest("POST", "http://localhost/services/collector/health", nil))
+	r.handleHealthReq(w, httptest.NewRequest("GET", "http://localhost/services/collector/health", nil))


it's fine to test GET only.

atoulme · 2023-04-12T16:19:17Z

receiver/splunkhecreceiver/receiver.go

@@ -202,7 +203,8 @@ func (r *splunkReceiver) Start(_ context.Context, host component.Host) error {
 	}

 	mx := mux.NewRouter()
-	mx.NewRoute().Path(r.config.HealthPath).HandlerFunc(r.handleHealthReq)
+	mx.NewRoute().Path(r.config.HealthPath).HandlerFunc(r.handleHealthReq).Methods("GET")
+	mx.NewRoute().Path(r.config.HealthPath + "/1.0").HandlerFunc(r.handleHealthReq).Methods("GET")


we can limit this one to GET.

Ok, /1.0 one I left unchanged

omrozowicz-splunk · 2023-04-13T07:04:39Z

@omrozowicz-splunk the 2 responses in the description look the same to me except for some spacing. Is the spacing the issue or is there a typo in the description?

Sorry, I phrased it wrongly. I meant to show that now these responses are the same. You can check the current behaviour in the issue I linked.

omrozowicz-splunk added 3 commits April 12, 2023 11:08

make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour

c0b93d3

add to changelog

c3d0ff7

update HealthPath unit tests

c494bff

omrozowicz-splunk requested a review from a team April 12, 2023 10:37

omrozowicz-splunk requested a review from atoulme as a code owner April 12, 2023 10:37

github-actions bot assigned TylerHelmuth Apr 12, 2023

github-actions bot added the receiver/splunkhec label Apr 12, 2023

Merge branch 'open-telemetry:main' into splunk-hec-receiver-health-ch…

0eb0922

…eck-fix

TylerHelmuth reviewed Apr 12, 2023

View reviewed changes

atoulme reviewed Apr 12, 2023

View reviewed changes

bring back the change with accepting only GETs

e79081d

fix linter error

a0fe831

TylerHelmuth approved these changes Apr 13, 2023

View reviewed changes

atoulme approved these changes Apr 13, 2023

View reviewed changes

atoulme added the ready to merge Code review completed; ready to merge by maintainers label Apr 13, 2023

TylerHelmuth merged commit 6f0cc7a into open-telemetry:main Apr 14, 2023

github-actions bot added this to the next release milestone Apr 14, 2023

atoulme mentioned this pull request May 12, 2023

[receiver/splunkhec] Make Splunk HEC receiver mimic the real Splunk HEC in terms of healthcheck #20871

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[receiver/splunkhecreceiver] make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour #20873

[receiver/splunkhecreceiver] make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour #20873

omrozowicz-splunk commented Apr 12, 2023 •

edited

Loading

runforesight bot commented Apr 12, 2023 •

edited

Loading

`build-and-test-windows` duration(3 seconds) has decreased 30 minutes 31 seconds compared to main branch avg(30 minutes 34 seconds).

✅ check-links workflow has finished in 54 seconds and finished at 13th Apr, 2023.

✅ telemetrygen workflow has finished in 1 minute 1 second and finished at 13th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 3 minutes 5 seconds (3 minutes 19 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

✅ e2e-tests workflow has finished in 14 minutes 17 seconds and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 16 minutes 1 second (⚠️ 5 minutes 28 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ build-and-test workflow has finished in 35 minutes 26 seconds (11 minutes 16 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

⭕ build-and-test-windows workflow has finished in 3 seconds (30 minutes 31 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

✅ changelog workflow has finished in 2 minutes 11 seconds and finished at 13th Apr, 2023.

TylerHelmuth commented Apr 12, 2023

TylerHelmuth Apr 12, 2023

omrozowicz-splunk Apr 13, 2023

omrozowicz-splunk Apr 13, 2023

atoulme Apr 12, 2023

atoulme Apr 12, 2023

omrozowicz-splunk Apr 13, 2023

omrozowicz-splunk commented Apr 13, 2023

[receiver/splunkhecreceiver] make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour #20873

[receiver/splunkhecreceiver] make splunk_hec receiver healthcheck mimic the real Splunk HEC behaviour #20873

Conversation

omrozowicz-splunk commented Apr 12, 2023 • edited Loading

runforesight bot commented Apr 12, 2023 • edited Loading

Foresight Summary

build-and-test-windows duration(3 seconds) has decreased 30 minutes 31 seconds compared to main branch avg(30 minutes 34 seconds).

✅ check-links workflow has finished in 54 seconds and finished at 13th Apr, 2023.

✅ telemetrygen workflow has finished in 1 minute 1 second and finished at 13th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 3 minutes 5 seconds (3 minutes 19 seconds less than main branch avg.) and finished at 13th Apr, 2023.

✅ e2e-tests workflow has finished in 14 minutes 17 seconds and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 16 minutes 1 second (⚠️ 5 minutes 28 seconds more than main branch avg.) and finished at 13th Apr, 2023.

✅ build-and-test workflow has finished in 35 minutes 26 seconds (11 minutes 16 seconds less than main branch avg.) and finished at 13th Apr, 2023.

⭕ build-and-test-windows workflow has finished in 3 seconds (30 minutes 31 seconds less than main branch avg.) and finished at 13th Apr, 2023.

✅ changelog workflow has finished in 2 minutes 11 seconds and finished at 13th Apr, 2023.

TylerHelmuth commented Apr 12, 2023

TylerHelmuth Apr 12, 2023

Choose a reason for hiding this comment

omrozowicz-splunk Apr 13, 2023

Choose a reason for hiding this comment

omrozowicz-splunk Apr 13, 2023

Choose a reason for hiding this comment

atoulme Apr 12, 2023

Choose a reason for hiding this comment

atoulme Apr 12, 2023

Choose a reason for hiding this comment

omrozowicz-splunk Apr 13, 2023

Choose a reason for hiding this comment

omrozowicz-splunk commented Apr 13, 2023

omrozowicz-splunk commented Apr 12, 2023 •

edited

Loading

runforesight bot commented Apr 12, 2023 •

edited

Loading

`build-and-test-windows` duration(3 seconds) has decreased 30 minutes 31 seconds compared to main branch avg(30 minutes 34 seconds).

✅ prometheus-compliance-tests workflow has finished in 3 minutes 5 seconds (3 minutes 19 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 16 minutes 1 second (⚠️ 5 minutes 28 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ build-and-test workflow has finished in 35 minutes 26 seconds (11 minutes 16 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

⭕ build-and-test-windows workflow has finished in 3 seconds (30 minutes 31 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.