[receiver/splunkhec] Make Splunk HEC receiver mimic the real Splunk HEC in terms of healthcheck

[omrozowicz-splunk]

Component(s)

receiver/splunkhec

Is your feature request related to a problem? Please describe.

There was an issue reported that Splunk HEC receiver doesn't respond exactly like the real Splunk HEC receiver.
Currently no body is present, only a 200 status. The curl looks the following way:

curl -s -X GET http://0.0.0.0:8003/services/collector/health --verbose    
*   Trying 0.0.0.0:8003...
* Connected to 0.0.0.0 (127.0.0.1) port 8003 (#0)
> GET /services/collector/health HTTP/1.1
> Host: 0.0.0.0:8003
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 07 Apr 2023 09:58:11 GMT
< Content-Length: 0
< 
* Connection #0 to host 0.0.0.0 left intact

While the real HEC response is:

╰─$ curl -k -X GET https://X.X.X.X:8088/services/collector/health/1.0 -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 52.199.91.7:8088...
* Connected to 52.199.91.7 (52.199.91.7) port 8088 (#0)
...
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /services/collector/health/1.0 HTTP/1.1
> Host: 52.199.91.7:8088
> User-Agent: curl/7.86.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 12 Apr 2023 08:22:23 GMT
< Content-Type: application/json; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 35
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
< 
* Connection #0 to host 52.199.91.7 left intact
{"text":"HEC is healthy","code":17}%   

Additionally, Splunk HEC supports also /services/collector/health/1.0 path.

Describe the solution you'd like

I'd like to improve the logic in HealthCheck handler to return response body as well, and create another path to be accepted:
r.config.HealthPath + "/1.0".

Describe alternatives you've considered

No response

Additional context

While testing Splunk HEC receiver responses I've found out that we always return plain text (connected with #18097), for example as a response for an empty request we receive plain text json-alike response:

"{\"text\":\"No data\",\"code\":5}"

Shouldn't we return the correct json and not string-casted plain text?
In this implementation I intend to return the real json to have the same response body:

{"text": "HEC is healthy", "code": 17}

[github-actions]

Pinging code owners:

• receiver/splunkhec: @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

I believe this is now fixed with #20873. Closing.