[SignalFx Exporter] Expose HTTP Client settings

[MovieStoreGuy]

Component(s)

exporter/signalfx

Is your feature request related to a problem? Please describe.

When trying to export metrics to SignalFx, the following error can occur:

{"kind": "exporter", "data_type": "metrics", "name": "signalfx", "error": "Post \"https://ingest.us1.signalfx.com/v2/datapoint\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "31.841960232s"}

The current cause of why it is exceeding timeout is unknown, the current theory is that it is using some HTTP/2 functionality that causing issues but currently not exposed to the end user.

Describe the solution you'd like

Extend the current exporter to support using confighttp.HTTPClientSettings to allow for endusers to configure the http client with greater controls.

Describe alternatives you've considered

Some alternatives could be that http/2 being used is exposed via the config to reduce the complexity of the configuration.
However, I worry if that is reinventing the wheel here since the collector has that functionality abstracted already.

Additional context

I am not sure if this boarder lines on a bug, or an issue on ingestion but having this configuration exposed would allow for the end user to perform their own validation to help with diagnosis with support.

[github-actions]

Pinging code owners:

• exporter/signalfx: @pmcollins @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tiedotguy]

Some additional information - golang used to (~1.13 era) have incompatibilities when talking with AWS LBs over HTTP/2 (I believe when LBs needed to shut down a long running connection, during LB scaling), I don't know the state of it these days.

Whether this is actually the issue here or not is also unknown, but all other investigation paths so far have failed.

[atoulme]

Thank you. Please assign this issue to me.

[fatsheep9146]

Thanks! It's yours. @atoulme