humio_metrics sink ignores host config

[jszwedko]

Vector Version

vector 0.11.0 (gc526860 x86_64-unknown-linux-gnu 2020-11-06) 

Vector Configuration File

data_dir = "/var/lib/vector"

[sources.vector_metrics]
type = "internal_metrics"

[sinks.humio_json]
host = "http://host.docker.internal:8080"
inputs = ["vector_metrics"]
token = "30c0ee75-205f-47e6-a071-1d81e64c2c32"
type = "humio_metrics"

Debug Output

Nov 06 14:45:40.490  INFO vector::app: Log level is enabled. level="trace"                                                                                                                                          
Nov 06 14:45:40.491  INFO vector::app: Loading configs. path=["/tmp/tmp.toml"]                                                                                                                                      
Nov 06 14:45:40.512 DEBUG vector::tls::settings: Fetching system root certs.                                                                                                                                        
Nov 06 14:45:40.513  INFO vector::topology: Running healthchecks.                                                                                                                                                   
Nov 06 14:45:40.513  INFO vector::topology: Starting source. name="vector_metrics"                                                                                                                                  
Nov 06 14:45:40.513  INFO vector::topology::builder: Healthcheck: Disabled.                                                                                                                                         
Nov 06 14:45:40.513  INFO vector::topology: Starting sink. name="humio_json"                                                                                                                                        
Nov 06 14:45:40.514  INFO vector: Vector has started. version="0.11.0" git_version="v0.9.0-1275-gc526860" released="Fri, 06 Nov 2020 14:33:06 +0000" arch="x86_64"
Nov 06 14:45:40.514 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: No batches; driving service to completion.                                 
Nov 06 14:45:40.515 TRACE vector: Beep.                                                                                                                                                                             
Nov 06 14:45:40.516 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::internal_events::metric_to_log: Processed one event.                                          
Nov 06 14:45:40.534 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: Starting new batch timer.                                                  
Nov 06 14:45:40.534 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: Polling batch linger.                                                      
Nov 06 14:45:40.534 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: Polling batch linger.                                                      
Nov 06 14:45:41.516 TRACE vector: Beep.                                                                                                                                                                             
Nov 06 14:45:41.535 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: Service ready; Sending batch.    
Nov 06 14:45:41.535 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: Submitting service request. in_flight_requests=1
Nov 06 14:45:41.536 TRACE sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}: vector::sinks::util::sink: No batches; driving service to completion.                                 
Nov 06 14:45:41.537 DEBUG http: vector::http: Sending request. uri=https://cloud.humio.com/services/collector/event method=POST                                                                                     
Nov 06 14:45:42.020 DEBUG http: vector::http: Response. status=405 version=HTTP/1.1                                                                                                                                 
Nov 06 14:45:42.021 ERROR sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}:request{request_id=0}: vector::sinks::util::retries: Not retriable; dropping the request. reason="respo
nse status: 405 Method Not Allowed"                                                                                                                                                                                 
Nov 06 14:45:42.021 ERROR sink{component_kind="sink" component_name=humio_json component_type=humio_metrics}:request{request_id=0}: vector::sinks::util::sink: Response wasn't successful. response=Response { statu
s: 405, version: HTTP/1.1, headers: {"server": "nginx", "date": "Fri, 06 Nov 2020 14:45:41 GMT", "content-type": "text/plain; charset=UTF-8", "content-length": "56", "connection": "keep-alive", "allow": "OPTIONS,
 GET"}, body: b"HTTP method not allowed, supported methods: OPTIONS, GET" }

Expected Behavior

It uses the configured host

Actual Behavior

It uses the default

Additional Context

The new name for this config option is endpoint but we have code supporting the old host option. That just doesn't seem to be happening in this case.

[StephenWakely]

See here

It looks like there is a bug in serde whereby aliased fields do not work with flattened structs. A pull request to fix this (here) has been sitting around for 18 months.

Reading around the issue list for serde there are a number of issues with flattened structs. It sounds like there are a some nontrivial problems with the current implementation.

To fix this, it looks like I will just have to copy the fields from the Humio logs config rather than rely on serde to flatten the struct. Not as elegant, but the compiler will catch any errors should Humio logs change and the metrics sink not be updated accordingly.