Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parsers.xpath): Allow to specify byte-array fields to encode in HEX #11999

Merged
merged 10 commits into from
Nov 21, 2022

Conversation

srebhan
Copy link
Member

@srebhan srebhan commented Oct 12, 2022

resolves #11952

This PR adds the last missing part for supporting PowerDNS protocol-buffer messages. To do this, the PR adds an ability to specify byte-array fields (via fields_bytes_as_hex) that should be represented as HEX in the xpath parser. This is necessary to be able to handle IP addresses and IDs correctly which would otherwise encoded as UTF-8 strings which are hard to reverse. The following config

[[inputs.socket_listener]]
  service_address = "tcp://127.0.0.1:8000"
  splitting_strategy = "variable length"
  splitting_length_field = {offset = 0, bytes = 2, endianness = "BE", header_length=2}

  data_format = "xpath_protobuf"

  xpath_native_types = true
  xpath_protobuf_file = "powerdns_message.proto"
  xpath_protobuf_type = "PBDNSMessage"
  xpath_protobuf_import_paths = [".","test_configs"]

  [[inputs.socket_listener.xpath]]
    fields_bytes_as_hex = ["from", "to", "messageId", "originalRequestorSubnet"]
    field_selection = "descendant::*"

[[processors.starlark]]
  source = '''
load("time.star", "time")

def ip(s):
  if len(s) == 8:
    # IPv4 addresses
    return '.'.join([str(int(s[i:i+2], base=16)) for i in range(0, len(s), 2)])

  # IPv6 addresses
  return ':'.join([s[i:i+4] for i in range(0, len(s), 4)])

def apply(metric):
  # Convert IP addresses
  ip_fields = ["from", "to", "originalRequestorSubnet"]
  for k in ip_fields:
    if k in metric.fields:
      metric.fields[k] = ip(metric.fields[k])

  # Extract the time
  t = metric.fields.pop("timeSec")*1000000000 + metric.fields.pop("timeUsec")*1000
  metric.time = time.from_timestamp(0, t).unix_nano

  return metric
'''

allows to send a message to the socket_listener input and retrieve a metric in the form

powerdns from="7f000001",fromPort=45729u,to="7f000001",toPort=53u,inBytes=48u,serverIdentity="xxxxxxxxxxxxxxxxxxx.com",messageId="943f90bea57a4eecbc5b0bea820a8aae",qName="ilse.nl.",qClass=1u,qType=1u,id=64100u,timeSec=1665050957u,timeUsec=500976u

@telegraf-tiger telegraf-tiger bot added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Oct 12, 2022
@srebhan srebhan marked this pull request as draft October 12, 2022 20:03
@srebhan srebhan force-pushed the socket_listener_powerdns branch from 7af5353 to 20dc85a Compare November 7, 2022 19:11
@srebhan srebhan added area/xml plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins and removed plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Nov 7, 2022
@srebhan srebhan changed the title feat(inputs.socket_listener): Allow to handle PowerDNS protobuf messages feat(parsers.xpath): Allow to specify byte-array fields to encode in HEX Nov 7, 2022
@srebhan srebhan force-pushed the socket_listener_powerdns branch from e2ecf6b to 8648241 Compare November 18, 2022 15:21
@srebhan srebhan marked this pull request as ready for review November 18, 2022 15:21
@powersj
Copy link
Contributor

powersj commented Nov 18, 2022

per tests: sounds like you need a make docs

@telegraf-tiger
Copy link
Contributor

@srebhan srebhan merged commit e311435 into influxdata:master Nov 21, 2022
@srebhan srebhan deleted the socket_listener_powerdns branch November 21, 2022 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/xml feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support PowerDNS Protobuf 'streams'
2 participants