enhance support for syslog ingestion

[mmguero]

Malcolm is now configurable to accept third-party syslogs directly. To describe the feature, here's copy/paste from the updated documentation:

From the end-to-end configuration documentation:

• Should Malcolm accept logs and metrics from a Hedgehog Linux sensor or other forwarder?
  • Answer yes or no in order for Malcolm's firewall to allow or block connections for OpenSearch, Logstash, and Filebeat TCP, bypassing the following several questions in this list. Answer customize to proceed to answer the following related questions individually:
  • ...
  • Accept standard syslog messages?
    • Answer Y for Malcolm to accept syslog messages according to the RFC3164 and RFC5424 standards over TCP or UDP.
      • Enter port for syslog over TCP (e.g., 514) or 0 to disable and Enter port for syslog over UDP (e.g., 514) or 0 to disable
        • Specify the port numbers on which to accept syslog messages for TCP or UDP, respectively. Other options for configuring how Malcolm accepts and processes syslog messages can be configured via environment variables in filebeat.env

From the filebeat.env environment variable documentation:

• filebeat.env - settings specific to Filebeat, particularly for how Filebeat watches for new log files to parse and how it receives and stores third-Party logs
  • ...
  • The following variables configure Malcolm's ability to accept syslog messages:
    • FILEBEAT_SYSLOG_TCP_LISTEN and FILEBEAT_SYSLOG_UDP_LISTEN - if set to true, Malcolm will accept syslog messages over TCP and/or UDP, respectively
    • FILEBEAT_SYSLOG_TCP_PORT and FILEBEAT_SYSLOG_UDP_PORT - the port on which Malcolm will accept syslog messages over TCP and/or UDP, respectively
    • FILEBEAT_SYSLOG_TCP_FORMAT and FILEBEAT_SYSLOG_UDP_FORMAT - one of auto, rfc3164, or rfc5424, to specify the allowed format for syslog messages over TCP and/or UDP, respectively (default auto)
    • FILEBEAT_SYSLOG_TCP_MAX_MESSAGE_SIZE and FILEBEAT_SYSLOG_UDP_MAX_MESSAGE_SIZE - defines the maximum message size of the message received over TCP and/or UDP, respectively (default: 10KiB for UDP, 20MiB for TCP)
    • FILEBEAT_SYSLOG_TCP_MAX_CONNECTIONS - specifies the maximum current number of TCP connections for syslog messages
    • FILEBEAT_SYSLOG_TCP_SSL - if set to true, syslog messages over TCP will require the use of TLS. When ./scripts/auth_setup is run, self-signed certificates are generated which may be used by remote log forwarders. Located in the filebeat/certs/ directory, the certificate authority and client certificate and key files should be copied to the host on which the forwarder is running and used when defining its settings for connecting to Malcolm.

From the Third Party Logs documentation:

Malcolm can accept syslog messages directly. During configuration, select customize when prompted Should Malcolm accept logs and metrics from a Hedgehog Linux sensor or other forwarder? to specify whether Malcolm should accept syslog over TCP, UDP, or both, and the respective ports on which the messages should be accepted.

Other options for configuring how Malcolm accepts and processes syslog messages can be configured via environment variables in filebeat.env.

[mmguero]

Comment from original bug creation:

@mmguero cloned issue idaholab/Malcolm#606 on 2024-10-29:

An INL-internal group using Malcolm was discussing Malcolm's ability to ingest host logs with us, and syslog in particular. As syslog is a common format for linux-and-related host logs, this might be worth looking at.

There are three parts to this I can see:

• From the aspect of collecting the syslogs, fluent bit has a syslog input that will monitor a TCP or UDP socket for syslogs sent to it.

  • In other words, on some other box you could set up a socket that rsyslog is sending to, then have fluent-bit monitor that socket and forward things along to Malcolm in the regular JSON-over-TCP/TLS method we do for everything else.

  • Our logstash pipeline already has support to parse these so in a major sense this is already working and supported

  • we don't have a prebuilt dashboard for this so we would want to build one (similar to the journald/systemd dashboard found under Malcolm and Third-Party Logs in the navigation panel)

• Now, what about Malcolm directly accepting syslogs itself, rather than having an external fluent-bit process forwarding them?

  • filebeat has a syslog processor, so we could leverage another filebeat process (filebeat-syslog) in the filebeat container (similar to how we added the one filebeat TCP somewhat recently); this is probably the way to go

  • logstash has a syslog input that would be another options

  • we'd want to normalize, as much as possible, so that the inputs from syslog directly and the inputs from fluent-bit have the same data/field format and are viewable in the same dashboard (use ECS as much as possible)

  • this would potentially have to expose another port, but I think that if we do it it merits a separate question in the configuration as it's unencrypted/unauthenticated and as such doesn't rise to the standard of the stuff we're doing for filebeat-tcp/5045 right now. I think we'd want to give them a more explicit "are you sure" kind of thing.

[mmguero]

tested using flog using UDP, TCP without SSL, and TCP with SSL.

Example command:

for ADDR in \
  'udp:10.9.0.215:514' \
  'ssl:10.9.0.215:514,cafile=ca.crt,cert=client.crt,key=client.key,verify=0'; do \
    for FORMAT in rfc5424 rfc3164; do \
        flog -f "$FORMAT" -n 50 | while read MESSAGE; do echo "$MESSAGE" | socat - "$ADDR"; done; \
    done; \
done