diff --git a/docs/malcolm-config.md b/docs/malcolm-config.md index 55e2b7b96..dc2e85812 100644 --- a/docs/malcolm-config.md +++ b/docs/malcolm-config.md @@ -101,6 +101,7 @@ Although the configuration script automates many of the following configuration - `SURICATA_LIVE_CAPTURE` - if set to `true`, Suricata will monitor live traffic on the local interface(s) defined by `PCAP_FILTER` - `SURICATA_ROTATED_PCAP` - if set to `true`, Suricata can analyze PCAP files captured by `netsniff-ng` or `tcpdump` (see `PCAP_ENABLE_NETSNIFF` and `PCAP_ENABLE_TCPDUMP`, as well as `SURICATA_AUTO_ANALYZE_PCAP_FILES`); if `SURICATA_LIVE_CAPTURE` is `true`, this should be `false`; otherwise Suricata will see duplicate traffic - `SURICATA_DISABLE_ICS_ALL` - if set to `true`, this variable can be used to disable Malcolm's [built-in Suricata rules for Operational Technology/Industrial Control Systems (OT/ICS) vulnerabilities and exploits]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/suricata/rules-default/OT) + - `SURICATA_STATS_ENABLED`, `SURICATA_STATS_EVE_ENABLED`, and `SURICATA_STATS_INTERVAL` - these variables control the generation of [live traffic capture](live-analysis.md#LocalPCAP) statistics for [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats), which data is used to populate the **Packet Capture Statistics** dashboard - See [**Tuning Suricata**](live-analysis.md#LiveAnalysisTuningSuricata) for other variables related to managing Suricata's performance and resource utilization. * **`upload-common.env`** - settings for dealing with PCAP files [uploaded](upload.md#Upload) to Malcolm for analysis - `AUTO_TAG` – if set to `true`, Malcolm will automatically create Arkime sessions and Zeek logs with tags based on the filename, as described in [Tagging](upload.md#Tagging) (default `true`) @@ -133,6 +134,7 @@ Although the configuration script automates many of the following configuration - `ZEEK_JA4SSH_PACKET_COUNT` - the Zeek [JA4+ plugin](https://github.com/FoxIO-LLC/ja4) calculates the JA4SSH value once for every *x* SSH packets; *x* is set here (default `200`) - `ZEEK_LIVE_CAPTURE` - if set to `true`, Zeek will monitor live traffic on the local interface(s) defined by `PCAP_FILTER` + See [**Tuning Zeek**](live-analysis.md#LiveAnalysisTuningZeek) for other variables related to managing Zeek's performance and resource utilization. + - `ZEEK_DISABLE_STATS` - if `ZEEK_LIVE_CAPTURE` is `true` and this variable is set to `false` or blank, Malcolm will enable [capture statistics Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info), which data is used to populate the **Packet Capture Statistics** dashboard - `ZEEK_LOCAL_NETS` - specifies the value for Zeek's [`Site::local_nets`](https://docs.zeek.org/en/master/scripts/base/utils/site.zeek.html#id-Site::local_nets) variable (and `networks.cfg` for live capture) (e.g., `1.2.3.0/24,5.6.7.0/24`); note that by default, Zeek considers IANA-registered private address space such as `10.0.0.0/8` and `192.168.0.0/16` site-local - `ZEEK_ROTATED_PCAP` - if set to `true`, Zeek can analyze captured PCAP files captured by `netsniff-ng` or `tcpdump` (see `PCAP_ENABLE_NETSNIFF` and `PCAP_ENABLE_TCPDUMP`, as well as `ZEEK_AUTO_ANALYZE_PCAP_FILES`); if `ZEEK_LIVE_CAPTURE` is `true`, this should be `false`; otherwise Zeek will see duplicate traffic - See [**Managing disk usage**](#DiskUsage) below for a discussion of the variables control automatic threshold-based deletion of the oldest [Zeek-extracted files](file-scanning.md#ZeekFileExtraction). diff --git a/docs/malcolm-hedgehog-e2e-iso-install.md b/docs/malcolm-hedgehog-e2e-iso-install.md index bb7463020..708948b63 100644 --- a/docs/malcolm-hedgehog-e2e-iso-install.md +++ b/docs/malcolm-hedgehog-e2e-iso-install.md @@ -298,8 +298,10 @@ The [configuration and tuning](malcolm-config.md#ConfigAndTuning) wizard's quest - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) as described above, users may optionally provide a capture filter. This filter will be used to limit what traffic the PCAP service ([netsniff-ng](http://netsniff-ng.org/) or [tcpdump](https://www.tcpdump.org/)) and the traffic analysis services ([Zeek](https://www.zeek.org/) and [Suricata](https://suricata.io/)) will see. Capture filters are specified using [Berkeley Packet Filter (BPF)](http://biot.com/capstats/bpf.html) syntax. For example, to indicate that Malcolm should ignore the ports it uses to communicate with Hedgehog Linux, users could specify `not port 5044 and not port 5045 and not port 8005 and not port 8006 and not port 9200`. - **Disable capture interface hardware offloading and adjust ring buffer sizes?** - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) and users answer **Y** to this question, Malcolm will [use `ethtool`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/shared/bin/nic-capture-setup.sh) to disable NIC hardware offloading features and adjust ring buffer sizes for capture interface(s); this should be enabled if the interface(s) are being used for capture **only**, otherwise answer **N**. If unsure, users should probably answer **N**. -* **Specify capture interface(s) (comma-separated)** - - Specify the network interface(s) for [live traffic analysis](live-analysis.md#LocalPCAP) if it is enabled for netsniff-ng, tcpdump, Suricata or Zeek as described above. For multiple interfaces, separate the interface names with a comma (e.g., `enp0s25` or `enp10s0,enp11s0`). + - **Enable live packet capture statistics?** + - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) and users answer **Y** to this question, Malcolm will enable statistics collection for [Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info) and [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats), which data is used to populate the **Packet Capture Statistics** dashboard. + - **Specify capture interface(s) (comma-separated)** + + Specify the network interface(s) for [live traffic analysis](live-analysis.md#LocalPCAP) if it is enabled for netsniff-ng, tcpdump, Suricata or Zeek as described above. For multiple interfaces, separate the interface names with a comma (e.g., `enp0s25` or `enp10s0,enp11s0`). * **Enable dark mode for OpenSearch Dashboards?** - Answer **Y** for dark-themed dashboards or **N** for light-themed ones. @@ -458,6 +460,8 @@ Upon choosing the capture interfaces and selecting OK, users may optionally prov ![Specify capture filters](./images/hedgehog/images/capture_filter.png) +Users will be prompted whether or not they wish to enable live packet capture statistics [Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info) and [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats). If enabled, these statistics will be used to populate Malcolm's **Packet Capture Statistics** dashboard. + Next users must specify the paths where captured PCAP files and logs will be stored locally on the sensor. If the installation worked as expected, these paths should be prepopulated to reflect paths on the volumes formatted at install time for the purpose storing these artifacts. Usually these paths will exist on separate storage volumes. Enabling the PCAP and log pruning autostart services (see the section on autostart services below) will enable monitoring of these paths to ensure that their contents do not consume more than 90% of their respective volumes' space. Choose **OK** to continue. ![Specify capture paths](./images/hedgehog/images/capture_paths.png) diff --git a/hedgehog-iso/config/includes.chroot/usr/local/bin/configure-capture.py b/hedgehog-iso/config/includes.chroot/usr/local/bin/configure-capture.py index c71660e88..b108d6820 100755 --- a/hedgehog-iso/config/includes.chroot/usr/local/bin/configure-capture.py +++ b/hedgehog-iso/config/includes.chroot/usr/local/bin/configure-capture.py @@ -147,6 +147,7 @@ class Constants: MSG_IDENTIFY_NICS = 'Do you need help identifying network interfaces?' MSG_BACKGROUND_TITLE = 'Sensor Configuration' MSG_CONFIG_AUTOSTARTS = 'Specify autostart processes' + MSG_CONFIG_CAPTURE_STATS = 'Enable live packet capture statistics for Zeek and Suricata?' MSG_CONFIG_ICS_ANALYZERS = ( 'Is the sensor being used to monitor an Operational Technology/Industrial Control Systems (OT/ICS) network?' ) @@ -488,6 +489,8 @@ def main(): available_adapters = get_available_adapters() # previously used capture interfaces preselected_ifaces = set([x.strip() for x in capture_config_dict["CAPTURE_INTERFACE"].split(',')]) + # generate capture statistics + capture_stats = False while (len(available_adapters) > 0) and ( d.yesno(Constants.MSG_IDENTIFY_NICS, yes_label="No", no_label="Yes") != Dialog.OK @@ -559,6 +562,8 @@ def main(): ) prev_capture_filter = capture_filter + capture_stats = d.yesno(Constants.MSG_CONFIG_CAPTURE_STATS) == Dialog.OK + # get paths for captured PCAP and Zeek files while True: code, path_values = d.form( @@ -776,6 +781,9 @@ def main(): capture_config_dict["EXTRACTED_FILE_HTTP_SERVER_KEY"] = zeek_carved_file_http_serve_encrypt_key capture_config_dict["ZEEK_DISABLE_ICS_ALL"] = '' if ics_network else 'true' capture_config_dict["ZEEK_DISABLE_BEST_GUESS_ICS"] = '' if ics_best_guess else 'true' + capture_config_dict["ZEEK_DISABLE_STATS"] = '' if capture_stats else 'true' + capture_config_dict["SURICATA_STATS_ENABLED"] = 'true' if capture_stats else 'false' + capture_config_dict["SURICATA_STATS_EVE_ENABLED"] = 'true' if capture_stats else 'false' # get confirmation from user that we really want to do this code = d.yesno( @@ -799,6 +807,9 @@ def main(): { "CAPTURE_FILTER": '"' + capture_config_dict["CAPTURE_FILTER"] + '"', "CAPTURE_INTERFACE": capture_config_dict["CAPTURE_INTERFACE"], + "ZEEK_DISABLE_STATS": capture_config_dict["ZEEK_DISABLE_STATS"], + "SURICATA_STATS_ENABLED": capture_config_dict["SURICATA_STATS_ENABLED"], + "SURICATA_STATS_EVE_ENABLED": capture_config_dict["SURICATA_STATS_EVE_ENABLED"], "EXTRACTED_FILE_HTTP_SERVER_KEY": '"' + capture_config_dict["EXTRACTED_FILE_HTTP_SERVER_KEY"] + '"', diff --git a/scripts/install.py b/scripts/install.py index 4116fcff7..e9e991bde 100755 --- a/scripts/install.py +++ b/scripts/install.py @@ -1808,6 +1808,7 @@ def tweak_malcolm_runtime(self, malcolm_install_path): pcapIface = 'lo' tweakIface = False pcapFilter = '' + captureStats = False captureSelection = ( 'c' if ( @@ -1835,6 +1836,7 @@ def tweak_malcolm_runtime(self, malcolm_install_path): pcapNetSniff = not liveArkime liveSuricata = True liveZeek = True + captureStats = True tweakIface = True elif captureSelection == 'c': if InstallerYesOrNo( @@ -1889,6 +1891,11 @@ def tweak_malcolm_runtime(self, malcolm_install_path): default=args.tweakIface, extraLabel=BACK_LABEL, ) + captureStats = (liveZeek or liveSuricata) and InstallerYesOrNo( + 'Enable live packet capture statistics?', + default=args.captureStats, + extraLabel=BACK_LABEL, + ) if pcapNetSniff or pcapTcpDump or liveArkime or liveZeek or liveSuricata: pcapIface = '' @@ -2313,6 +2320,17 @@ def tweak_malcolm_runtime(self, malcolm_install_path): 'SURICATA_LIVE_CAPTURE', TrueOrFalseNoQuote(liveSuricata), ), + # live capture statistics for Suricata + EnvValue( + os.path.join(args.configDir, 'suricata-live.env'), + 'SURICATA_STATS_ENABLED', + TrueOrFalseNoQuote(captureStats), + ), + EnvValue( + os.path.join(args.configDir, 'suricata-live.env'), + 'SURICATA_STATS_EVE_ENABLED', + TrueOrFalseNoQuote(captureStats), + ), # rotated captured PCAP analysis with Suricata (not live capture) EnvValue( os.path.join(args.configDir, 'suricata-offline.env'), @@ -2421,6 +2439,12 @@ def tweak_malcolm_runtime(self, malcolm_install_path): 'ZEEK_LIVE_CAPTURE', TrueOrFalseNoQuote(liveZeek), ), + # live capture statistics for Zeek + EnvValue( + os.path.join(args.configDir, 'zeek-live.env'), + 'ZEEK_DISABLE_STATS', + TrueOrFalseNoQuote(not captureStats), + ), # rotated captured PCAP analysis with Zeek (not live capture) EnvValue( os.path.join(args.configDir, 'zeek-offline.env'), @@ -4504,6 +4528,16 @@ def main(): default=True, help="Disable capture interface hardware offloading and adjust ring buffer sizes", ) + captureArgGroup.add_argument( + '--live-capture-stats', + dest='captureStats', + type=str2bool, + metavar="true|false", + nargs='?', + const=True, + default=False, + help=f"Enable live packet capture statistics for Zeek and/or Suricata", + ) captureArgGroup.add_argument( '--live-capture-arkime', dest='liveArkime',