Prompt user whether or not to enable live packet capture statistics

docs/malcolm-config.md

@@ -101,6 +101,7 @@ Although the configuration script automates many of the following configuration
     - `SURICATA_LIVE_CAPTURE` - if set to `true`, Suricata will monitor live traffic on the local interface(s) defined by `PCAP_FILTER`
     - `SURICATA_ROTATED_PCAP` - if set to `true`, Suricata can analyze PCAP files captured by `netsniff-ng` or `tcpdump` (see `PCAP_ENABLE_NETSNIFF` and `PCAP_ENABLE_TCPDUMP`, as well as `SURICATA_AUTO_ANALYZE_PCAP_FILES`); if `SURICATA_LIVE_CAPTURE` is `true`, this should be `false`; otherwise Suricata will see duplicate traffic
     - `SURICATA_DISABLE_ICS_ALL` - if set to `true`, this variable can be used to disable Malcolm's [built-in Suricata rules for Operational Technology/Industrial Control Systems (OT/ICS) vulnerabilities and exploits]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/suricata/rules-default/OT)
+    - `SURICATA_STATS_ENABLED`, `SURICATA_STATS_EVE_ENABLED`, and `SURICATA_STATS_INTERVAL` - these variables control the generation of [live traffic capture](live-analysis.md#LocalPCAP) statistics for [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats), which data is used to populate the **Packet Capture Statistics** dashboard
     - See [**Tuning Suricata**](live-analysis.md#LiveAnalysisTuningSuricata) for other variables related to managing Suricata's performance and resource utilization.    
 * **`upload-common.env`** - settings for dealing with PCAP files [uploaded](upload.md#Upload) to Malcolm for analysis
     - `AUTO_TAG` – if set to `true`, Malcolm will automatically create Arkime sessions and Zeek logs with tags based on the filename, as described in [Tagging](upload.md#Tagging) (default `true`)
@@ -133,6 +134,7 @@ Although the configuration script automates many of the following configuration
     - `ZEEK_JA4SSH_PACKET_COUNT` - the Zeek [JA4+ plugin](https://github.com/FoxIO-LLC/ja4) calculates the JA4SSH value once for every *x* SSH packets; *x* is set here (default `200`)
     - `ZEEK_LIVE_CAPTURE` - if set to `true`, Zeek will monitor live traffic on the local interface(s) defined by `PCAP_FILTER`
         + See [**Tuning Zeek**](live-analysis.md#LiveAnalysisTuningZeek) for other variables related to managing Zeek's performance and resource utilization.
+    - `ZEEK_DISABLE_STATS` - if `ZEEK_LIVE_CAPTURE` is `true` and this variable is set to `false` or blank, Malcolm will enable [capture statistics Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info), which data is used to populate the **Packet Capture Statistics** dashboard
     - `ZEEK_LOCAL_NETS` - specifies the value for Zeek's [`Site::local_nets`](https://docs.zeek.org/en/master/scripts/base/utils/site.zeek.html#id-Site::local_nets) variable (and `networks.cfg` for live capture) (e.g., `1.2.3.0/24,5.6.7.0/24`); note that by default, Zeek considers IANA-registered private address space such as `10.0.0.0/8` and `192.168.0.0/16` site-local
     - `ZEEK_ROTATED_PCAP` - if set to `true`, Zeek can analyze captured PCAP files captured by `netsniff-ng` or `tcpdump` (see `PCAP_ENABLE_NETSNIFF` and `PCAP_ENABLE_TCPDUMP`, as well as `ZEEK_AUTO_ANALYZE_PCAP_FILES`); if `ZEEK_LIVE_CAPTURE` is `true`, this should be `false`; otherwise Zeek will see duplicate traffic
     - See [**Managing disk usage**](#DiskUsage) below for a discussion of the variables control automatic threshold-based deletion of the oldest [Zeek-extracted files](file-scanning.md#ZeekFileExtraction).

docs/malcolm-hedgehog-e2e-iso-install.md

@@ -298,8 +298,10 @@ The [configuration and tuning](malcolm-config.md#ConfigAndTuning) wizard's quest
         - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) as described above, users may optionally provide a capture filter. This filter will be used to limit what traffic the PCAP service ([netsniff-ng](http://netsniff-ng.org/) or [tcpdump](https://www.tcpdump.org/)) and the traffic analysis services ([Zeek](https://www.zeek.org/) and [Suricata](https://suricata.io/)) will see. Capture filters are specified using [Berkeley Packet Filter (BPF)](http://biot.com/capstats/bpf.html) syntax. For example, to indicate that Malcolm should ignore the ports it uses to communicate with Hedgehog Linux, users could specify `not port 5044 and not port 5045 and not port 8005 and not port 8006 and not port 9200`.
     - **Disable capture interface hardware offloading and adjust ring buffer sizes?**
         - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) and users answer **Y** to this question, Malcolm will [use `ethtool`]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/shared/bin/nic-capture-setup.sh) to disable NIC hardware offloading features and adjust ring buffer sizes for capture interface(s); this should be enabled if the interface(s) are being used for capture **only**, otherwise answer **N**. If unsure, users should probably answer **N**.
-* **Specify capture interface(s) (comma-separated)**
-    - Specify the network interface(s) for [live traffic analysis](live-analysis.md#LocalPCAP) if it is enabled for netsniff-ng, tcpdump, Suricata or Zeek as described above. For multiple interfaces, separate the interface names with a comma (e.g., `enp0s25` or `enp10s0,enp11s0`).
+    - **Enable live packet capture statistics?**
+        - If Malcolm is doing its own [live traffic analysis](live-analysis.md#LocalPCAP) and users answer **Y** to this question, Malcolm will enable statistics collection for [Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info) and [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats), which data is used to populate the **Packet Capture Statistics** dashboard.
+    - **Specify capture interface(s) (comma-separated)**
+        + Specify the network interface(s) for [live traffic analysis](live-analysis.md#LocalPCAP) if it is enabled for netsniff-ng, tcpdump, Suricata or Zeek as described above. For multiple interfaces, separate the interface names with a comma (e.g., `enp0s25` or `enp10s0,enp11s0`).
 * **Enable dark mode for OpenSearch Dashboards?**
     - Answer **Y** for dark-themed dashboards or **N** for light-themed ones.
 
@@ -458,6 +460,8 @@ Upon choosing the capture interfaces and selecting OK, users may optionally prov
 
 ![Specify capture filters](./images/hedgehog/images/capture_filter.png)
 
+Users will be prompted whether or not they wish to enable live packet capture statistics [Zeek](https://docs.zeek.org/en/master/scripts/policy/misc/stats.zeek.html#type-Stats::Info) and [Suricata](https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#stats). If enabled, these statistics will be used to populate Malcolm's **Packet Capture Statistics** dashboard.
+
 Next users must specify the paths where captured PCAP files and logs will be stored locally on the sensor. If the installation worked as expected, these paths should be prepopulated to reflect paths on the volumes formatted at install time for the purpose storing these artifacts. Usually these paths will exist on separate storage volumes. Enabling the PCAP and log pruning autostart services (see the section on autostart services below) will enable monitoring of these paths to ensure that their contents do not consume more than 90% of their respective volumes' space. Choose **OK** to continue.
 
 ![Specify capture paths](./images/hedgehog/images/capture_paths.png)

hedgehog-iso/config/includes.chroot/usr/local/bin/configure-capture.py

@@ -147,6 +147,7 @@ class Constants:
     MSG_IDENTIFY_NICS = 'Do you need help identifying network interfaces?'
     MSG_BACKGROUND_TITLE = 'Sensor Configuration'
     MSG_CONFIG_AUTOSTARTS = 'Specify autostart processes'
+    MSG_CONFIG_CAPTURE_STATS = 'Enable live packet capture statistics for Zeek and Suricata?'
     MSG_CONFIG_ICS_ANALYZERS = (
         'Is the sensor being used to monitor an Operational Technology/Industrial Control Systems (OT/ICS) network?'
     )
@@ -488,6 +489,8 @@ def main():
                 available_adapters = get_available_adapters()
                 # previously used capture interfaces
                 preselected_ifaces = set([x.strip() for x in capture_config_dict["CAPTURE_INTERFACE"].split(',')])
+                # generate capture statistics
+                capture_stats = False
 
                 while (len(available_adapters) > 0) and (
                     d.yesno(Constants.MSG_IDENTIFY_NICS, yes_label="No", no_label="Yes") != Dialog.OK
@@ -559,6 +562,8 @@ def main():
                             )
                         prev_capture_filter = capture_filter
 
+                    capture_stats = d.yesno(Constants.MSG_CONFIG_CAPTURE_STATS) == Dialog.OK
+
                 # get paths for captured PCAP and Zeek files
                 while True:
                     code, path_values = d.form(
@@ -776,6 +781,9 @@ def main():
                 capture_config_dict["EXTRACTED_FILE_HTTP_SERVER_KEY"] = zeek_carved_file_http_serve_encrypt_key
                 capture_config_dict["ZEEK_DISABLE_ICS_ALL"] = '' if ics_network else 'true'
                 capture_config_dict["ZEEK_DISABLE_BEST_GUESS_ICS"] = '' if ics_best_guess else 'true'
+                capture_config_dict["ZEEK_DISABLE_STATS"] = '' if capture_stats else 'true'
+                capture_config_dict["SURICATA_STATS_ENABLED"] = 'true' if capture_stats else 'false'
+                capture_config_dict["SURICATA_STATS_EVE_ENABLED"] = 'true' if capture_stats else 'false'
 
                 # get confirmation from user that we really want to do this
                 code = d.yesno(
@@ -799,6 +807,9 @@ def main():
                         {
                             "CAPTURE_FILTER": '"' + capture_config_dict["CAPTURE_FILTER"] + '"',
                             "CAPTURE_INTERFACE": capture_config_dict["CAPTURE_INTERFACE"],
+                            "ZEEK_DISABLE_STATS": capture_config_dict["ZEEK_DISABLE_STATS"],
+                            "SURICATA_STATS_ENABLED": capture_config_dict["SURICATA_STATS_ENABLED"],
+                            "SURICATA_STATS_EVE_ENABLED": capture_config_dict["SURICATA_STATS_EVE_ENABLED"],
                             "EXTRACTED_FILE_HTTP_SERVER_KEY": '"'
                             + capture_config_dict["EXTRACTED_FILE_HTTP_SERVER_KEY"]
                             + '"',

scripts/install.py

@@ -1808,6 +1808,7 @@ def tweak_malcolm_runtime(self, malcolm_install_path):
                     pcapIface = 'lo'
                     tweakIface = False
                     pcapFilter = ''
+                    captureStats = False
                     captureSelection = (
                         'c'
                         if (
@@ -1835,6 +1836,7 @@ def tweak_malcolm_runtime(self, malcolm_install_path):
                         pcapNetSniff = not liveArkime
                         liveSuricata = True
                         liveZeek = True
+                        captureStats = True
                         tweakIface = True
                     elif captureSelection == 'c':
                         if InstallerYesOrNo(
@@ -1889,6 +1891,11 @@ def tweak_malcolm_runtime(self, malcolm_install_path):
                                 default=args.tweakIface,
                                 extraLabel=BACK_LABEL,
                             )
+                        captureStats = (liveZeek or liveSuricata) and InstallerYesOrNo(
+                            'Enable live packet capture statistics?',
+                            default=args.captureStats,
+                            extraLabel=BACK_LABEL,
+                        )
 
                     if pcapNetSniff or pcapTcpDump or liveArkime or liveZeek or liveSuricata:
                         pcapIface = ''
@@ -2313,6 +2320,17 @@ def tweak_malcolm_runtime(self, malcolm_install_path):
                 'SURICATA_LIVE_CAPTURE',
                 TrueOrFalseNoQuote(liveSuricata),
             ),
+            # live capture statistics for Suricata
+            EnvValue(
+                os.path.join(args.configDir, 'suricata-live.env'),
+                'SURICATA_STATS_ENABLED',
+                TrueOrFalseNoQuote(captureStats),
+            ),
+            EnvValue(
+                os.path.join(args.configDir, 'suricata-live.env'),
+                'SURICATA_STATS_EVE_ENABLED',
+                TrueOrFalseNoQuote(captureStats),
+            ),
             # rotated captured PCAP analysis with Suricata (not live capture)
             EnvValue(
                 os.path.join(args.configDir, 'suricata-offline.env'),
@@ -2421,6 +2439,12 @@ def tweak_malcolm_runtime(self, malcolm_install_path):
                 'ZEEK_LIVE_CAPTURE',
                 TrueOrFalseNoQuote(liveZeek),
             ),
+            # live capture statistics for Zeek
+            EnvValue(
+                os.path.join(args.configDir, 'zeek-live.env'),
+                'ZEEK_DISABLE_STATS',
+                TrueOrFalseNoQuote(not captureStats),
+            ),
             # rotated captured PCAP analysis with Zeek (not live capture)
             EnvValue(
                 os.path.join(args.configDir, 'zeek-offline.env'),
@@ -4504,6 +4528,16 @@ def main():
         default=True,
         help="Disable capture interface hardware offloading and adjust ring buffer sizes",
     )
+    captureArgGroup.add_argument(
+        '--live-capture-stats',
+        dest='captureStats',
+        type=str2bool,
+        metavar="true|false",
+        nargs='?',
+        const=True,
+        default=False,
+        help=f"Enable live packet capture statistics for Zeek and/or Suricata",
+    )
     captureArgGroup.add_argument(
         '--live-capture-arkime',
         dest='liveArkime',