documentation and screenshot changes

dashboards/dashboards/9ee51f94-3316-4fc5-bd89-93a52af69714.json

@@ -353,7 +353,7 @@
       "version": "WzEyODAsMV0=",
       "attributes": {
         "title": "Extracted File Downloads",
-        "visState": "{\"title\":\"Extracted File Downloads\",\"type\":\"transform\",\"aggs\":[],\"params\":{\"meta\":\"({})\",\"multiquerydsl\":\"{\\n  \\\"topn\\\": {\\n    \\\"index\\\": \\\"MALCOLM_NETWORK_INDEX_PATTERN_REPLACER\\\",\\n    \\\"query\\\": {\\n      \\\"bool\\\": {\\n        \\\"must\\\": [\\n          \\\"_DASHBOARD_CONTEXT_\\\",\\n          \\\"_TIME_RANGE_[firstPacket]\\\",\\n          {\\n            \\\"match\\\": {\\n              \\\"event.dataset\\\": \\\"files\\\"\\n            }\\n          },\\n          {\\n            \\\"match\\\": {\\n              \\\"event.provider\\\": \\\"zeek\\\"\\n            }\\n          }\\n        ]\\n      }\\n    },\\n    \\\"aggs\\\": {\\n      \\\"uris\\\": {\\n        \\\"terms\\\": {\\n          \\\"field\\\": \\\"zeek.files.extracted_uri\\\",\\n          \\\"size\\\": 10,\\n          \\\"order\\\": { \\\"_key\\\": \\\"asc\\\" }\\n        }\\n      }\\n    }\\n  }\\n}\",\"formula\":\"<link rel=\\\"stylesheet\\\" href=\\\"/css/styles.css\\\">\\n\\n<h2><a href=\\\"/readme/docs/file-scanning.html#ZeekFileExtraction\\\" target=\\\"_blank\\\">Extracted File</a> Downloads</h2>\\n<p><small>Only the first 10 matching results are displayed, sorted alphabetically. Apply filters ⊕ to narrow scope.</small></p>\\n<table class=\\\"table caption-top\\\">\\n  <thead>\\n    <tr>\\n      <th scope=\\\"col\\\">Download Link (if preserved)</th>\\n    </tr>\\n  </thead>\\n  <tbody>\\n    {{#response.topn.aggregations.uris.buckets}} \\n    <tr>\\n      <th scope=\\\"row\\\"><a href=\\\"/{{key}}\\\" target=\\\"_blank\\\">💾 {{key}}</a></th>\\n    </tr>\\n    {{/response.topn.aggregations.uris.buckets}} \\n    </tbody>\\n</table>\\n<p><small>You can also <a href=\\\"/extracted-files/\\\" target=\\\"_blank\\\">📁<strong>Browse extracted files</strong></a>. See <em><a href=\\\"/readme/docs/file-scanning.html#ZeekFileExtraction\\\" target=\\\"_blank\\\">Automatic file extraction and scanning</a></em> for more information.</small></p>\"}}",
+        "visState": "{\"title\":\"Extracted File Downloads\",\"type\":\"transform\",\"aggs\":[],\"params\":{\"meta\":\"({})\",\"multiquerydsl\":\"{\\n  \\\"topn\\\": {\\n    \\\"index\\\": \\\"MALCOLM_NETWORK_INDEX_PATTERN_REPLACER\\\",\\n    \\\"query\\\": {\\n      \\\"bool\\\": {\\n        \\\"must\\\": [\\n          \\\"_DASHBOARD_CONTEXT_\\\",\\n          \\\"_TIME_RANGE_[firstPacket]\\\",\\n          {\\n            \\\"match\\\": {\\n              \\\"event.dataset\\\": \\\"files\\\"\\n            }\\n          },\\n          {\\n            \\\"match\\\": {\\n              \\\"event.provider\\\": \\\"zeek\\\"\\n            }\\n          }\\n        ]\\n      }\\n    },\\n    \\\"aggs\\\": {\\n      \\\"uris\\\": {\\n        \\\"terms\\\": {\\n          \\\"field\\\": \\\"zeek.files.extracted_uri\\\",\\n          \\\"size\\\": 10,\\n          \\\"order\\\": { \\\"_key\\\": \\\"asc\\\" }\\n        }\\n      }\\n    }\\n  }\\n}\",\"formula\":\"<link rel=\\\"stylesheet\\\" href=\\\"/css/styles.css\\\">\\n\\n<h6><a href=\\\"/readme/docs/file-scanning.html#ZeekFileExtraction\\\" target=\\\"_blank\\\">Extracted File</a> Downloads</h6>\\n<p><small>Only the first 10 matching results are displayed, sorted alphabetically. Apply filters ⊕ to narrow scope.</small></p>\\n<table class=\\\"table caption-top\\\">\\n  <thead>\\n    <tr>\\n      <th scope=\\\"col\\\">Download Link (if preserved)</th>\\n    </tr>\\n  </thead>\\n  <tbody>\\n    {{#response.topn.aggregations.uris.buckets}} \\n    <tr>\\n      <th scope=\\\"row\\\"><a href=\\\"/{{key}}\\\" target=\\\"_blank\\\">💾 {{key}}</a></th>\\n    </tr>\\n    {{/response.topn.aggregations.uris.buckets}} \\n    </tbody>\\n</table>\\n<p><small>You can also <a href=\\\"/extracted-files/\\\" target=\\\"_blank\\\">📁<strong>Browse extracted files</strong></a>. See <em><a href=\\\"/readme/docs/file-scanning.html#ZeekFileExtraction\\\" target=\\\"_blank\\\">Automatic file extraction and scanning</a></em> for more information.</small></p>\"}}",
         "uiStateJSON": "{}",
         "description": "",
         "version": 1,

docs/file-scanning.md

@@ -52,14 +52,14 @@ The `EXTRACTED_FILE_HTTP_SERVER_…` [environment variables in `zeek.env` and `z
 
 The files extracted by Zeek and the data about those files can be accessed through several of Malcolm's user interfaces.
 
-* The [Files dashboard](dashboards.md#PrebuiltVisualizations) summarizes the file transfers observed in network traffic:
+* The [Files dashboard](dashboards.md#PrebuiltVisualizations) summarizes the file transfers observed in network traffic. The **Extracted File Downloads** table provides download links for the extracted files matching the currently applied filters. Note that the presence of these links don't necessarily imply that the files they represent are available: depending on factors such as file preservation settings (above) and retention policies, files that were extracted and scanned may no longer be available. When this is the case, clicking one of the file download links will result in a "file not found" error. If one of these links refers to a file that was extracted and scanned on a [Hedgehog Linux](hedgehog.md) network sensor, Malcolm must be able to communicate with that sensor in order to retrieve and download the file.
 
 ![The files dashboard displays metrics about the files transferred over the network](./images/screenshots/dashboards_files_source.png)
 
-* Viewing logs from Zeek's `files.log` (e.g., `event.provider == zeek && event.dataset == files`), the Arkime [session](arkime.md#ArkimeSessions) detail's **Extracted Filename** field can be clicked for a context menu item to **Download** the extracted file, if it was preserved as described above.
+* Viewing logs from Zeek's `files.log` (e.g., `event.provider == zeek && event.dataset == files`), the Arkime [session](arkime.md#ArkimeSessions) detail's **Extracted Filename URL** field can be clicked for a context menu item to download the extracted file, if it was preserved as described above.
 
 ![Arkime's session details for files.log entries](./images/screenshots/arkime_sessions_files_log_dl.png)
 
-* Malcolm provides an extracted files directory listing to browse and download Zeek-extracted files. This interface is available at at **https://localhost/extracted-files/** if connecting locally. The Zeek `uid` and `fuid` values associated with these files and the sessions from which they were extracted are listed in the **IDs** column as filter links back into Dashboards.
+* Malcolm provides an extracted files directory listing to browse and download Zeek-extracted files. This interface is available at **https://localhost/extracted-files/** if connecting locally. The Zeek `uid` and `fuid` values associated with these files and the sessions from which they were extracted are listed in the **IDs** column as filter links back into Dashboards. Similarly, files extracted and preserved on a [Hedgehog Linux](hedgehog.md) network sensor can be accessed at **https://localhost/hh-extracted-files/X.X.X.X/**, where **X.X.X.X** represents the IP address or hostname of the sensor (e.g., `https://localhost/hh-extracted-files/192.168.122.57/` if the sensor's IP address were 192.168.122.57).
 
 ![The extracted files directory interface](./images/screenshots/extracted_files_dl_ui.png)

docs/system-requirements.md

@@ -2,6 +2,6 @@
 
 Malcolm runs on top of [Docker](https://www.docker.com/), which runs on recent releases of Linux, Apple [macOS](host-config-macos.md#HostSystemConfigMac), and [Microsoft Windows](host-config-windows.md#HostSystemConfigWindows) 10 and up. Malcolm can also be deployed in the cloud [with Kubernetes](kubernetes.md#Kubernetes).
 
-To quote the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html), "If there is one resource that you will run out of first, it will likely be memory." Malcolm developers recommend a minimum of 8 cores and 16 gigabytes of RAM on a dedicated server. Malcolm can run on less, but more is better. Of course, users will want as much hard drive space as possible, as the amount of PCAP data a machine can analyze and store will be limited by its hard drive.
+To quote the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html), "If there is one resource that you will run out of first, it will likely be memory." Malcolm requires a minimum of 8 CPU cores and 16 gigabytes of RAM on a dedicated server, but Malcolm developers recommend 16+ CPU cores and 32+ gigabytes of RAM for an optimal experience. Users will want as much available disk storage as possible (preferrably solid state storage), as the amount of PCAP data a machine can analyze and store will be limited by available storage space.
 
 Arkime's wiki has documents ([here](https://github.com/arkime/arkime#hardware-requirements) and [here](https://github.com/arkime/arkime/wiki/FAQ#what-kind-of-capture-machines-should-we-buy) and [here](https://github.com/arkime/arkime/wiki/FAQ#how-many-elasticsearch-nodes-or-machines-do-i-need) and a [calculator here](https://arkime.com/estimators)) that may be helpful, although not everything in those documents will apply to a Docker-based setup such as Malcolm.