Staging (#7)

* Bump development for v25.01.0, also update copyright year * bump netbox to v4.1.10, osd_transform to v2.18.0, and fluent-bit to v3.2.4 * for cisagov#354, work in progress for Malcolm directly accepting syslog * for cisagov#354, work in progress for Malcolm directly accepting syslog; (dashboard) * cisagov#543, add naviation pane to non-network dashboards * bump jinja to 3.1.5 * Documentation for cisagov#354, syslog * replace old filebeat input for syslog with tcp/udp input and syslog processor, for cisagov#354 * Documentation for cisagov#354, syslog * install.py tweak for cisagov#354 * minor fix for for cisagov#354, set host.name correctly * bump netbox to v4.11.1 and elasticsearch-dsl to v8.17.1 * start of cisagov#356, normalize winlogbeats * WIP of cisagov#356, normalize winlogbeats * WIP of cisagov#356, normalize winlogbeats * WIP of cisagov#356, fix for a dashboard * WIP of cisagov#356, normalize winlogbeats * Work in progress for cisagov#541, making sure conn.log and known_services.log get the ICS protocols assigned to them corrrectly and tagged appropriately * Work in progress for cisagov#541 * standardize ICS protocols in network.protocol field, so they all get tagged with 'ics' properly cisagov#541 * fix cisagov#533, allow keystores to be created on startup even in hedgehog mode * forgot to add file for cisagov#356 * For cisagov#524, handle filenames with spaces in extracted_files_http_server.py * work for cisagov#542, preserve custom field formatting for index pattern on update of index pattern * work for cisagov#542, preserve custom field formatting for index pattern on update of index pattern * bump yq to v4.45.1 * for cisagov#551, URL pivot links from dashboards to arkime * for cisagov#551, URL pivot links from dashboards to arkime * fix pivot from arkime to dashboards and vice-versa when using a traefik or other reverse proxy * for cisagov#551, URL pivot links from dashboards to netbox * for cisagov#551, URL pivot links from dashboards to netbox * for cisagov#551, URL pivot links from netbox to arkime/dashboards * start of cisagov#553, update zeek to v7.1.0 * cisagov#553, handle conn.log for zeek v7.1.0 and documentation update * cisagov#553, handle postgresql.log * cisagov#553, handle postgresql.log * cisagov#553, added PostgreSQL dashboard * for cisagov#551, URL pivot links in dashboards (ignore date/times) * start of omron fins integration, cisagov#554 * wip omron fins integration, , cisagov#554 * arkime to v5.6.0 * bump logstash and filebeat to v8.17.0 * Fix nginx filebeat * WIP omron fins integration, cisagov#554 * WIP omron fins integration, cisagov#554 * WIP omron fins integration, cisagov#554 * WIP omron fins integration, cisagov#554 * WIP omron fins integration, cisagov#554 * dashboards tweaks * fix links for hh redirect download * First pass at adding suricata socket optimization * fix issue with nginx proxy * Setting debug to false * Fixing permissions for socket * html formatting * documentation for workaround for UFW software firewall for Malcolm ISO should automatically open ports for syslog cisagov#560) * Bump for v25.02.0 development * restore _config.yml * fix version * I don't think we need a seperate pod for the socket-based suricata, that's what the offline one does now anyway, right? * restore some comments, black python style * some tweaks for cisagov#457, pulled jjrush's branch into mine for some fixes * some tweaks for cisagov#457 * allow suricata to spawn threads * logging tweaks * more flexible verbosity for suricata * some tweaks for cisagov#457, try to wait until PCAP is finished processing before moving on * First pass at adding suricata socket optimization * Setting debug to false * Fixing permissions for socket * for cisagov#457, a few tweaks of the suricata pcap processing mode after reviewing @jjrush's code * for cisagov#457, monitor suricata.log to know when PCAP is done processing * for cisagov#457, monitor suricata.log to know when PCAP is done processing * for cisagov#457, signal suricata rules to reload after update * for cisagov#457, signal suricata rules to reload after update * for cisagov#457, fix processing of other log types * for cisagov#457, fix processing of other log types * for cisagov#457, signal suricata rules to reload after update * decrease verbosity for log * fix logic for autoarkime/forcearkime * some tweaks for cisagov#457, don't bother keeping track of when suricata is done with a PCAP file. just let filebeat handle it and pick up the resultant eve.json files directly * Standardizing healthcheck scripts, updating docker-compose, updating kubernetes * Adding livenessProbe to htadmin * cisagov#457, handle multiple Suricata PCAP processing threads * cisagov#574, clear screen after auth_setup when using Dialog mode * add the related.user field to the 'nginx Access Logs' table * bump fluent bit to v3.2.5 * fixed import of ECS templates * handle ARKIME_PORT value formatted like a URL in the init of the API container * cisagov#565, warn user about overwriting netbox passwords if they've already been set * fix cisagov#559, ANSI color codes from croc displayed * Exception in build triggers * for cisagov#557, try building dirinit with arm runner * cisagov#557, use arm-hosted runners for github build actions * restore _config.yml * a bit of cleanup for Dockefiles/health check scripts * minor fixes for health checks * Tweaks for health checks * restore _config.yml * Tweaks for health checks * build tweaks for health scripts * bump capa to v9.0.0 * workaround for issue blocking cisagov#475, integration of sigma rules * improvements to workaround for issue blocking cisagov#475, integration of sigma rules * improvements to workaround for issue blocking cisagov#475, integration of sigma rules * for cisagov#475, automatically apply aliases via index templates * for cisagov#475, starting on mappings for security analytics * for cisagov#585, include corelight/zeek-long-connections plugin for long connections (wIP) * for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP) * for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP) * for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP) * demo fix * for cisagov#585, show long connection count on connections dashboard * decouple redis from netbox (cisagov#580) * one more minor change to cisagov#491, moved all container health scripts into one place to make it easier to keep track of them * decouple redis from netbox (cisagov#580) and reorganized some of the other netbox password stuff * updated fluent bit * fix filebeat health --------- Co-authored-by: Seth Grover <seth.d.grover@gmail.com> Co-authored-by: Jason Rush <jjrush-github@proton.me>
piercema · Feb 12, 2025 · 278d974 · 278d974
1 parent 2630938
commit 278d974
Show file tree

Hide file tree

Showing 159 changed files with 2,784 additions and 2,109 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -39,8 +39,8 @@ scripts
 !scripts/malcolm_utils.py
 zeek-logs
 suricata-logs
-netbox/netbox/media
-netbox/netbox/postgres
-netbox/netbox/redis
+redis
+netbox/media
+netbox/postgres
 netbox/custom-plugins
 zeek/custom
diff --git a/.github/workflows/api-build-and-push-ghcr.yml b/.github/workflows/api-build-and-push-ghcr.yml
@@ -11,6 +11,7 @@ on:
       - 'shared/bin/*'
       - '!shared/bin/capa-build.sh'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/extracted_files_http_server.py'
       - '!shared/bin/web-ui-asset-download.sh'
@@ -24,7 +25,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -33,9 +34,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -72,12 +77,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -93,7 +92,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/api.Dockerfile

diff --git a/.github/workflows/arkime-build-and-push-ghcr.yml b/.github/workflows/arkime-build-and-push-ghcr.yml
@@ -11,6 +11,7 @@ on:
       - 'shared/bin/*'
       - '!shared/bin/capa-build.sh'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/extracted_files_http_server.py'
       - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
@@ -24,7 +25,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -33,9 +34,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -72,12 +77,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -93,7 +92,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/arkime.Dockerfile

diff --git a/.github/workflows/dashboards-build-and-push-ghcr.yml b/.github/workflows/dashboards-build-and-push-ghcr.yml
@@ -11,6 +11,7 @@ on:
       - 'shared/bin/*'
       - '!shared/bin/capa-build.sh'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/extracted_files_http_server.py'
       - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
@@ -24,7 +25,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -33,9 +34,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -72,12 +77,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -93,7 +92,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/dashboards.Dockerfile

diff --git a/.github/workflows/dashboards-helper-build-and-push-ghcr.yml b/.github/workflows/dashboards-helper-build-and-push-ghcr.yml
@@ -11,6 +11,7 @@ on:
       - 'shared/bin/*'
       - '!shared/bin/capa-build.sh'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/extracted_files_http_server.py'
       - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
@@ -24,7 +25,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -33,9 +34,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -72,12 +77,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -93,7 +92,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/dashboards-helper.Dockerfile

diff --git a/.github/workflows/dirinit-build-and-push-ghcr.yml b/.github/workflows/dirinit-build-and-push-ghcr.yml
@@ -14,7 +14,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -23,9 +23,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -62,12 +66,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -83,7 +81,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/dirinit.Dockerfile

diff --git a/.github/workflows/file-monitor-build-and-push-ghcr.yml b/.github/workflows/file-monitor-build-and-push-ghcr.yml
@@ -10,6 +10,7 @@ on:
       - 'Dockerfiles/file-monitor.Dockerfile'
       - 'shared/bin/*'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
@@ -21,7 +22,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -30,9 +31,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -69,12 +74,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -90,7 +89,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/file-monitor.Dockerfile

diff --git a/.github/workflows/file-upload-build-and-push-ghcr.yml b/.github/workflows/file-upload-build-and-push-ghcr.yml
@@ -11,6 +11,7 @@ on:
       - 'shared/bin/*'
       - '!shared/bin/capa-build.sh'
       - '!shared/bin/common-init.sh'
+      - '!shared/bin/tx-rx-secure.sh'
       - '!shared/bin/extracted_files_http_server.py'
       - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
@@ -24,7 +25,7 @@ on:
 
 jobs:
   docker:
-    runs-on: ubuntu-22.04
+    runs-on: ${{ matrix.os }}
     permissions:
       actions: write
       packages: write
@@ -33,9 +34,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        platform:
-          - linux/amd64
-          - linux/arm64
+        include:
+          - os: ubuntu-24.04
+            arch: amd64
+            platform: linux/amd64
+          - os: ubuntu-24.04-arm
+            arch: arm64
+            platform: linux/arm64
     steps:
       -
         name: Cancel previous run in progress
@@ -72,12 +77,6 @@ jobs:
         shell: bash
         run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
         id: extract_malcolm_version
-      -
-        name: Set up QEMU
-        id: setup-qemu
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: ${{ matrix.platform }}
       -
         name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
@@ -93,7 +92,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfiles/file-upload.Dockerfile