Skip to content

Commit

Permalink
Staging (#7)
Browse files Browse the repository at this point in the history 
* Bump development for v25.01.0, also update copyright year

* bump netbox to v4.1.10, osd_transform to v2.18.0, and fluent-bit to v3.2.4

* for cisagov#354, work in progress for Malcolm directly accepting syslog

* for cisagov#354, work in progress for Malcolm directly accepting syslog; (dashboard)

* cisagov#543, add naviation pane to non-network dashboards

* bump jinja to 3.1.5

* Documentation for cisagov#354, syslog

* replace old filebeat input for syslog with tcp/udp input and syslog processor, for cisagov#354

* Documentation for cisagov#354, syslog

* install.py tweak for cisagov#354

* minor fix for for cisagov#354, set host.name correctly

* bump netbox to v4.11.1 and elasticsearch-dsl to v8.17.1

* start of cisagov#356, normalize winlogbeats

* WIP of cisagov#356, normalize winlogbeats

* WIP of cisagov#356, normalize winlogbeats

* WIP of cisagov#356, fix for a dashboard

* WIP of cisagov#356, normalize winlogbeats

* Work in progress for cisagov#541, making sure conn.log and known_services.log get the ICS protocols assigned to them corrrectly and tagged appropriately

* Work in progress for cisagov#541

* standardize ICS protocols in network.protocol field, so they all get tagged with 'ics' properly cisagov#541

* fix cisagov#533, allow keystores to be created on startup even in hedgehog mode

* forgot to add file for cisagov#356

* For cisagov#524, handle filenames with spaces in extracted_files_http_server.py

* work for cisagov#542, preserve custom field formatting for index pattern on update of index pattern

* work for cisagov#542, preserve custom field formatting for index pattern on update of index pattern

* bump yq to v4.45.1

* for cisagov#551, URL pivot links from dashboards to arkime

* for cisagov#551, URL pivot links from dashboards to arkime

* fix pivot from arkime to dashboards and vice-versa when using a traefik or other reverse proxy

* for cisagov#551, URL pivot links from dashboards to netbox

* for cisagov#551, URL pivot links from dashboards to netbox

* for cisagov#551, URL pivot links from netbox to arkime/dashboards

* start of cisagov#553, update zeek to v7.1.0

* cisagov#553, handle conn.log for zeek v7.1.0 and documentation update

* cisagov#553, handle postgresql.log

* cisagov#553, handle postgresql.log

* cisagov#553, added PostgreSQL dashboard

* for cisagov#551, URL pivot links in dashboards (ignore date/times)

* start of omron fins integration, cisagov#554

* wip omron fins integration, , cisagov#554

* arkime to v5.6.0

* bump logstash and filebeat to v8.17.0

* Fix nginx filebeat

* WIP omron fins integration, cisagov#554

* WIP omron fins integration, cisagov#554

* WIP omron fins integration, cisagov#554

* WIP omron fins integration, cisagov#554

* WIP omron fins integration, cisagov#554

* dashboards tweaks

* fix links for hh redirect download

* First pass at adding suricata socket optimization

* fix issue with nginx proxy

* Setting debug to false

* Fixing permissions for socket

* html formatting

* documentation for workaround for UFW software firewall for Malcolm ISO should automatically open ports for syslog cisagov#560)

* Bump for v25.02.0 development

* restore _config.yml

* fix version

* I don't think we need a seperate pod for the socket-based suricata, that's what the offline one does now anyway, right?

* restore some comments, black python style

* some tweaks for cisagov#457, pulled jjrush's branch into mine for some fixes

* some tweaks for cisagov#457

* allow suricata to spawn threads

* logging tweaks

* more flexible verbosity for suricata

* some tweaks for cisagov#457, try to wait until PCAP is finished processing before moving on

* First pass at adding suricata socket optimization

* Setting debug to false

* Fixing permissions for socket

* for cisagov#457, a few tweaks of the suricata pcap processing mode after reviewing @jjrush's code

* for cisagov#457, monitor suricata.log to know when PCAP is done processing

* for cisagov#457, monitor suricata.log to know when PCAP is done processing

* for cisagov#457, signal suricata rules to reload after update

* for cisagov#457, signal suricata rules to reload after update

* for cisagov#457, fix processing of other log types

* for cisagov#457, fix processing of other log types

* for cisagov#457, signal suricata rules to reload after update

* decrease verbosity for log

* fix logic for autoarkime/forcearkime

* some tweaks for cisagov#457, don't bother keeping track of when suricata is done with a PCAP file. just let filebeat handle it and pick up the resultant eve.json files directly

* Standardizing healthcheck scripts, updating docker-compose, updating kubernetes

* Adding livenessProbe to htadmin

* cisagov#457, handle multiple Suricata PCAP processing threads

* cisagov#574, clear screen after auth_setup when using Dialog mode

* add the related.user field to the 'nginx Access Logs' table

* bump fluent bit to v3.2.5

* fixed import of ECS templates

* handle ARKIME_PORT value formatted like a URL in the init of the API container

* cisagov#565, warn user about overwriting netbox passwords if they've already been set

* fix cisagov#559, ANSI color codes from croc displayed

* Exception in build triggers

* for cisagov#557, try building dirinit with arm runner

* cisagov#557, use arm-hosted runners for github build actions

* restore _config.yml

* a bit of cleanup for Dockefiles/health check scripts

* minor fixes for health checks

* Tweaks for health checks

* restore _config.yml

* Tweaks for health checks

* build tweaks for health scripts

* bump capa to v9.0.0

* workaround for issue blocking cisagov#475, integration of sigma rules

* improvements to workaround for issue blocking cisagov#475, integration of sigma rules

* improvements to workaround for issue blocking cisagov#475, integration of sigma rules

* for cisagov#475, automatically apply aliases via index templates

* for cisagov#475, starting on mappings for security analytics

* for cisagov#585, include corelight/zeek-long-connections plugin for long connections (wIP)

* for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP)

* for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP)

* for cisagov#585, include corelight/zeek-long-connections plugin for long connections (WIP)

* demo fix

* for cisagov#585, show long connection count on connections dashboard

* decouple redis from netbox (cisagov#580)

* one more minor change to cisagov#491, moved all container health scripts into one place to make it easier to keep track of them

* decouple redis from netbox (cisagov#580) and reorganized some of the other netbox password stuff

* updated fluent bit

* fix filebeat health

---------

Co-authored-by: Seth Grover <seth.d.grover@gmail.com>
Co-authored-by: Jason Rush <jjrush-github@proton.me>
  • Loading branch information
@piercema @mmguero @jjrush
3 people committed Feb 13, 2025
1 parent eee23bc commit b9036d4
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 5 deletions.
2 changes: 1 addition & 1 deletion dashboards/dashboards/abdd7550-2c7c-40dc-947e-f6d186a158c4.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@
"title": "Connections - Log Count Over Time",
"uiStateJSON": "{\"vis\":{\"legendOpen\":false}}",
"version": 1,
"visState": "{\"title\":\"Connections - Log Count Over Time\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER\",\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"auto\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\" \"},\"schema\":\"segment\"}],\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER per 12 hours\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"normal\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"linear\",\"lineWidth\":2,\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"showCircles\":true,\"interpolate\":\"linear\",\"scale\":\"linear\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"type\":\"histogram\",\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"
"visState": "{\"title\":\"Connections - Log Count Over Time\",\"type\":\"line\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"params\":{\"field\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER\",\"timeRange\":{\"from\":\"now-25y\",\"to\":\"now\"},\"useNormalizedOpenSearchInterval\":true,\"scaleMetricValues\":false,\"interval\":\"auto\",\"drop_partials\":false,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\" \"},\"schema\":\"segment\"}],\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"MALCOLM_NETWORK_INDEX_TIME_FIELD_REPLACER per 12 hours\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"normal\",\"type\":\"histogram\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"linear\",\"lineWidth\":2,\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"showCircles\":true,\"interpolate\":\"linear\",\"scale\":\"linear\",\"drawLinesBetweenPoints\":true,\"radiusRatio\":9,\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"type\":\"histogram\",\"labels\":{},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"
},
"id": "03eba854-72b5-47d0-a92a-b671a0d7ed19",
"migrationVersion": {
Expand Down
8 changes: 4 additions & 4 deletions dashboards/scripts/shared-object-creation.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -589,21 +589,21 @@ if [[ "${CREATE_OS_ARKIME_SESSION_INDEX:-true}" = "true" ]] ; then
# OpenSearch security analytics fields mappings
echo "Creating $DATASTORE_TYPE security analytics mappings..."

SA_MAPPINGS_IMPORT_DIR="$(mktemp -p "$TMP_WORK_DIR" -d -t sa-mappings-XXXXXX)"
SA_MAPPINGS_IMPORT_DIR="$(mktemp -d -t sa-mappings-XXXXXX)"
rsync -a /opt/security_analytics_mappings/ "$SA_MAPPINGS_IMPORT_DIR"/
DoReplacersForDir "$SA_MAPPINGS_IMPORT_DIR" "$DATASTORE_TYPE" sa_mapping
for i in "${SA_MAPPINGS_IMPORT_DIR}"/*.json; do
set +e
RULE_TOPIC="$(jq -r '.rule_topic' 2>/dev/null < "$i")"
INDEX_NAME="$(jq -r '.index_name' 2>/dev/null < "$i")"
echo "Creating mappings for \"${INDEX_NAME}\" / \"${RULE_TOPIC}\" ..." && \
CURL_OUT=$(get_tmp_output_filename)
curl "${CURL_CONFIG_PARAMS[@]}" --location --fail-with-body --output "$CURL_OUT" --silent \
curl "${CURL_CONFIG_PARAMS[@]}" -w "\n" --location --silent --output /dev/null --show-error \
-XPOST "$OPENSEARCH_URL_TO_USE/_plugins/_security_analytics/mappings" \
-H "$XSRF_HEADER:true" -H 'Content-type:application/json' \
-d "@$i" || ( cat "$CURL_OUT" && echo )
-d "@$i"
set -e
done
rm -rf "${SA_MAPPINGS_IMPORT_DIR}"

# end OpenSearch security analytics
#############################################################################################################################
Expand Down

0 comments on commit b9036d4

Please sign in to comment.