Merge remote-tracking branch 'origin/8.x' into mergify/bp/8.x/pr-42041

CHANGELOG.asciidoc

@@ -3,6 +3,91 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-8.17.0]]
+=== Beats version 8.17.0
+https://github.com/elastic/beats/compare/v8.16.1\...v8.17.0[View commits]
+
+==== Known issue
+
+- Standalone Beats docker image will not start if `-e` option is not added {issue}42038[42038].
+
+==== Breaking changes
+
+*Affecting all Beats*
+
+- Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31. {pull}41402[41402]
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Ensure Elasticsearch output can always recover from network errors. {pull}40794[40794]
+- Add `translate_ldap_attribute` processor. {pull}41472[41472]
+- Remove unnecessary debug logs during idle connection teardown. {issue}40824[40824]
+- Remove unnecessary reload for Elastic Agent managed beats when APM tracing config changes from nil to nil. {pull}41794[41794]
+
+*Auditbeat*
+
+- auditd: Use ECS `event.type: end` instead of `stop` for SERVICE_STOP, DAEMON_ABORT, and DAEMON_END messages. {pull}41558[41558]
+- auditd: Update syscall names for Linux 6.11. {pull}41558[41558]
+- hasher: Geneneral improvements and fixes. {pull}41863[41863]
+
+*Filebeat*
+
+- Fix double encoding of client_secret in the Entity Analytics input's Azure Active Directory provider. {pull}41393[41393]
+- Add support for Access Points in the `aws-s3` input. {pull}41495[41495]
+- Fix the "No such input type exist: 'salesforce'" error on the Windows/AIX platform. {pull}41664[41664]
+- Fix handling of http_endpoint request exceeding memory limits. {issue}41764[41764] {pull}41765[41765]
+- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]
+
+*Metricbeat*
+
+- Log Cisco Meraki `getDevicePerformanceScores` errors without stopping metrics collection. {pull}41622[41622]
+- Fix incorrect handling of types in SQL module. {issue}40090[40090] {pull}41607[41607]
+
+*Winlogbeat*
+
+- Fix message handling in the experimental API. {issue}19338[19338] {pull}41730[41730]
+
+==== Added
+
+*Affecting all Beats*
+
+- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
+- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
+- Replace `compress/gzip` with https://github.com/klauspost/compress/gzip library for gzip compression. {pull}41584[41584]
+
+*Auditbeat*
+
+- Split module/system/process into common and provider bits. {pull}41868[41868]
+
+*Filebeat*
+
+- Improved Azure Blob Storage input documentation. {pull}41252[41252]
+- Make ETW input GA. {pull}41389[41389]
+- Added input metrics to GCS input. {issue}36640[36640] {pull}41505[41505]
+- Add support for Okta entity analytics provider to collect role and factor data for users. {pull}41460[41460]
+- Add support for Journald in the System module. {pull}41555[41555]
+- Improve S3 polling mode states registry when using list prefix option. {pull}41869[41869]
+- AWS S3 input registry cleanup for untracked s3 objects. {pull}41694[41694]
+- The environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true` enables internal logs tracer for the azure-eventhub input. {issue}41931[41931] {pull}41932[41932]
+
+*Libbeat*
+
+- Enrich events with EC2 tags in add_cloud_metadata processor. {pull}41477[41477]
+
+*Metricbeat*
+
+- Add `id` field to all the vSphere metricsets. {pull}41097[41097]
+- Bump aerospike-client-go to version v7.7.1 and add support for basic auth in Aerospike module. {pull}41233[41233]
+- Add support for region/zone for Vertex AI service in GCP module. {pull}41551[41551]
+- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]
+
+*Winlogbeat*
+
+- Add handling for missing `EvtVarType`s in experimental API. {issue}19337[19337] {pull}41418[41418]
+- Implement exclusion range support for event_id. {issue}38623[38623] {pull}41639[41639]
+
 [[release-notes-8.16.1]]
 === Beats version 8.16.1
 https://github.com/elastic/beats/compare/v8.16.0\...v8.16.1[View commits]

CHANGELOG.next.asciidoc

@@ -113,6 +113,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Remove unnecessary reload for Elastic Agent managed beats when apm tracing config changes from nil to nil {pull}41794[41794]
 - Fix incorrect cloud provider identification in add_cloud_metadata processor using provider priority mechanism {pull}41636[41636]
 - Prevent panic if libbeat processors are loaded more than once. {issue}41475[41475] {pull}41857[51857]
+- Allow network condition to handle field values that are arrays of IP addresses. {pull}41918[41918]
 - Fix a bug where log files are rotated on startup when interval is configured and rotateonstartup is disabled {issue}41894[41894] {pull}41895[41895]
 
 *Auditbeat*
@@ -411,6 +412,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]
 - Add support for podman metrics in docker module. {pull}41889[41889]
 - Added `tier_preference`, `creation_date` and `version` fields to the `elasticsearch.index` metricset. {pull}41944[41944]
+- Add `use_performance_counters` to collect CPU metrics using performance counters on Windows for `system/cpu` and `system/core` {pull}41965[41965]
 
 *Metricbeat*
 - Add benchmark module {pull}41801[41801]

NOTICE.txt

@@ -13617,11 +13617,11 @@ Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-l
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/elastic-agent-system-metrics
-Version: v0.11.4
+Version: v0.11.5
 Licence type (autodetected): Apache-2.0
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-system-metrics@v0.11.4/LICENSE.txt:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-system-metrics@v0.11.5/LICENSE.txt:
 
                                  Apache License
                            Version 2.0, January 2004

go.mod

@@ -185,7 +185,7 @@ require (
 	github.com/elastic/ebpfevents v0.6.0
 	github.com/elastic/elastic-agent-autodiscover v0.9.0
 	github.com/elastic/elastic-agent-libs v0.17.4
-	github.com/elastic/elastic-agent-system-metrics v0.11.4
+	github.com/elastic/elastic-agent-system-metrics v0.11.5
 	github.com/elastic/go-elasticsearch/v8 v8.14.0
 	github.com/elastic/go-quark v0.2.0
 	github.com/elastic/go-sfdc v0.0.0-20241010131323-8e176480d727

go.sum

@@ -340,8 +340,8 @@ github.com/elastic/elastic-agent-client/v7 v7.15.0 h1:nDB7v8TBoNuD6IIzC3z7Q0y+7b
 github.com/elastic/elastic-agent-client/v7 v7.15.0/go.mod h1:6h+f9QdIr3GO2ODC0Y8+aEXRwzbA5W4eV4dd/67z7nI=
 github.com/elastic/elastic-agent-libs v0.17.4 h1:kWK5Kn2EQjM97yHqbeXv+cFAIti4IiI9Qj8huM+lZzE=
 github.com/elastic/elastic-agent-libs v0.17.4/go.mod h1:5CR02awPrBr+tfmjBBK+JI+dMmHNQjpVY24J0wjbC7M=
-github.com/elastic/elastic-agent-system-metrics v0.11.4 h1:Z/8CML5RKvGpi6/QUFok1K3EriBAv2kUAXnsk8hCifk=
-github.com/elastic/elastic-agent-system-metrics v0.11.4/go.mod h1:TTW2ysv78uHBQ68hG8TXiaX1m6f29ZHgGWb8XONYsU8=
+github.com/elastic/elastic-agent-system-metrics v0.11.5 h1:JSjXFEn8uYZ9hoC/GxZNMgJ622UoP96sjYP/49/Uvuo=
+github.com/elastic/elastic-agent-system-metrics v0.11.5/go.mod h1:nzkrGajQA29YNcfP62gfzhxX9an3/xdQ3RmfQNw9YTI=
 github.com/elastic/elastic-transport-go/v8 v8.6.0 h1:Y2S/FBjx1LlCv5m6pWAF2kDJAHoSjSRSJCApolgfthA=
 github.com/elastic/elastic-transport-go/v8 v8.6.0/go.mod h1:YLHer5cj0csTzNFXoNQ8qhtGY1GTvSqPnKWKaqQE3Hk=
 github.com/elastic/fsevents v0.0.0-20181029231046-e1d381a4d270 h1:cWPqxlPtir4RoQVCpGSRXmLqjEHpJKbR60rxh1nQZY4=

libbeat/conditions/conditions_test.go

@@ -105,6 +105,29 @@ var httpResponseTestEvent = &beat.Event{
 	},
 }
 
+var httpResponseEventIPList = &beat.Event{
+	Timestamp: time.Now(),
+	Fields: mapstr.M{
+		"@timestamp": "2024-12-05T09:51:23.642Z",
+		"ecs": mapstr.M{
+			"version": "8.11.0",
+		},
+		"host": mapstr.M{
+			"hostname": "testhost",
+			"os": mapstr.M{
+				"type":     "linux",
+				"family":   "debian",
+				"version":  "11 (bullseye)",
+				"platform": "debian",
+			},
+			"ip": []string{
+				"10.1.0.55",
+				"fe80::4001:aff:fe9a:55",
+			},
+		},
+	},
+}
+
 func testConfig(t *testing.T, expected bool, event *beat.Event, config *Config) {
 	t.Helper()
 	logp.TestingSetup()

libbeat/conditions/network.go

@@ -20,6 +20,7 @@ package conditions
 import (
 	"fmt"
 	"net"
+	"slices"
 	"strings"
 
 	"github.com/elastic/elastic-agent-libs/logp"
@@ -94,31 +95,31 @@ func (m multiNetworkMatcher) String() string {
 	return strings.Join(names, " OR ")
 }
 
+func makeMatcher(network string) (networkMatcher, error) {
+	m := singleNetworkMatcher{name: network, netContainsFunc: namedNetworks[network]}
+	if m.netContainsFunc == nil {
+		subnet, err := parseCIDR(network)
+		if err != nil {
+			return nil, err
+		}
+		m.netContainsFunc = subnet.Contains
+	}
+	return m, nil
+}
+
+func invalidTypeError(field string, value interface{}) error {
+	return fmt.Errorf("network condition attempted to set "+
+		"'%v' -> '%v' and encountered unexpected type '%T', only "+
+		"strings or []strings are allowed", field, value, value)
+}
+
 // NewNetworkCondition builds a new Network using the given configuration.
 func NewNetworkCondition(fields map[string]interface{}) (*Network, error) {
 	cond := &Network{
 		fields: map[string]networkMatcher{},
 		log:    logp.NewLogger(logName),
 	}
 
-	makeMatcher := func(network string) (networkMatcher, error) {
-		m := singleNetworkMatcher{name: network, netContainsFunc: namedNetworks[network]}
-		if m.netContainsFunc == nil {
-			subnet, err := parseCIDR(network)
-			if err != nil {
-				return nil, err
-			}
-			m.netContainsFunc = subnet.Contains
-		}
-		return m, nil
-	}
-
-	invalidTypeError := func(field string, value interface{}) error {
-		return fmt.Errorf("network condition attempted to set "+
-			"'%v' -> '%v' and encountered unexpected type '%T', only "+
-			"strings or []strings are allowed", field, value, value)
-	}
-
 	for field, value := range mapstr.M(fields).Flatten() {
 		switch v := value.(type) {
 		case string:
@@ -157,15 +158,17 @@ func (c *Network) Check(event ValuesMap) bool {
 			return false
 		}
 
-		ip := extractIP(value)
-		if ip == nil {
+		ipList := extractIP(value)
+		if len(ipList) == 0 {
 			c.log.Debugf("Invalid IP address in field=%v for network condition", field)
 			return false
 		}
-
-		if !network.Contains(ip) {
+		// match on an "any" basis when we find multiple IPs in the event;
+		// if the network matcher returns true for any seen IP, consider it a match
+		if !slices.ContainsFunc(ipList, network.Contains) {
 			return false
 		}
+
 	}
 
 	return true
@@ -202,12 +205,20 @@ func parseCIDR(value string) (*net.IPNet, error) {
 
 // extractIP return an IP address if unk is an IP address string or a net.IP.
 // Otherwise it returns nil.
-func extractIP(unk interface{}) net.IP {
+func extractIP(unk interface{}) []net.IP {
 	switch v := unk.(type) {
 	case string:
-		return net.ParseIP(v)
-	case net.IP:
+		return []net.IP{net.ParseIP(v)}
+	case []net.IP:
 		return v
+	case net.IP:
+		return []net.IP{v}
+	case []string:
+		parsed := make([]net.IP, len(v))
+		for i, rawIP := range v {
+			parsed[i] = net.ParseIP(rawIP)
+		}
+		return parsed
 	default:
 		return nil
 	}

libbeat/conditions/network_test.go

@@ -79,6 +79,26 @@ network:
 
 		testYAMLConfig(t, true, evt, yaml)
 	})
+
+	t.Run("IP list", func(t *testing.T) {
+		const yaml = `
+network:
+  ip:
+    client: [loopback]
+    server: [loopback]
+    host: 10.10.0.0/8
+`
+
+		evt := &beat.Event{Fields: mapstr.M{
+			"ip": mapstr.M{
+				"client": "127.0.0.1",
+				"server": "127.0.0.1",
+				"host":   []string{"10.10.0.83", "fe80::4001:aff:fe9a:53"},
+			},
+		}}
+
+		testYAMLConfig(t, true, evt, yaml)
+	})
 }
 
 func TestNetworkCreate(t *testing.T) {
@@ -166,6 +186,22 @@ func TestNetworkCheck(t *testing.T) {
 		})
 	})
 
+	t.Run("multiple IPs field single match", func(t *testing.T) {
+		testConfig(t, true, httpResponseEventIPList, &Config{
+			Network: map[string]interface{}{
+				"host.ip": "10.1.0.0/24",
+			},
+		})
+	})
+
+	t.Run("multiple IPs field negative match", func(t *testing.T) {
+		testConfig(t, false, httpResponseEventIPList, &Config{
+			Network: map[string]interface{}{
+				"host.ip": "127.0.0.0/24",
+			},
+		})
+	})
+
 	// Multiple conditions are treated as an implicit AND.
 	t.Run("multiple fields negative match", func(t *testing.T) {
 		testConfig(t, false, httpResponseTestEvent, &Config{
@@ -191,6 +227,22 @@ func TestNetworkCheck(t *testing.T) {
 			},
 		})
 	})
+
+	t.Run("multiple values multiple IPs match", func(t *testing.T) {
+		testConfig(t, true, httpResponseEventIPList, &Config{
+			Network: map[string]interface{}{
+				"host.ip": []interface{}{"10.1.0.0/24", "127.0.0.0/24"},
+			},
+		})
+	})
+
+	t.Run("multiple values multiple IPs no match", func(t *testing.T) {
+		testConfig(t, false, httpResponseEventIPList, &Config{
+			Network: map[string]interface{}{
+				"host.ip": []interface{}{"12.1.0.0/24", "127.0.0.0/24"},
+			},
+		})
+	})
 }
 
 func TestNetworkPrivate(t *testing.T) {

libbeat/docs/processors-using.asciidoc

@@ -311,10 +311,15 @@ range:
 [[condition-network]]
 ===== `network`
 
-The `network` condition checks if the field is in a certain IP network range.
-Both IPv4 and IPv6 addresses are supported. The network range may be specified
-using CIDR notation, like "192.0.2.0/24" or "2001:db8::/32", or by using one of
-these named ranges:
+The `network` condition checks whether a field's value falls within a specified
+IP network range. If multiple fields are provided, each field value must match
+its corresponding network range. You can specify multiple network ranges for a
+single field, and a match occurs if any one of the ranges matches. If the field
+value is an array of IPs, it will match if any of the IPs fall within any of the
+given ranges. Both IPv4 and IPv6 addresses are supported.
+
+The network range may be specified using CIDR notation, like "192.0.2.0/24" or 
+"2001:db8::/32", or by using one of these named ranges:
 
 - `loopback` - Matches loopback addresses in the range of `127.0.0.0/8` or
   `::1/128`.

metricbeat/docs/modules/system.asciidoc

@@ -265,14 +265,11 @@ metricbeat.modules:
 
   # Filter systemd services based on a name pattern
   #service.pattern_filter: ["ssh*", "nfs*"]
-<<<<<<< HEAD
-=======
 
   # This option enables the use of performance counters to collect data for cpu/core metricset.
   # Only effective for Windows.
   # You should use this option if running beats on machins with more than 64 cores.
   #use_performance_counters: false
->>>>>>> f3a063f1d (chore: disable performance counters (#42041))
 ----
 
 [float]

metricbeat/metricbeat.reference.yml

@@ -142,14 +142,11 @@ metricbeat.modules:
   # Filter systemd services based on a name pattern
   #service.pattern_filter: ["ssh*", "nfs*"]
 
-<<<<<<< HEAD
-=======
   # This option enables the use of performance counters to collect data for cpu/core metricset.
   # Only effective for Windows.
   # You should use this option if running beats on machins with more than 64 cores.
   #use_performance_counters: false
 
->>>>>>> f3a063f1d (chore: disable performance counters (#42041))
 #------------------------------ Aerospike Module ------------------------------
 - module: aerospike
   metricsets: ["namespace"]