[Filebeat][GSuite] Initial implementation of SAML and User Accounts f…

…ilesets (#19329)

* GSuite initial implementation of SAML fileset

* Document fields and generate test file

* Add documentation

* Split fields and improve docs

* Add change to CHANGELOG

* Rename config file and clean docs

* Adds user accounts fileset

* Add delegated user to google oauth

* Add types and make changes to common pipeline

* Do not stop input if array key not found

* Fix docs

* Setup for date cursor

* Add beta tag

* CHANGELOG message

* Improve ECS mappings

* Change cateogrization and types of various fields

* Change event.type to start

* Improve doc references

CHANGELOG.next.asciidoc

@@ -54,6 +54,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Adds oauth support for httpjson input. {issue}18415[18415] {pull}18892[18892]
 - Adds `split_events_by` option to httpjson input. {pull}19246[19246]
 - Adds `date_cursor` option to httpjson input. {pull}19483[19483]
+- Adds Gsuite module with SAML support. {pull}19329[19329]
+- Adds Gsuite User Accounts support. {pull}19329[19329]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -32,6 +32,7 @@ grouped in the following categories:
 * <<exported-fields-envoyproxy>>
 * <<exported-fields-fortinet>>
 * <<exported-fields-googlecloud>>
+* <<exported-fields-gsuite>>
 * <<exported-fields-haproxy>>
 * <<exported-fields-host-processor>>
 * <<exported-fields-ibmmq>>
@@ -23222,6 +23223,140 @@ type: keyword
 Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
 
 
+type: long
+
+--
+
+[[exported-fields-gsuite]]
+== gsuite fields
+
+gsuite Module
+
+
+
+[float]
+=== gsuite
+
+Gsuite specific fields.
+More information about specific fields can be found at https://developers.google.com/admin-sdk/reports/v1/reference/activities/list
+
+
+
+*`gsuite.actor.type`*::
++
+--
+The type of actor.
+Values can be:
+  *USER*: Another user in the same domain.
+  *EXTERNAL_USER*: A user outside the domain.
+  *KEY*: A non-human actor.
+
+
+type: keyword
+
+--
+
+*`gsuite.actor.key`*::
++
+--
+Only present when `actor.type` is `KEY`. Can be the `consumer_key` of the requestor for OAuth 2LO API requests or an identifier for robot accounts.
+
+
+type: keyword
+
+--
+
+*`gsuite.event.type`*::
++
+--
+The type of GSuite event, mapped from `items[].events[].type` in the original payload. Each fileset can have a different set of values for it, more details can be found at https://developers.google.com/admin-sdk/reports/v1/reference/activities/list
+
+
+type: keyword
+
+example: audit#activity
+
+--
+
+*`gsuite.kind`*::
++
+--
+The type of API resource, mapped from `kind` in the original payload. More details can be found at https://developers.google.com/admin-sdk/reports/v1/reference/activities/list
+
+
+type: keyword
+
+example: audit#activity
+
+--
+
+*`gsuite.organization.domain`*::
++
+--
+The domain that is affected by the report's event.
+
+
+type: keyword
+
+--
+
+
+*`gsuite.saml.application_name`*::
++
+--
+Saml SP application name.
+
+
+type: keyword
+
+--
+
+*`gsuite.saml.failure_type`*::
++
+--
+Login failure type. For a list of possible values refer to https://developers.google.com/admin-sdk/reports/v1/appendix/activity/saml.
+
+
+type: keyword
+
+--
+
+*`gsuite.saml.initiated_by`*::
++
+--
+Requester of SAML authentication.
+
+
+type: keyword
+
+--
+
+*`gsuite.saml.orgunit_path`*::
++
+--
+User orgunit.
+
+
+type: keyword
+
+--
+
+*`gsuite.saml.status_code`*::
++
+--
+SAML status code.
+
+
+type: long
+
+--
+
+*`gsuite.saml.second_level_status_code`*::
++
+--
+SAML second level status code.
+
+
 type: long
 
 --

filebeat/docs/modules/gsuite.asciidoc

@@ -0,0 +1,107 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-gsuite]]
+[role="xpack"]
+
+:modulename: gsuite
+:has-dashboards: false
+
+== GSuite module
+
+beta[]
+
+This is a module for ingesting data from the different GSuite audit reports API's.
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+It is compatible with a subset of applications under the https://developers.google.com/admin-sdk/reports/v1/get-start/getting-started[Google Reports API v1]. As of today it supports:
+
+- https://developers.google.com/admin-sdk/reports/v1/appendix/activity/saml[SAML Audit Activity Events]
+- https://developers.google.com/admin-sdk/reports/v1/appendix/activity/user-accounts[User Accounts Activity Events]
+
+=== Configure the module
+
+In order for filebeat to ingest data from the Google Reports API you must set up a `ServiceAccount` that has access to the `Admin SDK API`. Additionally https://developers.google.com/admin-sdk/reports/v1/guides/delegation[Domain-Wide Delegation] is required for your application to work properly.
+
+This module will make use of the following `oauth2 scope`:
+
+- `https://www.googleapis.com/auth/admin.reports.audit.readonly`
+
+Once you have downloaded your service account credentials as a JSON file,
+you can set up your module:
+
+[float]
+===== Configuration options
+
+[source,yaml]
+----
+- module: gsuite
+  saml:
+    enabled: true
+    var.jwt_file: "./credentials_file.json"
+    var.delegated_account: "user@example.com"
+  user_accounts:
+    enabled: true
+    var.jwt_file: "./credentials_file.json"
+    var.delegated_account: "user@example.com"
+----
+
+Every fileset has the following configuration options:
+
+*`var.jwt_file`*::
+
+Specifies the path to the JWT credentials file.
+
+*`var.delegated_account`*::
+
+Email of the admin user used to access the API.
+
+*`var.http_client_timeout`*::
+
+Duration of the time limit on HTTP requests made by the module. Defaults to
+`60s`.
+
+*`var.interval`*::
+
+Duration between requests to the API. Defaults to `60s`.
+
+*`var.user_key`*::
+
+Specifies the user key to fetch reports from. Defaults to `all`.
+
+[float]
+==== GSuite Reports ECS fields
+
+This is a list of GSuite Reports fields that are mapped to ECS.
+
+[options="header"]
+|=======================================================================================
+| GSuite Reports             | ECS Fields                                              |
+| items[].id.time            | @timestamp                                              |
+| items[].id.uniqueQualifier | event.id                                                |
+| items[].id.applicationName | event.provider                                          |
+| items[].events[].name      | event.action                                            |
+| items[].customerId         | organization.id                                         |
+| items[].ipAddress          | client.ip, related.ip, client.as.*, client.geo.*        |
+| items[].actor.email        | client.user.email, client.user.name, client.user.domain |
+| items[].actor.profileId    | client.user.id                                          |
+|=======================================================================================
+
+These are the common ones to all filesets.
+
+:has-dashboards!:
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-gsuite,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -17,6 +17,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-envoyproxy>>
   * <<filebeat-module-fortinet>>
   * <<filebeat-module-googlecloud>>
+  * <<filebeat-module-gsuite>>
   * <<filebeat-module-haproxy>>
   * <<filebeat-module-ibmmq>>
   * <<filebeat-module-icinga>>
@@ -63,6 +64,7 @@ include::modules/elasticsearch.asciidoc[]
 include::modules/envoyproxy.asciidoc[]
 include::modules/fortinet.asciidoc[]
 include::modules/googlecloud.asciidoc[]
+include::modules/gsuite.asciidoc[]
 include::modules/haproxy.asciidoc[]
 include::modules/ibmmq.asciidoc[]
 include::modules/icinga.asciidoc[]

x-pack/filebeat/filebeat.reference.yml

@@ -547,6 +547,14 @@ filebeat.modules:
     # the subscription.
     var.credentials_file: ${path.config}/gcp-service-account-xyz.json
 
+#-------------------------------- Gsuite Module --------------------------------
+- module: gsuite
+  # All logs
+  saml:
+    enabled: true
+  user_accounts:
+    enabled: true
+
 #------------------------------- HAProxy Module -------------------------------
 - module: haproxy
   # All logs

x-pack/filebeat/input/httpjson/config_oauth.go

@@ -50,9 +50,10 @@ type OAuth2 struct {
 	TokenURL       string              `config:"token_url"`
 
 	// google specific
-	GoogleCredentialsFile string `config:"google.credentials_file"`
-	GoogleCredentialsJSON []byte `config:"google.credentials_json"`
-	GoogleJWTFile         string `config:"google.jwt_file"`
+	GoogleCredentialsFile  string `config:"google.credentials_file"`
+	GoogleCredentialsJSON  []byte `config:"google.credentials_json"`
+	GoogleJWTFile          string `config:"google.jwt_file"`
+	GoogleDelegatedAccount string `config:"google.delegated_account"`
 
 	// microsoft azure specific
 	AzureTenantID string `config:"azure.tenant_id"`
@@ -79,6 +80,15 @@ func (o *OAuth2) Client(ctx context.Context, client *http.Client) (*http.Client,
 		}
 		return creds.Client(ctx), nil
 	case OAuth2ProviderGoogle:
+		if o.GoogleJWTFile != "" {
+			cfg, err := google.JWTConfigFromJSON(o.GoogleCredentialsJSON, o.Scopes...)
+			if err != nil {
+				return nil, fmt.Errorf("oauth2 client: error loading jwt credentials: %w", err)
+			}
+			cfg.Subject = o.GoogleDelegatedAccount
+			return cfg.Client(ctx), nil
+		}
+
 		creds, err := google.CredentialsFromJSON(ctx, o.GoogleCredentialsJSON, o.Scopes...)
 		if err != nil {
 			return nil, fmt.Errorf("oauth2 client: error loading credentials: %w", err)
@@ -149,6 +159,9 @@ func (o *OAuth2) validateGoogleProvider() error {
 
 	// credentials_json
 	if len(o.GoogleCredentialsJSON) > 0 {
+		if o.GoogleDelegatedAccount != "" {
+			return errors.New("invalid configuration: google.delegated_account can only be provided with a jwt_file")
+		}
 		if !json.Valid(o.GoogleCredentialsJSON) {
 			return errors.New("invalid configuration: google.credentials_json must be valid JSON")
 		}
@@ -157,6 +170,9 @@ func (o *OAuth2) validateGoogleProvider() error {
 
 	// credentials_file
 	if o.GoogleCredentialsFile != "" {
+		if o.GoogleDelegatedAccount != "" {
+			return errors.New("invalid configuration: google.delegated_account can only be provided with a jwt_file")
+		}
 		return o.populateCredentialsJSONFromFile(o.GoogleCredentialsFile)
 	}
 

x-pack/filebeat/input/httpjson/config_test.go

@@ -377,6 +377,29 @@ func TestConfigOauth2Validation(t *testing.T) {
 				"url":         "localhost",
 			},
 		},
+		{
+			name:        "google must fail if the delegated_account is set without jwt_file",
+			expectedErr: "invalid configuration: google.delegated_account can only be provided with a jwt_file accessing 'oauth2'",
+			input: map[string]interface{}{
+				"oauth2": map[string]interface{}{
+					"provider":                 "google",
+					"google.credentials_file":  "./testdata/credentials.json",
+					"google.delegated_account": "delegated@account.com",
+				},
+				"url": "localhost",
+			},
+		},
+		{
+			name: "google must work with delegated_account and a valid jwt_file",
+			input: map[string]interface{}{
+				"oauth2": map[string]interface{}{
+					"provider":                 "google",
+					"google.jwt_file":          "./testdata/credentials.json",
+					"google.delegated_account": "delegated@account.com",
+				},
+				"url": "localhost",
+			},
+		},
 	}
 
 	for _, c := range cases {

x-pack/filebeat/input/httpjson/input.go

@@ -398,6 +398,9 @@ func (in *HttpjsonInput) processHTTPRequest(ctx context.Context, client *http.Cl
 			} else {
 				v, err = common.MapStr(obj).GetValue(in.config.JSONObjects)
 				if err != nil {
+					if err == common.ErrKeyNotFound {
+						return nil
+					}
 					return err
 				}
 				switch ts := v.(type) {