Cherry-pick #22273 to 7.x: [Filebeat][New Module] Mysql Enterprise Au…

…dit log (#23021)

* [Filebeat][New Module] Mysql Enterprise Audit log (#22273)

* initial stash of mysql module

* initial commit for filebeat mysql module

* removing syslog from manifest currently

* changing module and field names to mysqlenterprise

* fixing some wrong types and adding related fields

* adding changelog entry

* adding some more process fields and event category/types

* fixing merge changelog

* adding changes based on PR feedback

(cherry picked from commit 03383be)

* regenerate docs

Co-authored-by: Marius Iversen <pillus@chasenet.org>

CHANGELOG.next.asciidoc

@@ -469,6 +469,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add max_number_of_messages config into s3 input. {pull}21993[21993]
 - Update Okta documentation for new stateful restarts. {pull}22091[22091]
 - Add SSL option to checkpoint module {pull}19560[19560]
+- Added support for MySQL Enterprise audit logs. {pull}22273[22273]
 - Rename googlecloud module to gcp module. {pull}22214[22214]
 - Rename awscloudwatch input to aws-cloudwatch. {pull}22228[22228]
 - Rename google-pubsub input to gcp-pubsub. {pull}22213[22213]

filebeat/docs/fields.asciidoc

@@ -59,6 +59,7 @@ grouped in the following categories:
 * <<exported-fields-mongodb>>
 * <<exported-fields-mssql>>
 * <<exported-fields-mysql>>
+* <<exported-fields-mysqlenterprise>>
 * <<exported-fields-nats>>
 * <<exported-fields-netflow>>
 * <<exported-fields-netscout>>
@@ -98260,6 +98261,257 @@ alias to: source.ip
 
 --
 
+[[exported-fields-mysqlenterprise]]
+== MySQL Enterprise fields
+
+MySQL Enterprise Audit module
+
+
+
+[float]
+=== mysqlenterprise
+
+Fields from MySQL Enterprise Logs
+
+
+
+[float]
+=== audit
+
+Module for parsing MySQL Enterprise Audit Logs
+
+
+
+*`mysqlenterprise.audit.class`*::
++
+--
+A string representing the event class. The class defines the type of event, when taken together with the event item that specifies the event subclass.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.connection_id`*::
++
+--
+An integer representing the client connection identifier. This is the same as the value returned by the CONNECTION_ID() function within the session.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.id`*::
++
+--
+An unsigned integer representing an event ID.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.connection_data.connection_type`*::
++
+--
+The security state of the connection to the server. Permitted values are tcp/ip (TCP/IP connection established without encryption), ssl (TCP/IP connection established with encryption), socket (Unix socket file connection), named_pipe (Windows named pipe connection), and shared_memory (Windows shared memory connection).
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.connection_data.status`*::
++
+--
+An integer representing the command status: 0 for success, nonzero if an error occurred.
+
+
+type: long
+
+--
+
+*`mysqlenterprise.audit.connection_data.db`*::
++
+--
+A string representing a database name. For connection_data, it is the default database. For table_access_data, it is the table database.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.connection_data.connection_attributes`*::
++
+--
+Connection attributes that might be passed by different MySQL Clients.
+
+
+type: flattened
+
+--
+
+*`mysqlenterprise.audit.general_data.command`*::
++
+--
+A string representing the type of instruction that generated the audit event, such as a command that the server received from a client.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.general_data.sql_command`*::
++
+--
+A string that indicates the SQL statement type.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.general_data.query`*::
++
+--
+A string representing the text of an SQL statement. The value can be empty. Long values may be truncated. The string, like the audit log file itself, is written using UTF-8 (up to 4 bytes per character), so the value may be the result of conversion.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.general_data.status`*::
++
+--
+An integer representing the command status: 0 for success, nonzero if an error occurred. This is the same as the value of the mysql_errno() C API function.
+
+
+type: long
+
+--
+
+*`mysqlenterprise.audit.login.user`*::
++
+--
+A string representing the information indicating how a client connected to the server.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.login.proxy`*::
++
+--
+A string representing the proxy user. The value is empty if user proxying is not in effect.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.shutdown_data.server_id`*::
++
+--
+An integer representing the server ID. This is the same as the value of the server_id system variable.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.startup_data.server_id`*::
++
+--
+An integer representing the server ID. This is the same as the value of the server_id system variable.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.startup_data.mysql_version`*::
++
+--
+An integer representing the server ID. This is the same as the value of the server_id system variable.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.table_access_data.db`*::
++
+--
+A string representing a database name. For connection_data, it is the default database. For table_access_data, it is the table database.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.table_access_data.table`*::
++
+--
+A string representing a table name.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.table_access_data.query`*::
++
+--
+A string representing the text of an SQL statement. The value can be empty. Long values may be truncated. The string, like the audit log file itself, is written using UTF-8 (up to 4 bytes per character), so the value may be the result of conversion.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.table_access_data.sql_command`*::
++
+--
+A string that indicates the SQL statement type.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.account.user`*::
++
+--
+A string representing the user that the server authenticated the client as. This is the user name that the server uses for privilege checking.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.account.host`*::
++
+--
+A string representing the client host name.
+
+
+type: keyword
+
+--
+
+*`mysqlenterprise.audit.login.os`*::
++
+--
+A string representing the external user name used during the authentication process, as set by the plugin used to authenticate the client.
+
+
+type: keyword
+
+--
+
 [[exported-fields-nats]]
 == NATS fields
 

filebeat/docs/modules/mysqlenterprise.asciidoc

@@ -0,0 +1,81 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-mysqlenterprise]]
+[role="xpack"]
+
+:modulename: mysqlenterprise
+:has-dashboards: false
+
+
+== MySQL Enterprise module
+beta[]
+
+This is a module for different types of MySQL logs. Currently focusing on data from the MySQL Enterprise Audit Plugin in JSON format.
+
+To configure the the Enterprise Audit Plugin to output in JSON format please follow the directions in the https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html[MySQL Documentation.]
+
+
+include::../include/gs-link.asciidoc[]
+
+
+[float]
+=== Compatibility
+
+This module has been tested against MySQL Enterprise 5.7.x and 8.0.x
+
+include::../include/configuring-intro.asciidoc[]
+
+:fileset_ex: audit
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `audit` fileset settings
+
+Example config:
+
+[source,yaml]
+----
+- module: mysqlenterprise
+  audit:
+    var.input: file
+    var.paths: /home/user/mysqlauditlogs/audit.*.log
+----
+
+include::../include/var-paths.asciidoc[]
+
+*`var.tags`*::
+
+A list of tags to include in events. Including `forwarded` indicates that the
+events did not originate on this host and causes `host.name` to not be added to
+events. Defaults to `[mysqlenterprise-audit]`.
+
+[float]
+==== MySQL Enterprise ECS Fields
+
+MySQL Enterprise Audit fields are mapped to ECS in the following way:
+
+[options="header"]
+|==============================================================
+| MySQL Enterprise Fields    | ECS Fields                     |
+| account.user               | server.user.name               |
+| account.host               | client.domain                  |
+| login.os                   | client.user.name               |
+| login.ip                   | client.ip                      |
+| startup_data.os_version    | host.os.full                   |
+| startup_data.args          | process.args                   |
+| connection_attributes._pid | process.pid                    |
+| timestamp                  | @timestamp                     |
+|==============================================================
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-mysqlenterprise,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -40,6 +40,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-mongodb>>
   * <<filebeat-module-mssql>>
   * <<filebeat-module-mysql>>
+  * <<filebeat-module-mysqlenterprise>>
   * <<filebeat-module-nats>>
   * <<filebeat-module-netflow>>
   * <<filebeat-module-netscout>>
@@ -109,6 +110,7 @@ include::modules/misp.asciidoc[]
 include::modules/mongodb.asciidoc[]
 include::modules/mssql.asciidoc[]
 include::modules/mysql.asciidoc[]
+include::modules/mysqlenterprise.asciidoc[]
 include::modules/nats.asciidoc[]
 include::modules/netflow.asciidoc[]
 include::modules/netscout.asciidoc[]

x-pack/filebeat/filebeat.reference.yml

@@ -1340,6 +1340,19 @@ filebeat.modules:
     # can be added under this section.
     #input:
 
+#--------------------------- MySQL Enterprise Module ---------------------------
+- module: mysqlenterprise
+  audit:
+    enabled: true
+
+    # Sets the input type. Currently only supports file
+    #var.input: file
+
+    # Set paths for the log files when file input is used.
+    # Should only be used together with file input
+    # var.paths:
+    #  - /home/user/mysqlauditlogs/audit.*.log
+
 #--------------------------------- NATS Module ---------------------------------
 - module: nats
   # All logs