Skip to content

Latest commit

 

History

History
341 lines (272 loc) · 18.9 KB

CHANGELOG.next.asciidoc

File metadata and controls

341 lines (272 loc) · 18.9 KB

Beats version HEAD

Breaking changes

Affecting all Beats

  • Update add_cloud_metadata fields to adjust to ECS. 9265

  • Automaticall cap signed integers to 63bits. 8991

  • Rename beat.timezone to event.timezone. 9458

  • Use _doc as document type. 9056https://github.com/elastic/beats/pull/9573[9573]

  • Update to Golang 1.11.3. 9560

  • Embedded html is not escaped anymore by default. 9914

  • Remove port settings from Logstash and Redis output. 9934

  • Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920

  • Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949

  • Import ECS change ecs#308: leaf field user.group is now the group field set. 10275

  • Update the code of Central Management to align with the new returned format. 10019

  • Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338

  • Remove --setup command line flag. 10138

  • Remove --version command line flag. 10138

  • Remove --configtest command line flag. 10138

  • Move output.elasticsearch.ilm settings to setup.ilm. 10347

  • ILM will be available by default if Elasticsearch > 7.0 is used. 10347

  • Allow Central Management to send events back to kibana. 9382

  • Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

Auditbeat

  • Rename process.exe to process.executable in auditd module to align with ECS. 9949

  • Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195

  • Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195

  • Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195

  • Field file.origin changed type from text to keyword. 10544

  • Rename user fields to ECS in auditd module. 10456

  • Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536

  • Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

Filebeat

  • Set ecs: true in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875

Heartbeat

  • Remove monitor generator script that was rarely used. 9648

  • monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697

  • A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme → url.scheme, monitor.host → url.domain, resolve.host → url.domain, http.url → url.full, tcp.port → url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.

  • The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

Journalbeat

  • Rename read_timestamp to event.created to align with ECS. 10043, 10139

  • Rename host.name to host.hostname to align with ECS. 10043

  • Fix typo in the field name container.id_truncated. 10525

  • Rename container.image.tag to container.log.tag. 10561

  • Change type of text fields to keyword. 10542

Metricbeat

  • Migrate system process metricset fields to ECS. 10332

  • Refactor Prometheus metric mappings 9948

  • Removed Prometheus stats metricset in favor of just using Prometheus collector 9948

  • Migrate system socket metricset fields to ECS. 10339

  • Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339

  • Adjust Redis.info metricset fields to ECS. 10319

  • Change type of field docker.container.ip_addresses to ip instead of keyword. 10364

  • Rename http.request.body field to http.request.body.content. 10315

  • Adjust php_fpm.process metricset fields to ECS. 10366

  • Adjust mongodb.status metricset to to ECS. 10368

  • Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322

  • Change the following fields from type text to keyword: 10318

  • ceph.osd_df.name

  • ceph.osd_tree.name

  • ceph.osd_tree.children

  • kafka.consumergroup.meta

  • kibana.stats.name

  • mongodb.metrics.replication.executor.network_interface

  • php_fpm.process.request_uri

  • php_fpm.process.script

  • Add service.name option to all modules to explicitly set service.name if it is unset. 10427

  • Update a few elasticsearch.* fields to map to ECS. 10350

  • Update a few logstash.* fields to map to ECS. 10350

  • Update a few kibana.* fields to map to ECS. 10350

  • Update rabbitmq.* fields to map to ECS. 10563

  • Update haproxy.* fields to map to ECS. 10558 10568

  • Collect all EC2 meta data from all instances in all states. 10628

  • Migrate docker module to ECS. 10927

Packetbeat

  • Adjust Packetbeat http fields to ECS Beta 2 9645

  • http.request.body moves to http.request.body.content

  • http.response.body moves to http.response.body.content

  • Changed Packetbeat fields to align with ECS. 7968

  • Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

  • Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

  • Mark Functionbeat as GA. 10564

  • Correctly normalize Cloudformation resource name. 10087

  • Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]

  • Allow functionbeat to use the keystore. 9009

Bugfixes

Affecting all Beats

  • Enforce validation for the Central Management access token. 9621

  • Fix config appender registration. 9873

  • Gracefully handle TLS options when enrolling a Beat. 9129

  • The backing off now implements jitter to better distribute the load. 10172

  • Fix TLS certificate DoS vulnerability. 10302

  • Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

  • Fix encoding of timestamps when using disk spool. 10099

  • Fix stopping of modules started by kubernetes autodiscover. 10476

  • Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

  • Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675

  • Ensure all beat commands respect configured settings. 10721

Auditbeat

  • Enable System module config on Windows. 10237

  • Package: Disable librpm signal handlers. 10694

  • Login: Handle different bad login UTMP types. 10865

Filebeat

  • Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

  • Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.

  • Support haproxy log lines without captured headers. 9463 9958

  • Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

  • Fix bad bytes count in docker input when filtering by stream. 10211

  • Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

  • Ensure source.address is always populated by the nginx module (ECS). 10418

  • Add support for Cisco syslog format used by their switch. 10760

  • Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/pull/10730[10730]

  • Fix registry entries not being cleaned due to race conditions. 10747

  • Improve detection of file deletion on Windows. 10747

Heartbeat

  • Made monitors.d configuration part of the default config. 9004

  • Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

Journalbeat

  • Do not stop collecting events when journal entries change. 9994

Metricbeat

  • Fix panics in vsphere module when certain values where not returned by the API. 9784

  • Fix pod UID metadata enrichment in Kubernetes module. 10081

  • Fix issue that would prevent collection of processes without command line on Windows. 10196

  • Fixed data type for tags field in docker/container metricset 10307

  • Fixed data type for tags field in docker/image metricset 10307

  • Fixed data type for isr field in kafka/partition metricset 10307

  • Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

  • Added function to close sql database connection. 10355

  • Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

  • Migrate docker autodiscover to ECS. 10757 10862

Packetbeat

  • Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

  • Fixed a crash when using af_packet capture 10477

  • Prevent duplicate packet loss error messages in HTTP events. 10709

  • Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

Functionbeat

  • Ensure that functionbeat is logging at info level not debug. 10262

  • Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

  • Update field definitions for http to ECS Beta 2 9645

  • Add agent.id and agent.ephemeral_id fields to all beats. 9404

  • Add name config option to add_host_metadata processor. 9943

  • Add add_labels and add_tags processors. 9973

  • Add missing file encoding to readers. 10080

  • Introduce migration.enabled configuration. 9805

  • Add alias field support in Kibana index pattern. 10075

  • Add add_fields processor. 10119

  • Add Kibana field formatter to bytes fields. 10184

  • Document a few more auditd.log.* fields. 10192

  • Support Kafka 2.1.0. 10440

  • Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347

  • Add support to read ILM policy from external JSON file. 10347

  • Add overwrite and check_exists settings to ILM support. 10347

  • Generate Kibana index pattern on demand instead of using a local file. 10478

  • Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]

  • Add network condition to processors for matching IP addresses against CIDRs. 10743

  • Add if/then/else support to processors. 10744

  • Add community_id processor for computing network flow hashes. 10745

Auditbeat

  • Add system module. 9546

  • Add user.id (UID) and user.name for ECS. 10195

  • Add group.id (GID) and group.name for ECS. 10195

  • System module process dataset: Add user information to processes. 9963

  • Add system package dataset. 10225

  • Add system module login dataset. 9327

  • Add entity_id fields. 10500

  • Add seven dashboards for the system module. 10511

  • Move System module to beta. 10800

Filebeat

  • Added module for parsing Google Santa logs. 9540

  • Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

  • Add option to modules.yml file to indicate that a module has been moved 9432.

  • Fix parsing of GC entries in elasticsearch server log. 9513 9810

  • Support mysql 5.7.22 slowlog starting with time information. 7892 9647

  • Add support for ssl_request_log in apache2 module. 8088 9833

  • Add support for iis 7.5 log format. 9753 9967

  • Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042

  • Add support for MariaDB in the slowlog fileset of mysql module. 9731

  • Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273

  • Elasticsearch module’s slowlog now populates event.duration (ECS). 9293

  • HAProxy module now populates event.duration and http.response.bytes (ECS). 10143

  • Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

  • Add convert_timezone to nginx module. 9839 10148

  • Add support for Percona in the slowlog fileset of mysql module. 6665 10227

  • Added support for ingesting structured Elasticsearch audit logs 10352

  • Added support for ingesting structured Elasticsearch slow logs 10445

  • Added support for ingesting structured Elasticsearch deprecation logs 10445

  • New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

  • Added support for ingesting structured Elasticsearch server logs 10428

  • Populate more ECS fields in the Suricata module. 10006

  • Add ISO8601 timestamp support in syslog metricset. 8716 10736

  • Add more info to message logged when a duplicated symlink file is found 10845

  • Add Netflow module to enrich flow events with geoip data. 10877

  • Set event.category: network_traffic for Suricata. 10882

Heartbeat

  • Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

  • Migrate registry from previously incorrect path. 10486

Metricbeat

  • Add key metricset to the Redis module. 9582 9657 9746

  • Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709

  • Add docker event metricset. 9856

  • Add 'performance' metricset to x-pack mssql module 9826

  • Add DeDot for kubernetes labels and annotations. 9860 9939

  • Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module 10011

  • Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074

  • Add AWS EC2 module. 9257 9300

  • Release windows Metricbeat module as GA. 10163

  • Release traefik Metricbeat module as GA. 10166

  • Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

  • List filesystems on Windows that have an access path but not an assigned letter 8916 10196

  • Add nats module. 10071

  • Release uswgi Metricbeat module GA. 10164

  • Release php_fpm module as GA. 10198

  • Release Memcached module as GA. 10199

  • Release etcd module as GA. 10200

  • Release Ceph module as GA. 10202

  • Release aerospike module as GA. 10203

  • Release kubernetes apiserver and event metricsets as GA 10212

  • Release Couchbase module as GA. 10201

  • Release RabbitMQ module GA. 10165

  • Release envoyproxy module GA. 10223

  • Release mongodb.metrics and mongodb.replstatus as GA. 10242

  • Release mysql.galera_status as GA. 10242

  • Release postgresql.statement as GA. 10242

  • Release RabbitMQ Metricbeat module GA. 10165

  • Release Dropwizard module as GA. 10240

  • Release Graphite module as GA. 10240

  • Release kvm module as beta. 10279

  • Release http.server metricset as GA. 10240

  • Release Nats module as GA. 10281

  • Release munin module as GA. 10311

  • Release Golang module as GA. 10312

  • Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

  • Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

  • Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module 10109

  • Add process arguments and the path to its executable file in the system process metricset 10332

  • Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

  • Release AWS module as GA. 10345

  • Add overview dashboard to Zookeeper Metricbeat module 10379

  • Add Consul Metricbeat module with Agent Metricset 8631

  • Add filters and pie chart for AWS EC2 dashboard. 10596

  • Add AWS SQS metricset. 10684 10053

Packetbeat

  • Add network.community_id to Packetbeat flow events. 10061

  • Add aliases for flow fields that were renamed. 7968 10063

Functionbeat

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

  • Close handle on signalEvent. 9838

Functionbeat

Known Issue

Journalbeat

  • Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).