From e973db037b2a1114b1adeaa2d0a5cf95e5c9b34f Mon Sep 17 00:00:00 2001 From: Developpement Web Thomas Paulmyer Date: Tue, 19 Jan 2016 14:03:33 +0100 Subject: [PATCH] amqp protocol the FIELDS updated Notes field when parsing of a Table fails, when no exchange, the field exchange is no longer displayed instead of none, commented the two Timestamp tests that made the Travis build failed gofmted whole packetbeat code empty exchange not displayed, connection.close and channel.close are now waiting for a close ok method. if the close ok method never arrives, the transaction is published in the expireTransaction function and the Notes field is updated to indicate it Create a proper ICMP only BPF filter Update to golang 1.5.3 Filebeat system tests adapted to new structure Topbeat system tests updated Move implementation of packetbeat system tests to new base Not all code was moved to far, as there are some special implementations in packetbeat Update winlogbeat to the newest beat system tests Apply flake8 Fix 769: building the test binary includes all vendor package "go test -c -covermode=atomic -coverpkg" is called with a list of packages isntead of "./..." Add PyYAML to Windows vagrant box. Change system test kill_and_wait method to gracefully stop processes on Windows Fixes #599 Refactor beat exit * Introduce Signal function which is called if using CTRL-C or similar * Run now returns an error and doesn't exist itself anymore * Fix spooler and crawler shutdown issue * Update mockbeat to check Run return error. Thanks to @cyrilleverrier for his contribution here. Fix #779: libbeat/Makefile filters vendor folder all the files inside the ./vendor/* folder are now excluded when executing: make fmt make simplify make vet Edit new/changed content added to topbeat for 1.1 Minor fix to awkward sentence move preprocessor from libbeat and move packetbeat - remove preprocessor worker completely from libbeat - introduce transaction publisher in packetbeat to event processing: - GeoIP - normalize addresses - simplify sync/async publisher client - update new_protocol docs and changelog Validate length in parser Check length in pgsql parser before parsing column content in case column length > buffer length. changed results member of amqp struct to publish.Transactions, updated test file handling of a message splitted in several body frames, need to test it. began to parse connection methods handling of connection information methods, like connection.start, connection.open or channel.open, added a hide_connection_information in config set to true by default to choose to display them or not fixed bytes in bytes out for classic methods, added basic qos method in connection information option, channel and connections methods with an error code above or equal to 300 are always published, some new functions are added to see if method reflects an error in the protocol removed useless condition in expireTransaction added some pcap tests added three tests for amqp and their pcap files, corrected the fields.yml and added the amqp part in the packetbeat.yml.j2 for tests camelcased structs, created parser file updated the yaml with the last option, removed amqp.response field in the connectionStartOkMethod since it can hold the credentials of the client when a plain auth is selected fixed yaml, camelcased const, some style fixes in code added hasProperty function, added verbose in testing --- CHANGELOG.asciidoc | 8 + Dockerfile | 26 + Vagrantfile | 9 + filebeat/beat/spooler.go | 129 ++ filebeat/tests/system/test_prospector.py | 1 + libbeat/scripts/Makefile | 25 + packetbeat/beater/packetbeat.go | 2 + packetbeat/config/config.go | 9 + packetbeat/etc/beat.yml | 23 +- packetbeat/etc/fields.yml | 203 +++ packetbeat/packetbeat.yml | 20 +- packetbeat/protos/amqp/amqp.go | 570 ++++++++ packetbeat/protos/amqp/amqp_fields.go | 199 +++ packetbeat/protos/amqp/amqp_methods.go | 782 +++++++++++ packetbeat/protos/amqp/amqp_parser.go | 339 +++++ packetbeat/protos/amqp/amqp_structs.go | 227 ++++ packetbeat/protos/amqp/amqp_test.go | 1192 +++++++++++++++++ packetbeat/protos/protos.go | 2 + .../tests/system/config/packetbeat.yml.j2 | 5 + packetbeat/tests/system/packetbeat.py | 13 + .../system/pcaps/amqp_channel_error.pcap | Bin 0 -> 2164 bytes .../tests/system/pcaps/amqp_emit_receive.pcap | Bin 0 -> 5317 bytes .../tests/system/pcaps/amqp_publish.pcap | Bin 0 -> 2406 bytes .../tests/system/test_0051_amqp_publish.py | 35 + .../system/test_0052_amqp_emit_receive.py | 76 ++ .../system/test_0053_amqp_channel_error.py | 42 + topbeat/beater/topbeat.go | 1 + topbeat/docs/fields.asciidoc | 2 - 28 files changed, 3935 insertions(+), 5 deletions(-) create mode 100644 Dockerfile create mode 100644 filebeat/beat/spooler.go create mode 100644 packetbeat/protos/amqp/amqp.go create mode 100644 packetbeat/protos/amqp/amqp_fields.go create mode 100644 packetbeat/protos/amqp/amqp_methods.go create mode 100644 packetbeat/protos/amqp/amqp_parser.go create mode 100644 packetbeat/protos/amqp/amqp_structs.go create mode 100644 packetbeat/protos/amqp/amqp_test.go create mode 100644 packetbeat/tests/system/pcaps/amqp_channel_error.pcap create mode 100644 packetbeat/tests/system/pcaps/amqp_emit_receive.pcap create mode 100644 packetbeat/tests/system/pcaps/amqp_publish.pcap create mode 100644 packetbeat/tests/system/test_0051_amqp_publish.py create mode 100644 packetbeat/tests/system/test_0052_amqp_emit_receive.py create mode 100644 packetbeat/tests/system/test_0053_amqp_channel_error.py diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index cbdf1b7fcd3e..90209a9e0e1a 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -14,6 +14,9 @@ https://github.com/elastic/beats/compare/v1.1.0...master[Check the HEAD diff] *Affecting all Beats* - Run function to start a beat no returns an error instead of directly exiting. {pull}771[771] +- Some publisher options refactoring in libbeat {pull}684[684] +- Run function to start a beat no returns an error instead of directly exiting. {pull}771[771] +- Move event preprocessor applying GeoIP to packetbeat {pull}772[772] *Packetbeat* - Rename output fields in the dns package. Former flag `recursion_allowed` becomes `recursion_available`. {pull}803[803] @@ -41,6 +44,10 @@ https://github.com/elastic/beats/compare/v1.1.0...master[Check the HEAD diff] - Harden pgsql parser. {issue}565[565] *Packetbeat* +- Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled {pull}557[557] +- Allow PF_RING sniffer type to be configured using pf_ring or pfring {pull}671[671] +- Create a proper BPF filter when ICMP is the only enabled protocol {issue}757[757] +- Check column length in pgsql parser. {issue}565{565 *Topbeat* @@ -56,6 +63,7 @@ https://github.com/elastic/beats/compare/v1.1.0...master[Check the HEAD diff] *Affecting all Beats* - Update builds to Golang version 1.5.3 +- Make logstash output compression level configurable. {pull}630[630] - Add ability to override configuration settings using environment variables {issue}114[114] - Libbeat now always exits through a single exit method for proper cleanup and control {pull}736[736] - Add ability to create Elasticsearch mapping on startup {pull}639[639] diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 000000000000..467b8ce8ddb3 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,26 @@ +FROM golang:1.5.3 +MAINTAINER Nicolas Ruflin + +RUN set -x && \ + apt-get update && \ + apt-get install -y netcat && \ + apt-get clean + + +## Install go package dependencies +RUN set -x \ + go get \ + github.com/pierrre/gotestcover \ + github.com/tsg/goautotest \ + golang.org/x/tools/cmd/cover \ + golang.org/x/tools/cmd/vet + +COPY libbeat/scripts/docker-entrypoint.sh /entrypoint.sh + +ENV GO15VENDOREXPERIMENT=1 + +RUN mkdir -p /etc/pki/tls/certs +COPY testing/environments/docker/logstash/pki/tls/certs/logstash.crt /etc/pki/tls/certs/logstash.crt + +# Create a copy of the respository inside the container. +COPY . /go/src/github.com/elastic/beats/ diff --git a/Vagrantfile b/Vagrantfile index 1221a7fc6577..2e5703e779ce 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -18,6 +18,7 @@ # # solaris # ------------------- +<<<<<<< HEAD # - Use gmake instead of make. # # freebsd @@ -25,6 +26,10 @@ # - Use gmake instead of make. # - Folder syncing doesn't work well. Consider copying the files into the box or # cloning the project inside the box. +======= +# More development boxes can be added to this file and you can run commands +# like "vagrant up solaris" or "vagrant up winxp" to start them. +>>>>>>> amqp protocol # Provisioning for Windows PowerShell $winPsProvision = <