filter: postgres statistics network filter #10642

cpakulski · 2020-04-03T18:19:04Z

Description:
The filter implements decoding postgres wire protocol and parses messages exchanged between postgres server and client. Based on the decoded messages the filter generates statistics (counters) indicating how many messages of a specific type were exchanged.

Risk Level:
Low: The filter is implemented as extension and the code is not executed unless inserted into filter chain.

Testing:
Added unit and integration tests.

Docs Changes:
Yes - added architecture overview chapter and configuration specific sections

Release Notes:
Yes

Partially fixes: #9107

repokitteh-read-only · 2020-04-03T18:19:09Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/.

🐱

Caused by: #10642 was opened by cpakulski.

see: more, trace.

lizan · 2020-04-03T19:41:36Z

@cpakulski can you merge master to resolve conflicts?

Signed-off-by: Dhi Aurrahman <dio@tetrate.io>

…ce code. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

… messages spread over more than one buffer. Added unit tests for parsing single, multiple and large messages. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added hash map to find the message type and related actions based on the first byte of the message. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

successful authentication. Other R messages are ignored. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Formatted code with format and spell checkers. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

configuration. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

INFO: From ProtoGenValidateCcGenerate external/envoy_api/envoy/config/filter/network/postgresql_proxy/v2alpha/postgresql_proxy.pb.h: envoy/config/filter/network/postgresql_proxy/v2alpha/postgresql_proxy.proto: warning: Import udpa/annotations/migrate.proto but not used. envoy/config/filter/network/postgresql_proxy/v2alpha/postgresql_proxy.proto: warning: Import validate/validate.proto but not used. envoy/config/filter/network/postgresql_proxy/v2alpha/postgresql_proxy.proto: warning: Import udpa/annotations/status.proto but not used. INFO: From ProtoGenValidateCcGenerate external/envoy_api/envoy/extensions/filters/network/postgresql_proxy/v3alpha/postgresql_proxy.pb.h: envoy/extensions/filters/network/postgresql_proxy/v3alpha/postgresql_proxy.proto: warning: Import udpa/annotations/versioning.proto but not used. envoy/extensions/filters/network/postgresql_proxy/v3alpha/postgresql_proxy.proto: warning: Import validate/validate.proto but not used. envoy/extensions/filters/network/postgresql_proxy/v3alpha/postgresql_proxy.proto: warning: Import udpa/annotations/status.proto but not used. Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

This reverts commit 83bb9b4. Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

…d messages. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Moved logic from multi-level if..else to hash map. Updated documention. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

- sessions: Number of successful logins - sessions_encrypted: Number of times the proxy detected encrypted sessions - sessions_unencrypted: Number of messages indicating unencrypted successful login Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

- errors: Total number of backend decoded errors (ErrorResponse) - errors_error: Number of ERROR severity errors detected by the filter - errors_fatal: Number of FATAL severity errors detected by the filter - errors_panic: Number of PANIC severity errors detected by the filter - errors_unknown: Number of severity errors that filter can't detected Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

to use enums, not separate methods. Error and Notice parser searches for S and V prefixes. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

for keywords with S and V prefixes. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

- file names - directory names - variables and class names - config items - documentation Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

source/extensions/filters/network/postgres_proxy/postgres_decoder.h

lizan

Thanks, structurally looks good. Mostly C++ style/readability nits.

source/extensions/filters/network/postgres_proxy/postgres_decoder.h

test/extensions/filters/network/postgres_proxy/postgres_filter_test.cc

source/extensions/filters/network/postgres_proxy/postgres_decoder.cc

source/extensions/filters/network/postgres_proxy/postgres_decoder.h

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Use writeBEInt templates to write integers in network format to buffer. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

… into postgresql-cp Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

htuch · 2020-04-19T22:24:54Z

/lgtm api

lizan

LGTM, can you merge master to resolve conflict and fix CI?

fabriziomello · 2020-04-21T00:42:00Z

@lizan now all CI tests are ok.

API review addressed

Description: The filter implements decoding postgres wire protocol and parses messages exchanged between postgres server and client. Based on the decoded messages the filter generates statistics (counters) indicating how many messages of a specific type were exchanged. envoyproxy#9107 Risk Level: Low: The filter is implemented as extension and the code is not executed unless inserted into filter chain. Testing: Added unit and integration tests. Docs Changes: Yes - added architecture overview chapter and configuration specific sections Release Notes: Yes Signed-off-by: Christoph Pakulski <christoph@tetrate.io> Co-authored-by: Dhi Aurrahman <dio@tetrate.io> Co-authored-by: Fabrízio de Royes Mello <fabrizio@ongres.com> Signed-off-by: pengg <pengg@google.com>

tmc · 2020-04-22T23:26:36Z

Typo: sesions

repokitteh-read-only bot added the api label Apr 3, 2020

dio and others added 27 commits April 3, 2020 17:27

Initial structure

305174c

Signed-off-by: Dhi Aurrahman <dio@tetrate.io>

Moved postgresql filter code from internal repository to envoy's sour…

a2da1a5

…ce code. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Refactored decoder to handle multiple messages in a single buffer and…

5122145

… messages spread over more than one buffer. Added unit tests for parsing single, multiple and large messages. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Integrated decoded into filter. Added filter's unit tests.

2c2fab8

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Postgresql: linked frontend statistics to message processing.

6b55021

Added hash map to find the message type and related actions based on the first byte of the message. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added some SQL scripts as example of stats increment

3804105

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Separated processing of frontend and backend messages.

cf4ca28

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added processing backend messages and linked them to statistics.

d8709ec

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added deep inspectiob for R (Authentication) message to detect

a91537b

successful authentication. Other R messages are ignored. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added no-op handlers for Z and S postgres messages.

7a78b27

Formatted code with format and spell checkers. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Run proto format checker.

ced12e8

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added skeleton for integration tests.

13f64cf

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Updated documentation. Added statistics and linked to v2 and V3

a33d68b

configuration. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Revert "Silence compiler warnings:"

c510077

This reverts commit 83bb9b4. Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Added all Backend/Frontend known commands and some minor code cleanup.

045eb7b

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Code cleanup. Added unit tests for all postgresql frontend and backen…

2fc1c64

…d messages. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Refactored parsing CommandCompleted message.

226e2db

Moved logic from multi-level if..else to hash map. Updated documention. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Code cleanup. Removed postgres_utils.[cc,h] files.

f6a53b7

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added logic to detect encrypted sessions.

3d73fd8

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Added parsing routines and unit tests for NOTICE messages.

119774b

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Refactored methods incrementing statistics for Error and Notice messages

34a687d

to use enums, not separate methods. Error and Notice parser searches for S and V prefixes. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Changed logic of parsing Error and Notice messages when searching

0c4dbc7

for keywords with S and V prefixes. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Renamed directories to use postgres instead of postgresql.

9501890

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Renamed all instances of postgresql to postgres:

e9bf2b3

- file names - directory names - variables and class names - config items - documentation Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

cpakulski and others added 4 commits April 10, 2020 23:36

Updated example configs to use v3.

902c36a

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Merge branch 'master' into postgresql-cp

1aef2b3

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Merge branch 'master' into postgresql-cp

b325412

Merge branch 'master' into postgresql-cp

07277f3

Shikugawa reviewed Apr 15, 2020

View reviewed changes

source/extensions/filters/network/postgres_proxy/postgres_decoder.h Outdated Show resolved Hide resolved

lizan reviewed Apr 15, 2020

View reviewed changes

fabriziomello and others added 3 commits April 16, 2020 20:44

Fix typo and official Postgres website link

1a28549

Signed-off-by: Fabrízio de Royes Mello <fabrizio@ongres.com>

Refectored code to use structs instead of tuples.

d0e84d0

Use writeBEInt templates to write integers in network format to buffer. Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Merge branch 'postgresql-cp' of https://github.com/envoy-postgres/envoy…

97a7b3f

… into postgresql-cp Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

cpakulski requested review from lizan and htuch April 17, 2020 13:27

mattklein123 assigned dio Apr 18, 2020

repokitteh-read-only bot removed the api label Apr 19, 2020

lizan previously approved these changes Apr 20, 2020

View reviewed changes

Merge branch 'master' into postgresql-cp

aea61e2

fabriziomello dismissed lizan’s stale review via aea61e2 April 20, 2020 18:24

repokitteh-read-only bot added the api label Apr 20, 2020

lizan approved these changes Apr 21, 2020

View reviewed changes

repokitteh-read-only bot removed the api label Apr 21, 2020

lizan merged commit f599ad7 into envoyproxy:master Apr 21, 2020

mattklein123 mentioned this pull request Apr 22, 2020

PostgreSQL-aware filter #2861

Closed

rgs1 mentioned this pull request Apr 22, 2020

Crash regression from #10362 #10901

Closed

fabriziomello mentioned this pull request Apr 24, 2020

Fix typo on Postgres Proxy documentation. #10930

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter: postgres statistics network filter #10642

filter: postgres statistics network filter #10642

cpakulski commented Apr 3, 2020 •

edited

Loading

repokitteh-read-only bot commented Apr 3, 2020

lizan commented Apr 3, 2020

lizan left a comment

htuch commented Apr 19, 2020

lizan left a comment

fabriziomello commented Apr 21, 2020

tmc commented Apr 22, 2020

filter: postgres statistics network filter #10642

filter: postgres statistics network filter #10642

Conversation

cpakulski commented Apr 3, 2020 • edited Loading

repokitteh-read-only bot commented Apr 3, 2020

lizan commented Apr 3, 2020

lizan left a comment

Choose a reason for hiding this comment

htuch commented Apr 19, 2020

lizan left a comment

Choose a reason for hiding this comment

fabriziomello commented Apr 21, 2020

tmc commented Apr 22, 2020

cpakulski commented Apr 3, 2020 •

edited

Loading