Add secrets sanitization helpers

[florimondmanca]

What does this PR do?

Add helpers to sanitize any known secrets (i.e. those manipulated by an integration) from logs, service check messages, and exception tracebacks.

Motivation

Make it easier to ensure passwords aren't leaked in logs and UI messages.

Of limited use for credentials contained in URLs (as I believe the Agent already scrubs those?), but most useful for general logs that might leak secrets, whether those leaks are known:

integrations-core/sqlserver/datadog_checks/sqlserver/sqlserver.py

Lines 663 to 667 in 42ee1d3

	password = instance.get('password')
	if password is not None:
	message = message.replace(password, "*" * 6)
	self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=message)

or unknown/possible/unproven: #5715 (comment)

In general we should register any password/secret managed by an integration.

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #6107 into master will decrease coverage by 0.96%.
The diff coverage is 89.18%.

Impacted Files	Coverage Δ
datadog_checks_base/datadog_checks/base/log.py	84.05% <70.00%> (-6.14%)	⬇️
datadog_checks_base/tests/test_agent_check.py	98.34% <88.37%> (-1.14%)	⬇️
...dog_checks_base/datadog_checks/base/checks/base.py	84.93% <93.75%> (+0.36%)	⬆️
...g_checks_base/datadog_checks/base/utils/secrets.py	100.00% <100.00%> (ø)
datadog_checks_base/tests/test_utils.py	100.00% <100.00%> (ø)
couch/tests/test_couchv2.py	96.75% <0.00%> (-1.18%)	⬇️
pgbouncer/datadog_checks/pgbouncer/pgbouncer.py	76.22% <0.00%> (-0.39%)	⬇️
statsd/datadog_checks/statsd/statsd.py	81.81% <0.00%> (-0.33%)	⬇️
directory/datadog_checks/directory/directory.py	88.57% <0.00%> (-0.32%)	⬇️
php_fpm/datadog_checks/php_fpm/php_fpm.py	80.34% <0.00%> (-0.17%)	⬇️
... and 299 more

[ofek]

This still would not protect against exposure in chained tracebacks, maybe worth doing as a more general filtering function, used by logger and in run()?

[mgarabed]

This is great - will be very helpful!

There are some other escape vectors though, a good use case can be seen in #6003. That case requires sanitizing the message prior to service_check and an exception message.

One approach would be to have the sanitize filter as a method on the AgentCheck class, then checks could call it directly.

[florimondmanca]

@ofek @mgarabed Thanks, updated the PR to:

• Minimize the impact in the common case (down to an attribute lookup)
• Add registration/sanitization methods on AgentCheck directly.
• Apply scrubbing to exception messages/tracebacks too.

LMK what you think!

[ofek]

Looks great

[mgarabed]

💯

Stale

[ofek]

Diff is reasonably small, no need to break this up

Great job!

[florimondmanca]

Thanks all!

CI failures are unrelated:

• ClickHouse fails due to new metrics
• Gunicorn fails due to style checks bug - filed Fix style checks failure #6134 to resolve

Merging…