implement logging levels

[beckettsean]

Feature Request

Proposal: [Description of the feature]

Add typical levels to the InfluxDB logging package. E.g. DEBUG, INFO, WARN, ERROR.

Current behavior: [What currently happens]

Logging for individual subsystems [meta, wal, query, data, http, and continuous queries] can be enabled or disabled, and DEBUG-style write tracing can be turned on and off explicitly, but none of the logs offer varying levels of logging. Each is all or nothing.

Desired behavior: [What you would like to happen]

We should categorize the various log outputs into three or four of the above standard logging levels. Ideally with each subsystem having its own log level. E.g. I can turn on DEBUG for query without incurring DEBUG for meta, http, etc.

Use case: [Why is this important (helps with prioritizing requests)]

Monitoring of production systems is a balancing act. Too many logs fills disk and obscures signal. Too few logs squelches signal and makes analysis difficult. No solution works for every implementation, so there should be flexibility in the system.

The most compelling use case for InfluxData is so that the InfluxCloud product can have reasonable log levels and not drive up log analysis costs. This is a production problem that affects anyone using a commercial log monitoring solution against InfluxDB.

[beckettsean]

@gunnaraasen @nhaugo

[jsternberg]

I was doing work on refactoring logging in js-refactor-logging. I think during a conversation about refactoring logging, there was a lot of resistance to logging based on level, but there was a favorable reaction to refactoring logging to be more event based and supporting multiple output types (such as JSON for things like Splunk ingestion).

The reason why filtering on logging levels had resistance was because it is generally very hard for a developer to determine what level a log message should be. It is also because debug messages can generally slow down the system and just create logging noise most of the time with information overload, so there was general resistance to the debug level.

There wasn't as much resistance to info, warn, and error levels. Just debug level I think.

@joelegasse @benbjohnson

[beckettsean]

DEBUG almost always causes performance issues, but it is very nice to have in a forensics situation, as it allows support to receive lots of data in a controlled setting. Even if its only used in testing environments, DEBUG can be quite handy. I don't really understand the objection to providing it. We already allow users to log every write statement, which is hugely punishing on the OS and the filesystem. No one does it unless they need to, but when we need to see what the system thinks it's receiving, there's not really a better way to investigate.

As for classification of messages, I don't see that as an individual developer's problem. I think support and cloud teams would be happy to classify the logging messages into their appropriate levels, and we will evolve it over time as the userbase and the system mature.

[jsternberg]

Closing this in favor of #7036.