feat: Add support for JSON logging to stderr

[bpcreech]

Fixes #1026

[conventional-commit-lint-gcf]

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

[losalex]

@bpcreech , thanks for opening this PR! I have a conceptual feedback on this change - basically when you configure stdout/stderr to be used, it will redirect all logging traffic to stdout/stderr respectively. While developer always can create 2 loggers to be able to use both, stdout and stderr streams, it would be best to make this feature dependent on Severity. This way we can define more flexible logic to send log record to dedicated output streams based if the record has error-related severity or not (e.g. DEFAULT/DEBUG/INFO/NOTICE severities will go to stdout and the rest to stderr).
Thats said, I believe that Severity based output could be introduced as separate feature in addition to ability to redirect everything to stdout or stderr - please let me know WDYT about this and we can take it from there.

[nickkieffer]

Hey @losalex and @bpcreech, I think this change made it so the default python logging package no longer logs to gcloud without using a structured log to STOUT
I think it was logging to STERR by default previously (yes malformed but it still worked)
This was before the release as an example

Now we just get no logs, working to add these to STOUT and json structured but figured others will run into this

[bpcreech]

Hmm, this change was to the Java logging library, which is not used by any Python SDK I'm aware of.

[nickkieffer]

I was thinking it may consume logs that are formatted via the python SDK, nothing directly related if that makes sense

[bpcreech]

Can you describe your software stack? What environment are you running on, and what SDKs are you using?

[nickkieffer]

We run python scripts using kubernetes in gcloud,
in python just the basic

import logging
log = logging.getLogger(__name__)
log.debug('debuggin')

This is pretty confusing to me though, google stackdriver doesn't have our logs but
when using the kubernetes python SDK we actually DO get logs back so we know we are logging something still 😰

from kubernetes.client import BatchV1Api, LogsApi, CoreV1Api, EventsV1beta1Api
api = api_mgr.get(CoreV1Api)
pod = api.read_namespaced_pod(pod_name, self.namespace)
logs = api.read_namespaced_pod_log(
     pod_name, self.namespace, follow=True, _preload_content=False
 )
for line in logs.stream():
            yield Event(timestamp=0, type=TYPE_POD_LOG, name=pod_name, message=line.decode('utf-8'))

[nickkieffer]

On the left is the google console, the right is our UI displaying the logs from that kubernetes SDK

[nickkieffer]

it stands out to me that this is an if else and is replaced with a switch without a default case, but 🤷

[losalex]

@nickkieffer , are you using redirectToStdout feature?

[nickkieffer]

no, not directly anyways

[losalex]

@nickkieffer , I must admit that I have some confusion while following the issue you raised, but let me try answering best based on my understanding:

1. Correct me if I wrong, you use a Python logger - based on sample code you shared. This change to print to STDERR was performed only in Java library and I believe we do not have such feature in Python. If problem indeed related to Python SDK, can you please raise the issue in Python repo? Feel free to provide all details you mentioned here and also the details about environment you run in - adding @daniel-sanche for visibility.
2. This change supposed to use CLOUD_LOGGING by default - the STDOUT and STDERR logging needs to be explicitly enabled in code or configuration so the change should be fully backward compatible to clients which does not use the feature. If you believe this is not a case, please let us know - me and @bpcreech can take a look.

Please let me know if I understood you correctly and if I can assist you any further.

[nickkieffer]

You can disregard me on this, I think my ops team just so happened to turn off of logging agents around the same time as your release happened. Sorry about the distraction!