Add GAE and GKE fluentd Handlers #2668

waprin · 2016-11-03T00:22:27Z

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations.

This adds two new handlers and some helper functions to detect when they should be used.

waprin · 2016-11-03T00:23:30Z

@dhermes /cc @jonparrott

daspecster

Pretty awesome stuff!

docs/logging-handlers-GAEHandler.rst

@@ -0,0 +1,6 @@
+Google App Engine Flexible Log Handler
+=======================================


docs/logging-handlers-GKEHandler.rst

@@ -0,0 +1,6 @@
+Google Container Engine Log Handler
+=======================================


docs/logging-usage.rst

-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,
+use :meth:``google.cloud.logging.client.get_default_handler`.


docs/logging-usage.rst

    >>> cloud_logger = logging.getLogger('cloudLogger')
    >>> cloud_logger.setLevel(logging.INFO) # defaults to WARN
    >>> cloud_logger.addHandler(handler)
    >>> cloud_logger.error('bad news')

+It is also possible to attach the handler to the root Python logger, so that for example a plain
+`logging.warn` call would be sent to Cloud Logging, as well as any other loggers created. However,


docs/logging-usage.rst

-method :meth:`setup_logging <google.cloud.logging.handlers.setup_logging>` is provided to configure
-this automatically:
+CloudLoggingHandler
+=======================


logging/google/cloud/logging/handlers/__init__.py

@@ -15,4 +15,6 @@
 """Python :mod:`logging` handlers for Google Cloud Logging."""

 from google.cloud.logging.handlers.handlers import CloudLoggingHandler
+from google.cloud.logging.handlers.GAEHandler import GAEHandler


logging/google/cloud/logging/handlers/handlers.py

-    :param excluded_loggers: The loggers to not attach the handler to. This
-                             will always include the loggers in the path of
-                             the logging client itself.
+    :param excluded_loggers: (Optional)The loggers to not attach the handler


logging/google/cloud/logging/handlers/handlers.py

+                             to. This will always include the loggers in the
+                             path of the logging client itself.
+
+    :type log_level: :mod:`logging` log level


logging/unit_tests/handlers/transports/test_background_thread.py

@@ -184,7 +184,7 @@ def batch(self):

 class _Client(object):

-    def __init__(self, project):
+    def __init__(self, project, _=None, __=None):


logging/unit_tests/test_client.py

@@ -584,6 +637,9 @@ def create_scoped(self, scope):
        self._scopes = scope
        return self

+    def authorize(self, http):
+        pass


logging/google/cloud/logging/handlers/GAEHandler.py

+Logs to the well-known file that the fluentd sidecar container on App Engine
+Flexible is configured to read from and send to Stackdriver Logging.
+
+This file is largely copied from:


logging/google/cloud/logging/handlers/GAEHandler.py

+import math
+import os
+
+LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'


logging/google/cloud/logging/handlers/GAEHandler.py

+LOG_FILE_COUNT = 3
+
+
+class GAEHandler(logging.handlers.RotatingFileHandler):


logging/google/cloud/logging/handlers/GAEHandler.py

+
+
+class GAEHandler(logging.handlers.RotatingFileHandler):
+    """A handler that writes to the GAE fluentd Stackdriver logging file.


logging/google/cloud/logging/handlers/GAEHandler.py

+class GAEHandler(logging.handlers.RotatingFileHandler):
+    """A handler that writes to the GAE fluentd Stackdriver logging file.
+
+    Writes to the file that the fluentd agent on App Engine Flexbile is


logging/google/cloud/logging/handlers/transports/background_thread.py

-                target=self._run,
-                name='google.cloud.logging.handlers.transport.Worker')
+            thread_name = 'google.cloud.logging.handlers.transport.Worker'
+            self._thread = threading.Thread(target=self._run,


logging/unit_tests/handlers/test_GAEHandler.py

+        LOGNAME = 'loggername'
+        MESSAGE = 'hello world'
+        record = _Record(LOGNAME, logging.INFO, MESSAGE)
+        expected_payload = {'message': MESSAGE,


logging/unit_tests/handlers/test_GAEHandler.py

+        self.assertEqual(payload, json.dumps(expected_payload))
+
+
+class _Record(object):


logging/unit_tests/handlers/test_GAEHandler.py

+        import logging
+        import json
+        handler = self._makeOne()
+        LOGNAME = 'loggername'


logging/unit_tests/handlers/transports/test_background_thread.py

@@ -184,7 +184,7 @@ def batch(self):

 class _Client(object):

-    def __init__(self, project):
+    def __init__(self, project, _=None, __=None):


docs/index.rst

@@ -110,6 +110,8 @@
  logging-metric
  logging-sink
  logging-handlers
+  logging-handlers-GAEHandler
+  logging-handlers-GKEHandler


docs/logging-handlers-GAEHandler.rst

+Google App Engine Flexible Log Handler
+=======================================
+
+.. automodule:: google.cloud.logging.handlers.GAEHandler


docs/logging-handlers-GKEHandler.rst

+Google Container Engine Log Handler
+=======================================
+
+.. automodule:: google.cloud.logging.handlers.GKEHandler


docs/logging-usage.rst

-create a :class:`CloudLoggingHandler <google.cloud.logging.CloudLoggingHandler>` instance from your
-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,


docs/logging-usage.rst

-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,
+use :meth:``google.cloud.logging.client.get_default_handler`.


logging/unit_tests/test_client.py

+
+        client = self._makeOne(self.PROJECT, credentials=_Credentials())
+
+        with _Monkey(os, getenv=lambda var: var == 'GAE_APPENGINE_HOSTNAME'):


logging/unit_tests/test_client.py

+
+        self.assertIsInstance(handler, GKEHandler)
+
+        with _Monkey(os, getenv=lambda _: None):


logging/unit_tests/test_client.py

+
+        self.assertIsInstance(handler, GAEHandler)
+
+        with _Monkey(os, getenv=lambda var: var == 'KUBERNETES_SERVICE'):


logging/unit_tests/test_client.py

+    def test_setup_logging(self):
+        from google.cloud._testing import _Monkey
+        import google.cloud.logging.client as MUT
+        client = self._makeOne(


logging/unit_tests/test_client.py

+        self.called = False
+
+    def __call__(self, _, log_level,
+                 excluded_loggers=None):  # pylintdisable=unused-variable


waprin · 2016-11-04T00:09:33Z

Think I addressed all comments, let me make sure Travis is passing and then I'll do another once-over before I request another review.

docs/logging-usage.rst

@@ -390,11 +391,12 @@ Delete a sink:
   False

 Integration with Python logging module
---------------------------------------------
+---------------------------------------


logging/unit_tests/test_client.py

@@ -596,7 +596,7 @@ def test_get_default_handler_container_engine(self):
        client = self._makeOne(project=self.PROJECT,
                               credentials=_Credentials(),
                               use_gax=False)
-        os.environ[_CONTAINER_ENGINE_ENV] = True
+        os.environ[_CONTAINER_ENGINE_ENV] = "True"


docs/logging-usage.rst

+(i.e. your app.yaml contains ``runtime: python``), and
+:class:`~google.cloud.handlers.container_engine.ContainerEngineHandler`,
+which is
+recommended when running on `Google Container Engine` with the Stackdriver


logging/google/cloud/logging/client.py

+from google.cloud.logging.handlers import AppEngineHandler
+from google.cloud.logging.handlers import ContainerEngineHandler
+from google.cloud.logging.handlers import setup_logging
+from google.cloud.logging.handlers.handlers import EXCLUDED_LOGGER_DEFAULTS


logging/google/cloud/logging/handlers/app_engine.py

+"""
+
+# This file is largely copied from:
+# https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master/appengine-vmruntime/vmruntime/cloud_logging.py


docs/logging-usage.rst

-   >>> setup_logging(handler, excluded_loggers=('werkzeug',)))
+    >>> handler = CloudLoggingHandler(client, name="mycustomlog")
+
+fluentd logging handlers


docs/logging-usage.rst

+the environment to automatically detect whether the code is running in
+these platforms and use the appropriate handler.
+
+In both cases, the fluentd agent is configured to automatically parse log files


docs/logging-usage.rst

-method :meth:`setup_logging <google.cloud.logging.handlers.setup_logging>` is provided to configure
-this automatically:
+If you prefer not to use :meth:`google.cloud.logging.client
+.get_default_handler`, you can directly create a


logging/unit_tests/handlers/test_app_engine.py

+            'message': message,
+            'timestamp': {
+                'seconds': 5,
+                'nanos': int(.03 * 1e9)


logging/unit_tests/handlers/test_container_engine.py

+                            'timestamp': {'seconds': 5,
+                                          'nanos': int(.03 * 1e9)},
+                            'thread': record.thread,
+                            'severity': record.levelname}


logging/google/cloud/logging/handlers/app_engine.py

+                'nanos': int(subsecond * 1e9)
+            },
+            'thread': record.thread,
+            'severity': record.levelname


logging/google/cloud/logging/handlers/app_engine.py

+            'message': message,
+            'timestamp': {
+                'seconds': int(second),
+                'nanos': int(subsecond * 1e9)


logging/google/cloud/logging/handlers/container_engine.py

+        payload = {
+            'message': message,
+            'timestamp': {'seconds': int(second),
+                          'nanos': int(subsecond * 1e9)},


logging/google/cloud/logging/handlers/app_engine.py

+class AppEngineHandler(logging.handlers.RotatingFileHandler):
+    """A handler that writes to the App Engine fluentd Stackdriver log file.
+
+    Writes to the file that the fluentd agent on App Engine Flexbile is


logging/google/cloud/logging/handlers/app_engine.py

+    itself, and not as a formatting step, so as not to interfere with
+    user-defined logging formats.
+
+    Logging calls can also alternatively 'trace_id' in as a field in the


logging/google/cloud/logging/handlers/container_engine.py

+"""Logging handler for Google Container Engine (GKE).
+
+Formats log messages in a JSON format, so that Kubernetes clusters with the
+fluentd google cloud plugin installed can format their log messages so that


logging/google/cloud/logging/handlers/container_engine.py

+
+
+class ContainerEngineHandler(logging.StreamHandler):
+    """Handler to formats log messages in good fluentd GKE format.


logging/google/cloud/logging/handlers/handlers.py

@@ -15,27 +15,21 @@
 """Python :mod:`logging` handlers for Google Cloud Logging."""

 import logging
-


logging/unit_tests/handlers/test_container_engine.py

+class TestContainerEngineHandler(unittest.TestCase):
+    PROJECT = 'PROJECT'
+
+    def _getTargetClass(self):


logging/unit_tests/handlers/test_container_engine.py

+            'message': message,
+            'timestamp': {
+                'seconds': 5,
+                'nanos': int(.03 * 1e9)


logging/unit_tests/handlers/transports/test_background_thread.py

-            'message': MESSAGE,
-            'python_logger': PYTHON_LOGGER_NAME
+            'message': message,
+            'python_logger': python_logger_name


logging/unit_tests/handlers/transports/test_sync.py

-            'message': MESSAGE,
-            'python_logger': PYTHON_LOGGER_NAME
+            'message': message,
+            'python_logger': python_logger_name


logging/unit_tests/test_client.py

@@ -584,6 +637,9 @@ def create_scoped(self, scope):
        self._scopes = scope
        return self

+    def authorize(self, http):
+        pass


logging/unit_tests/test_client.py

@@ -562,6 +629,9 @@ def create_scoped(self, scope):
        self._scopes = scope
        return self

+    def authorize(self, _):


logging/unit_tests/test_client.py

@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+import mock


logging/unit_tests/test_client.py

                                use_gax=False)
        handler = client.get_default_handler()
        self.assertIsInstance(handler, CloudLoggingHandler)
+        self.assertTrue(credentials.authorized, http_mock)


logging/unit_tests/test_client.py

        from google.cloud.logging.handlers import CloudLoggingHandler

+        http_mock = mock.Mock()


logging/unit_tests/test_client.py

        from google.cloud._testing import _Monkey
        import google.cloud.logging.client as MUT

+        http_mock = mock.Mock()


logging/unit_tests/test_client.py


-    def test_setup_logging(self):
+    @mock.patch('copy.deepcopy')
+    def test_setup_logging(self, deepcopy):


dhermes · 2016-11-21T22:13:28Z

@waprin Pretty please squash (either in the GitHub UI or in your local copy).

@jonparrott Do you want to do a final pass?

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations. This adds two new handlers and some helper functions to detect when they should be used.

Add GAE and GKE fluentd Handlers

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 3, 2016

daspecster reviewed Nov 3, 2016

View reviewed changes

theacodes suggested changes Nov 3, 2016

View reviewed changes

theacodes self-assigned this Nov 3, 2016

dhermes reviewed Nov 3, 2016

View reviewed changes

This was referenced Nov 3, 2016

Python Logging API always defaults to Custom logs #2673

Closed

Convert logging doctest examples to testable snippets. #2675

Merged

dhermes reviewed Nov 4, 2016

View reviewed changes

docs/logging-usage.rst

(i.e. your app.yaml contains ``runtime: python``), and

:class:`~google.cloud.handlers.container_engine.ContainerEngineHandler`,

which is

recommended when running on `Google Container Engine` with the Stackdriver

This comment was marked as spam.

Sign in to view

waprin force-pushed the newhandlers branch from a81e8dd to 31641bd Compare November 4, 2016 20:56

dhermes reviewed Nov 4, 2016

View reviewed changes

waprin force-pushed the newhandlers branch 4 times, most recently from c755a71 to c132584 Compare November 16, 2016 21:50

dhermes reviewed Nov 18, 2016

View reviewed changes

logging/unit_tests/test_client.py

@@ -562,6 +629,9 @@ def create_scoped(self, scope):

self._scopes = scope

return self

def authorize(self, _):

This comment was marked as spam.

Sign in to view

dhermes reviewed Nov 21, 2016

View reviewed changes

dhermes approved these changes Nov 21, 2016

View reviewed changes

waprin force-pushed the newhandlers branch from 4477d59 to a65ef9f Compare November 21, 2016 22:21

waprin force-pushed the newhandlers branch from a65ef9f to 02ae299 Compare November 21, 2016 22:24

theacodes approved these changes Nov 21, 2016

View reviewed changes

dhermes merged commit 5d59291 into googleapis:master Nov 22, 2016

daspecster added the api: logging Issues related to the Cloud Logging API. label Nov 22, 2016

dhermes mentioned this pull request Dec 9, 2016

Update versions for mega-release. #2846

Merged

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Merge pull request googleapis#2668 from waprin/newhandlers

a787604

Add GAE and GKE fluentd Handlers

		@@ -0,0 +1,6 @@
		Google App Engine Flexible Log Handler
		=======================================

		@@ -0,0 +1,6 @@
		Google Container Engine Log Handler
		=======================================

		LOG_FILE_COUNT = 3


		class GAEHandler(logging.handlers.RotatingFileHandler):



		class GAEHandler(logging.handlers.RotatingFileHandler):
		"""A handler that writes to the GAE fluentd Stackdriver logging file.

		self.assertEqual(payload, json.dumps(expected_payload))


		class _Record(object):


		client = self._makeOne(self.PROJECT, credentials=_Credentials())

		with _Monkey(os, getenv=lambda var: var == 'GAE_APPENGINE_HOSTNAME'):


		self.assertIsInstance(handler, GKEHandler)

		with _Monkey(os, getenv=lambda _: None):


		self.assertIsInstance(handler, GAEHandler)

		with _Monkey(os, getenv=lambda var: var == 'KUBERNETES_SERVICE'):



		class ContainerEngineHandler(logging.StreamHandler):
		"""Handler to formats log messages in good fluentd GKE format.

		@@ -15,27 +15,21 @@
		"""Python :mod:`logging` handlers for Google Cloud Logging."""

		import logging

		from google.cloud.logging.handlers import CloudLoggingHandler

		http_mock = mock.Mock()

Add GAE and GKE fluentd Handlers #2668

Add GAE and GKE fluentd Handlers #2668

Conversation

waprin commented Nov 3, 2016

waprin commented Nov 3, 2016

daspecster left a comment

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

waprin commented Nov 4, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.