From dcc15e38871c0011345d783f5ee1d5dae7b5d31d Mon Sep 17 00:00:00 2001 From: Nok Date: Fri, 9 Jun 2023 12:50:30 +0000 Subject: [PATCH 01/23] update release note Signed-off-by: Nok --- RELEASE.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index 77453869dc..a457367c27 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -6,6 +6,7 @@ ## Breaking changes to the API + ## Migration guide from Kedro 0.18.* to 0.19.* # Upcoming Release 0.18.11 @@ -15,7 +16,7 @@ ## Bug fixes and other changes ## Breaking changes to the API - +* Logging is decoupled from `ConfigLoader`, use `KEDRO_LOGGING_CONFIG` to configure logging. ## Upcoming deprecations for Kedro 0.19.0 # Release 0.18.10 From 874f1a2617d49e4b32240ea83a16c70ca02e5870 Mon Sep 17 00:00:00 2001 From: Nok Date: Fri, 9 Jun 2023 15:17:29 +0000 Subject: [PATCH 02/23] update logging conf/base mention and filebase logging Signed-off-by: Nok --- docs/source/logging/index.md | 7 ++++--- docs/source/logging/logging.md | 16 +++++++++------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index a3750bb188..f87b9790a0 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -1,9 +1,10 @@ # Logging -Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in two places: -1. [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). This cannot be altered. -2. Your project-side logging configuration. Every project generated using Kedro's CLI `kedro new` command includes a file `conf/base/logging.yml`. You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md). +Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). + +## Advance Logging +You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md). ```{note} Providing project-side logging configuration is entirely optional. You can delete the `conf/base/logging.yml` file and Kedro will run using the framework's built in configuration. diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 93f3332660..ef35b9ab67 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -7,7 +7,7 @@ By default, Python only shows logging messages at level `WARNING` and above. Ked ## Project-side logging configuration -In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/base/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: +In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: * `console`: show logs on standard output (typically your terminal screen) without any rich formatting * `info_file_handler`: write logs of level `INFO` and above to `info.log` @@ -32,23 +32,25 @@ After setting the environment variable, any subsequent Kedro commands will use t ```{note} If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` -### Disable file-based logging +### Enable file-based logging -You might sometimes need to disable file-based logging, e.g. if you are running Kedro on a read-only file system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). The simplest way to do this is to delete your `conf/base/logging.yml` file. With no project-side logging configuration specified, Kedro uses the default framework-side logging configuration, which does not include any file-based handlers. +File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). -Alternatively, if you would like to keep other configuration in `conf/base/logging.yml` and just disable file-based logging, then you can remove the file-based handlers from the `root` logger as follows: +To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: ```diff root: -- handlers: [console, info_file_handler] -+ handlers: [console] +- handlers: [console] ++ handlers: [console, info_file_handler] ``` +By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. + ### Customise the `rich` Handler Kedro's `kedro.extras.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. ```{note} -If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/base/logging.yml`. +If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/logging.yml`. ``` When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings. From 62abdfd6f56451f6c02b64de73ec4bfc4243788a Mon Sep 17 00:00:00 2001 From: Nok Date: Fri, 9 Jun 2023 15:32:39 +0000 Subject: [PATCH 03/23] Reorder the logging page Signed-off-by: Nok --- docs/source/logging/index.md | 1 - docs/source/logging/logging.md | 107 ++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 44 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index f87b9790a0..bfaa769fa1 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -3,7 +3,6 @@ Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). -## Advance Logging You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md). ```{note} diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index ef35b9ab67..d289bedcfc 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -1,11 +1,55 @@ -# Default framework-side logging configuration +# Default logging configuration Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the [Rich logging handler](https://rich.readthedocs.io/en/stable/logging.html) to format messages. We also use the [Rich traceback handler](https://rich.readthedocs.io/en/stable/traceback.html) to render exceptions. By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. -## Project-side logging configuration +## Perform logging in your project + +To perform logging in your own code (e.g. in a node), you are advised to do as follows: + +```python +import logging + +log = logging.getLogger(__name__) +log.warning("Issue warning") +log.info("Send information") +``` + +```{note} +The name of a logger corresponds to a key in the `loggers` section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. +``` + +You can take advantage of rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls: +```python +log.error("[bold red blink]Important error message![/]", extra={"markup": True}) +``` + +## Show DEBUG level messages +In your `logging.yml`, you can change the level of log messages that you wish to see. + +```yml +loggers: + kedro: + level: INFO + + your_python_pacakge: + level: INFO +``` + +For example, if you want to show the `DEBUG` level message of your own project. You need to update your `logging.yml` as follow: + +```yml +loggers: + kedro: + level: INFO + + your_python_pacakge: + level: DEBUG +``` + +# Customise Logging In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: * `console`: show logs on standard output (typically your terminal screen) without any rich formatting @@ -17,14 +61,12 @@ The project-side logging configuration also ensures that [logs emitted from your We now give some common examples of how you might like to change your project's logging configuration. -### Using `KEDRO_LOGGING_CONFIG` environment variable - -`KEDRO_LOGGING_CONFIG` is an optional environment variable that you can use to specify the path of your logging configuration file, overriding the default Kedro's `default_logging.yml`. +## Using `KEDRO_LOGGING_CONFIG` environment variable -To use this environment variable, set it to the path of your desired logging configuration file before running any Kedro commands. For example, if you have a logging configuration file located at `/path/to/logging.yml`, you can set `KEDRO_LOGGING_CONFIG` as follows: +In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml` For example, you can set `KEDRO_LOGGING_CONFIG` as follows: ```bash -export KEDRO_LOGGING_CONFIG=/path/to/logging.yml +export KEDRO_LOGGING_CONFIG=/conf/logging.yml ``` After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path. @@ -32,22 +74,10 @@ After setting the environment variable, any subsequent Kedro commands will use t ```{note} If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` -### Enable file-based logging +# Advance Logging +## Customise the `rich` Handler -File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). - -To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: -```diff - root: -- handlers: [console] -+ handlers: [console, info_file_handler] -``` - -By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. - -### Customise the `rich` Handler - -Kedro's `kedro.extras.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. +Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. ```{note} If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/logging.yml`. @@ -59,7 +89,7 @@ For instance, you can enable the display of local variables inside `logging.yml` ```yaml rich: - class: kedro.extras.logging.RichHandler + class: kedro.logging.RichHandler rich_tracebacks: True tracebacks_show_locals: True ``` @@ -67,14 +97,14 @@ rich: A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). -### Use plain console logging +## Use plain console logging To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows: ```diff root: -- handlers: [rich, info_file_handler] -+ handlers: [console, info_file_handler] +- handlers: [rich] ++ handlers: [console] ``` ### Rich logging in a dumb terminal @@ -94,24 +124,15 @@ You must provide a value for both `COLUMNS` and `LINES` even if you only wish to ### Rich logging in Jupyter Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`. +## Enable file-based logging -## Perform logging in your project - -To perform logging in your own code (e.g. in a node), you are advised to do as follows: - -```python -import logging - -log = logging.getLogger(__name__) -log.warning("Issue warning") -log.info("Send information") -``` +File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). -```{note} -The name of a logger corresponds to a key in the `loggers` section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. +To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: +```diff + root: +- handlers: [console] ++ handlers: [console, info_file_handler] ``` -You can take advantage of rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls: -```python -log.error("[bold red blink]Important error message![/]", extra={"markup": True}) -``` +By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. From e6787d5e213ed3c81d2b2a058c100c66b22f6823 Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 14 Jun 2023 14:02:14 +0000 Subject: [PATCH 04/23] reorder and update logging docs Signed-off-by: Nok --- .../databricks/databricks_workspace.md | 3 - docs/source/logging/logging.md | 57 +++++++------------ 2 files changed, 22 insertions(+), 38 deletions(-) diff --git a/docs/source/deployment/databricks/databricks_workspace.md b/docs/source/deployment/databricks/databricks_workspace.md index 9f68b7ba01..14e1b8f92e 100644 --- a/docs/source/deployment/databricks/databricks_workspace.md +++ b/docs/source/deployment/databricks/databricks_workspace.md @@ -2,9 +2,6 @@ This tutorial uses the [PySpark Iris Kedro Starter](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris) to illustrate how to bootstrap a Kedro project using Spark and deploy it to a [Databricks cluster on AWS](https://databricks.com/aws). -```{note} -If you are using [Databricks Repos](https://docs.databricks.com/repos/index.html) to run a Kedro project then you should [disable file-based logging](../../logging/logging.md#disable-file-based-logging). This prevents Kedro from attempting to write to the read-only file system. -``` ```{note} If you are a Kedro contributor looking for information on deploying a custom build of Kedro to Databricks, see the [development guide](../../contribution/development_for_databricks.md). diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index d289bedcfc..44b310ccaa 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -6,7 +6,6 @@ Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/ By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. ## Perform logging in your project - To perform logging in your own code (e.g. in a node), you are advised to do as follows: ```python @@ -15,6 +14,7 @@ import logging log = logging.getLogger(__name__) log.warning("Issue warning") log.info("Send information") +log.debug("Useful information for debugging") ``` ```{note} @@ -26,8 +26,8 @@ You can take advantage of rich's [console markup](https://rich.readthedocs.io/en log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ``` -## Show DEBUG level messages -In your `logging.yml`, you can change the level of log messages that you wish to see. +### Show DEBUG level messages +To see your `DEBUG` level message, you can change the level of log messages that you wish to see in `logging.yml`. ```yml loggers: @@ -35,34 +35,21 @@ loggers: level: INFO your_python_pacakge: - level: INFO + level: DEBUG # Change this to DEBUG ``` -For example, if you want to show the `DEBUG` level message of your own project. You need to update your `logging.yml` as follow: - -```yml -loggers: - kedro: - level: INFO - - your_python_pacakge: - level: DEBUG -``` - -# Customise Logging - +## Customise Logging In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: * `console`: show logs on standard output (typically your terminal screen) without any rich formatting * `info_file_handler`: write logs of level `INFO` and above to `info.log` -The logging handlers that are actually used by default are `rich` and `info_file_handler`. +The logging handlers that are actually used by default is `rich`. -The project-side logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). +The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). We now give some common examples of how you might like to change your project's logging configuration. -## Using `KEDRO_LOGGING_CONFIG` environment variable - +### Using `KEDRO_LOGGING_CONFIG` environment variable In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml` For example, you can set `KEDRO_LOGGING_CONFIG` as follows: ```bash @@ -96,6 +83,18 @@ rich: A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). +## Enable file-based logging + +File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). + +To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: +```diff + root: +- handlers: [console] ++ handlers: [console, info_file_handler] +``` + +By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. ## Use plain console logging @@ -107,7 +106,7 @@ To use plain rather than rich logging, swap the `rich` handler for the `console` + handlers: [console] ``` -### Rich logging in a dumb terminal +## Rich logging in a dumb terminal Rich [detects whether your terminal is capable](https://rich.readthedocs.io/en/stable/console.html#terminal-detection) of displaying richly formatted messages. If your terminal is "dumb" then formatting is automatically stripped out so that the logs are just plain text. This is likely to happen if you perform `kedro run` on CI (e.g. GitHub Actions or CircleCI). @@ -121,18 +120,6 @@ export COLUMNS=120 LINES=25 You must provide a value for both `COLUMNS` and `LINES` even if you only wish to change the width of the log message. Rich's default values for these variables are `COLUMNS=80` and `LINE=25`. ``` -### Rich logging in Jupyter +## Rich logging in Jupyter Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`. -## Enable file-based logging - -File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). - -To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: -```diff - root: -- handlers: [console] -+ handlers: [console, info_file_handler] -``` - -By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. From d97198c0cc67d29dff94762a1e043c468ef1c811 Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 14 Jun 2023 14:52:24 +0000 Subject: [PATCH 05/23] update docs Signed-off-by: Nok --- docs/source/logging/logging.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 44b310ccaa..9bb0ad68ef 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -1,9 +1,7 @@ - # Default logging configuration +Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. -Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the [Rich logging handler](https://rich.readthedocs.io/en/stable/logging.html) to format messages. We also use the [Rich traceback handler](https://rich.readthedocs.io/en/stable/traceback.html) to render exceptions. - -By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. +By default, Python only shows logging messages at level WARNING and above. Kedro's logging configuration specifies that INFO level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. ## Perform logging in your project To perform logging in your own code (e.g. in a node), you are advised to do as follows: @@ -38,6 +36,8 @@ loggers: level: DEBUG # Change this to DEBUG ``` +By changing the level value to `DEBUG` for the desired logger (e.g., ), you will start seeing `DEBUG` level messages in the log output. + ## Customise Logging In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: * `console`: show logs on standard output (typically your terminal screen) without any rich formatting @@ -47,7 +47,7 @@ The logging handlers that are actually used by default is `rich`. The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). -We now give some common examples of how you might like to change your project's logging configuration. +Now, let's provide some common examples of how you might like to change your project's logging configuration. ### Using `KEDRO_LOGGING_CONFIG` environment variable In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml` For example, you can set `KEDRO_LOGGING_CONFIG` as follows: From 1823dcb621e7c3f2aad0c681c9c4ce2c3da894be Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Mon, 19 Jun 2023 12:00:55 +0100 Subject: [PATCH 06/23] Apply suggestions from code review Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> --- docs/source/logging/index.md | 2 +- docs/source/logging/logging.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index bfaa769fa1..ae6f3e4021 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -1,7 +1,7 @@ # Logging -Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). +Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md). diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 9bb0ad68ef..6e875f771b 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -4,7 +4,7 @@ Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/ By default, Python only shows logging messages at level WARNING and above. Kedro's logging configuration specifies that INFO level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. ## Perform logging in your project -To perform logging in your own code (e.g. in a node), you are advised to do as follows: +To add logging to your own code (e.g. in a node), you are advised to do as follows: ```python import logging From 7a91f11687ba600cd8d5ff6be0c3e9e10974aa52 Mon Sep 17 00:00:00 2001 From: Nok Date: Mon, 19 Jun 2023 11:12:23 +0000 Subject: [PATCH 07/23] add logging.yml template and reorder section Signed-off-by: Nok --- docs/source/logging/logging.md | 75 ++++++++++++++++++++++++++++------ 1 file changed, 63 insertions(+), 12 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 9bb0ad68ef..8e554d0b49 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -39,18 +39,9 @@ loggers: By changing the level value to `DEBUG` for the desired logger (e.g., ), you will start seeing `DEBUG` level messages in the log output. ## Customise Logging -In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers: -* `console`: show logs on standard output (typically your terminal screen) without any rich formatting -* `info_file_handler`: write logs of level `INFO` and above to `info.log` - -The logging handlers that are actually used by default is `rich`. - -The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). - -Now, let's provide some common examples of how you might like to change your project's logging configuration. ### Using `KEDRO_LOGGING_CONFIG` environment variable -In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml` For example, you can set `KEDRO_LOGGING_CONFIG` as follows: +In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml`. We recommend to put your `logging.yml` inside the `conf` folder. For example, you can set `KEDRO_LOGGING_CONFIG` as follows: ```bash export KEDRO_LOGGING_CONFIG=/conf/logging.yml @@ -61,7 +52,67 @@ After setting the environment variable, any subsequent Kedro commands will use t ```{note} If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` + + # Advance Logging +In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template that you can use to start with. + +
+Click to expand the `logging.yml` template +```yml +version: 1 + +disable_existing_loggers: False + +formatters: + simple: + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + +handlers: + console: + class: logging.StreamHandler + level: INFO + formatter: simple + stream: ext://sys.stdout + + info_file_handler: + class: logging.handlers.RotatingFileHandler + level: INFO + formatter: simple + filename: info.log + maxBytes: 10485760 # 10MB + backupCount: 20 + encoding: utf8 + delay: True + + rich: + class: kedro.logging.RichHandler + rich_tracebacks: True + # Advance options for customisation. + # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration + # tracebacks_show_locals: False + +loggers: + kedro: + level: INFO + + {{ cookiecutter.python_package }}: + level: INFO + +root: + handlers: [rich] +``` +
+ +* `console`: show logs on standard output (typically your terminal screen) without any rich formatting +* `info_file_handler`: write logs of level `INFO` and above to `info.log` + +The logging handlers that are actually used by default is `rich`. + +The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). + +Now, let's provide some common examples of how you might like to change your project's logging configuration. + ## Customise the `rich` Handler Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. @@ -90,8 +141,8 @@ File-based logging in Python projects aids troubleshooting and debugging. It off To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: ```diff root: -- handlers: [console] -+ handlers: [console, info_file_handler] +- handlers: [rich] ++ handlers: [rich, info_file_handler] ``` By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. From 0764cbf4ea1bf06222164158fc39112ce10c0220 Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Thu, 22 Jun 2023 14:03:53 +0100 Subject: [PATCH 08/23] Apply suggestions from code review Co-authored-by: Jo Stichbury --- docs/source/logging/logging.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 692e6708b7..2ed44904e9 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -1,10 +1,10 @@ # Default logging configuration Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. -By default, Python only shows logging messages at level WARNING and above. Kedro's logging configuration specifies that INFO level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. +By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. ## Perform logging in your project -To add logging to your own code (e.g. in a node), you are advised to do as follows: +To add logging to your own code (e.g. in a node): ```python import logging @@ -19,13 +19,13 @@ log.debug("Useful information for debugging") The name of a logger corresponds to a key in the `loggers` section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. ``` -You can take advantage of rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls: +You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls: ```python log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ``` ### Show DEBUG level messages -To see your `DEBUG` level message, you can change the level of log messages that you wish to see in `logging.yml`. +To see `DEBUG` level messages, change the level of logging in `logging.yml`. ```yml loggers: @@ -41,7 +41,7 @@ By changing the level value to `DEBUG` for the desired logger (e.g., /conf/logging.yml @@ -55,7 +55,7 @@ If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will defaul # Advance Logging -In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template that you can use to start with. +In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template.
Click to expand the `logging.yml` template @@ -107,7 +107,7 @@ root: * `console`: show logs on standard output (typically your terminal screen) without any rich formatting * `info_file_handler`: write logs of level `INFO` and above to `info.log` -The logging handlers that are actually used by default is `rich`. +The logging handlers used by default are from `rich`. The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). @@ -136,7 +136,7 @@ A comprehensive list of available options can be found in the [RichHandler docum ## Enable file-based logging -File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). +File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behaviour and it's easy to search. However, it does not work well with read-only systems such as [Databricks Repos](https://docs.databricks.com/repos/index.html). To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: ```diff From 5eefee3fbeabb201db8b83e0c24fee0b1194c40c Mon Sep 17 00:00:00 2001 From: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Date: Fri, 23 Jun 2023 16:12:38 +0100 Subject: [PATCH 09/23] Structure docs paragraphs (#2717) * Move paragraphs around Signed-off-by: Ahdra Merali * Fix RTD build and tidy copy in logging Signed-off-by: Jo Stichbury --------- Signed-off-by: Ahdra Merali Signed-off-by: Jo Stichbury Co-authored-by: Jo Stichbury --- docs/source/development/set_up_pycharm.md | 2 +- docs/source/logging/logging.md | 58 ++++++++++++----------- 2 files changed, 31 insertions(+), 29 deletions(-) diff --git a/docs/source/development/set_up_pycharm.md b/docs/source/development/set_up_pycharm.md index a18bb9ff12..b7a34d5c8b 100644 --- a/docs/source/development/set_up_pycharm.md +++ b/docs/source/development/set_up_pycharm.md @@ -67,7 +67,7 @@ Edit the new Run configuration as follows: Replace **Script path** with path obtained above and **Working directory** with the path of your project directory and then click **OK**. ```{note} -**Emulate terminal in output console** enables PyCharm to show [rich terminal output](../logging/logging.md#default-framework-side-logging-configuration). +**Emulate terminal in output console** enables PyCharm to show [rich terminal output](../logging/logging.md). ``` To execute the Run configuration, select it from the **Run / Debug Configurations** dropdown in the toolbar (if that toolbar is not visible, you can enable it by going to **View > Toolbar**). Click the green triangle: diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 2ed44904e9..31758ab1f9 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -3,7 +3,7 @@ Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/ By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. -## Perform logging in your project +## How to perform logging in your Kedro project To add logging to your own code (e.g. in a node): ```python @@ -16,29 +16,16 @@ log.debug("Useful information for debugging") ``` ```{note} -The name of a logger corresponds to a key in the `loggers` section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. +The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. ``` -You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls: +You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: + ```python log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ``` -### Show DEBUG level messages -To see `DEBUG` level messages, change the level of logging in `logging.yml`. - -```yml -loggers: - kedro: - level: INFO - - your_python_pacakge: - level: DEBUG # Change this to DEBUG -``` - -By changing the level value to `DEBUG` for the desired logger (e.g., ), you will start seeing `DEBUG` level messages in the log output. - -## Customise Logging +## How to customise Kedro logging ### Using `KEDRO_LOGGING_CONFIG` environment variable To customise logging, specify the path of your logging configuration file by setting the environment variable `KEDRO_LOGGING_CONFIG` to override the default, which is Kedro's `default_logging.yml`. We recommend you put `logging.yml` inside the `conf` folder. For example, you can set `KEDRO_LOGGING_CONFIG` as follows: @@ -53,13 +40,27 @@ After setting the environment variable, any subsequent Kedro commands will use t If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` +### Show DEBUG level messages +To see `DEBUG` level messages, change the level of logging in `logging.yml`: + +```yaml +loggers: + kedro: + level: INFO + + your_python_pacakge: + level: DEBUG # Change this to DEBUG +``` + +By changing the level value to `DEBUG` for the desired logger (e.g., ), you will start seeing `DEBUG` level messages in the log output. -# Advance Logging +# Advanced logging In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template.
Click to expand the `logging.yml` template -```yml + +```yaml version: 1 disable_existing_loggers: False @@ -109,11 +110,11 @@ root: The logging handlers used by default are from `rich`. -The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). +The default logging configuration also ensures that [logs emitted from your project's logger](#how-to-perform-logging-in-your-kedro-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). -Now, let's provide some common examples of how you might like to change your project's logging configuration. +The following section illustrates some common examples of how to change your project's logging configuration. -## Customise the `rich` Handler +## How to customise the `rich` handler Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. @@ -134,11 +135,12 @@ rich: A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). -## Enable file-based logging +## How to enable file-based logging File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behaviour and it's easy to search. However, it does not work well with read-only systems such as [Databricks Repos](https://docs.databricks.com/repos/index.html). -To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow: +To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follows: + ```diff root: - handlers: [rich] @@ -147,7 +149,7 @@ To enable file-based logging, add `info_file_handler` in your `root` logger as By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. -## Use plain console logging +## How to use plain console logging To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows: @@ -157,7 +159,7 @@ To use plain rather than rich logging, swap the `rich` handler for the `console` + handlers: [console] ``` -## Rich logging in a dumb terminal +## How to enable rich logging in a dumb terminal Rich [detects whether your terminal is capable](https://rich.readthedocs.io/en/stable/console.html#terminal-detection) of displaying richly formatted messages. If your terminal is "dumb" then formatting is automatically stripped out so that the logs are just plain text. This is likely to happen if you perform `kedro run` on CI (e.g. GitHub Actions or CircleCI). @@ -171,6 +173,6 @@ export COLUMNS=120 LINES=25 You must provide a value for both `COLUMNS` and `LINES` even if you only wish to change the width of the log message. Rich's default values for these variables are `COLUMNS=80` and `LINE=25`. ``` -## Rich logging in Jupyter +## How to enable rich logging in Jupyter Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`. From e27e334c70e07e444adc83629eed6fb5cd119b90 Mon Sep 17 00:00:00 2001 From: Nok Date: Fri, 23 Jun 2023 16:42:15 +0000 Subject: [PATCH 10/23] Remove the broken aws step function link Signed-off-by: Nok --- docs/source/deployment/aws_step_functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/deployment/aws_step_functions.md b/docs/source/deployment/aws_step_functions.md index c6f8d6974a..dff2eaeade 100644 --- a/docs/source/deployment/aws_step_functions.md +++ b/docs/source/deployment/aws_step_functions.md @@ -200,7 +200,7 @@ ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ] CMD [ "lambda_handler.handler" ] ``` -This `Dockerfile` is adapted from the official guide on [how to create a custom image](https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-create-from-alt) for Lambda to include Kedro-specific steps. +This `Dockerfile` is adapted from the official guide on how to create a custom image for Lambda to include Kedro-specific steps. * **Step 2.4**: Build the Docker image and push it to AWS Elastic Container Registry (ECR): From c12ddc6f65035b7a21cf216c26a81a60bb799b21 Mon Sep 17 00:00:00 2001 From: Nok Date: Mon, 26 Jun 2023 11:31:01 +0000 Subject: [PATCH 11/23] Fix docs according to comments Signed-off-by: Nok --- docs/source/logging/index.md | 8 -------- docs/source/logging/logging.md | 3 --- 2 files changed, 11 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index ae6f3e4021..4ab86b7571 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -3,14 +3,6 @@ Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). -You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md). - -```{note} -Providing project-side logging configuration is entirely optional. You can delete the `conf/base/logging.yml` file and Kedro will run using the framework's built in configuration. -``` - -Framework-side and project-side logging configuration are loaded through subsequent calls to [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig). This means that, when it is provided, the project-side logging configuration typically _fully overwrites_ the framework-side logging configuration. [Incremental configuration](https://docs.python.org/3/library/logging.config.html#incremental-configuration) is also possible if the `incremental` key is explicitly set to `True` in your project-side logging configuration. - ```{toctree} :hidden: diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 31758ab1f9..760a1c01b9 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -97,9 +97,6 @@ loggers: kedro: level: INFO - {{ cookiecutter.python_package }}: - level: INFO - root: handlers: [rich] ``` From c234f11b8148de07d304c6948a414e170621fdd2 Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Mon, 26 Jun 2023 12:35:21 +0100 Subject: [PATCH 12/23] Update docs/source/logging/logging.md Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> --- docs/source/logging/logging.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 760a1c01b9..430d70b78f 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -107,7 +107,6 @@ root: The logging handlers used by default are from `rich`. -The default logging configuration also ensures that [logs emitted from your project's logger](#how-to-perform-logging-in-your-kedro-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`). The following section illustrates some common examples of how to change your project's logging configuration. From 29ffd0543fc678ec10257c1a65c07910b9968614 Mon Sep 17 00:00:00 2001 From: Nok Date: Mon, 26 Jun 2023 15:12:00 +0000 Subject: [PATCH 13/23] update index Signed-off-by: Nok --- docs/source/logging/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index 4ab86b7571..2240afde38 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -1,7 +1,7 @@ # Logging -Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). +Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). You can also have more [Advanced Logging](logging.md#advanced-logging) ```{toctree} From 517fc2f6184030deecf8b1f02855d7616e0184c5 Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Tue, 27 Jun 2023 14:36:56 +0100 Subject: [PATCH 14/23] Apply suggestions from code review Co-authored-by: Jo Stichbury Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> --- docs/source/logging/logging.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 430d70b78f..4a5b9e2c17 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -1,7 +1,7 @@ # Default logging configuration Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. -By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run. +By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. ## How to perform logging in your Kedro project To add logging to your own code (e.g. in a node): @@ -27,8 +27,9 @@ log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ## How to customise Kedro logging -### Using `KEDRO_LOGGING_CONFIG` environment variable -To customise logging, specify the path of your logging configuration file by setting the environment variable `KEDRO_LOGGING_CONFIG` to override the default, which is Kedro's `default_logging.yml`. We recommend you put `logging.yml` inside the `conf` folder. For example, you can set `KEDRO_LOGGING_CONFIG` as follows: +To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration, which is Kedro's `default_logging.yml`. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. + +For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into your terminal: ```bash export KEDRO_LOGGING_CONFIG=/conf/logging.yml @@ -40,21 +41,21 @@ After setting the environment variable, any subsequent Kedro commands will use t If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` -### Show DEBUG level messages -To see `DEBUG` level messages, change the level of logging in `logging.yml`: +### How to show DEBUG level messages +To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`): ```yaml loggers: kedro: level: INFO - your_python_pacakge: + your_python_package: level: DEBUG # Change this to DEBUG ``` -By changing the level value to `DEBUG` for the desired logger (e.g., ), you will start seeing `DEBUG` level messages in the log output. +By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. -# Advanced logging +## Advanced logging In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template.
@@ -143,7 +144,7 @@ To enable file-based logging, add `info_file_handler` in your `root` logger as + handlers: [rich, info_file_handler] ``` -By default it only tracks `INFO` level message, but it can be configured to capture any level of logs. +By default it only tracks `INFO` level messages, but it can be configured to capture any level of logs. ## How to use plain console logging From dad0012204985cb86ec8a902838c4e88677ec190 Mon Sep 17 00:00:00 2001 From: Nok Date: Tue, 27 Jun 2023 13:49:42 +0000 Subject: [PATCH 15/23] add block to render the logging.yml example Signed-off-by: Nok --- docs/source/logging/logging.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 4a5b9e2c17..3bf0cd086c 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -60,6 +60,7 @@ In addition to the `rich` handler defined in Kedro's framework, we provide a `lo
Click to expand the `logging.yml` template + ```yaml version: 1 @@ -101,6 +102,7 @@ loggers: root: handlers: [rich] ``` +
* `console`: show logs on standard output (typically your terminal screen) without any rich formatting From f8ecc01c847d7bbefca42e74444894edb60e98b7 Mon Sep 17 00:00:00 2001 From: Nok Date: Tue, 27 Jun 2023 14:01:26 +0000 Subject: [PATCH 16/23] reorganise Signed-off-by: Nok --- docs/source/logging/logging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index 3bf0cd086c..c9d6888999 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -1,8 +1,8 @@ +By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. + # Default logging configuration Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. -By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. - ## How to perform logging in your Kedro project To add logging to your own code (e.g. in a node): From a383b4df496d5be15414045a90a8408617957ee6 Mon Sep 17 00:00:00 2001 From: Nok Date: Tue, 27 Jun 2023 16:58:19 +0000 Subject: [PATCH 17/23] Make the style of logging consistent Signed-off-by: Nok --- docs/source/logging/logging.md | 65 +++++++++++++++++----------------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md index c9d6888999..84cd1c5e87 100644 --- a/docs/source/logging/logging.md +++ b/docs/source/logging/logging.md @@ -9,15 +9,13 @@ To add logging to your own code (e.g. in a node): ```python import logging -log = logging.getLogger(__name__) -log.warning("Issue warning") -log.info("Send information") -log.debug("Useful information for debugging") +logger = logging.getLogger(__name__) +logger.warning("Issue warning") +logger.info("Send information") +logger.debug("Useful information for debugging") ``` -```{note} -The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. -``` + You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: @@ -27,7 +25,7 @@ log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ## How to customise Kedro logging -To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration, which is Kedro's `default_logging.yml`. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. +To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into your terminal: @@ -42,21 +40,7 @@ If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will defaul ``` ### How to show DEBUG level messages -To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`): - -```yaml -loggers: - kedro: - level: INFO - - your_python_package: - level: DEBUG # Change this to DEBUG -``` - -By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. - -## Advanced logging -In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template. +To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template.:
Click to expand the `logging.yml` template @@ -98,6 +82,9 @@ handlers: loggers: kedro: level: INFO + + your_python_package: + level: INFO root: handlers: [rich] @@ -105,14 +92,28 @@ root:
-* `console`: show logs on standard output (typically your terminal screen) without any rich formatting -* `info_file_handler`: write logs of level `INFO` and above to `info.log` -The logging handlers used by default are from `rich`. +```diff + root: + your_python_package: +- level: INFO ++ level: DEBUG +``` + +```{note} +The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. +``` +By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. -The following section illustrates some common examples of how to change your project's logging configuration. +## Advanced logging +In addition to the `rich` handler defined in Kedro's framework, we provide two additional handlers in the template. + +* `console`: show logs on standard output (typically your terminal screen) without any rich formatting +* `info_file_handler`: write logs of level `INFO` and above to `info.log` + +The following section illustrates some common examples of how to change your project's logging configuration. ## How to customise the `rich` handler Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. @@ -125,11 +126,11 @@ When `rich_tracebacks` is set to `True`, the configuration is propagated to [`ri For instance, you can enable the display of local variables inside `logging.yml` to aid with debugging. -```yaml -rich: - class: kedro.logging.RichHandler - rich_tracebacks: True - tracebacks_show_locals: True +```diff + rich: + class: kedro.logging.RichHandler + rich_tracebacks: True ++ tracebacks_show_locals: True ``` A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). From c4780dcd932e3895b72f9f5a9d2b712a1012a078 Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 28 Jun 2023 10:23:43 +0100 Subject: [PATCH 18/23] Revising logging to move page into index Signed-off-by: Jo Stichbury --- docs/source/logging/index.md | 180 ++++++++++++++++++++++++++++++++- docs/source/logging/logging.md | 178 -------------------------------- 2 files changed, 176 insertions(+), 182 deletions(-) delete mode 100644 docs/source/logging/logging.md diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index 2240afde38..a836302a1a 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -1,11 +1,183 @@ # Logging -Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). You can also have more [Advanced Logging](logging.md#advanced-logging) +Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in Kedro's default logging configuration, as described below. +By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. -```{toctree} -:hidden: +# Default logging configuration +Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. -logging +## How to perform logging in your Kedro project +To add logging to your own code (e.g. in a node): + +```python +import logging + +logger = logging.getLogger(__name__) +logger.warning("Issue warning") +logger.info("Send information") +logger.debug("Useful information for debugging") +``` + + + +You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: + +```python +log.error("[bold red blink]Important error message![/]", extra={"markup": True}) +``` + +## How to customise Kedro logging + +To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. + +For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into your terminal: + +```bash +export KEDRO_LOGGING_CONFIG=/conf/logging.yml +``` + +After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path. + +```{note} +If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. +``` + +### How to show DEBUG level messages +To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template.: + +
+Click to expand the logging.yml template + + +```yaml +version: 1 + +disable_existing_loggers: False + +formatters: + simple: + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + +handlers: + console: + class: logging.StreamHandler + level: INFO + formatter: simple + stream: ext://sys.stdout + + info_file_handler: + class: logging.handlers.RotatingFileHandler + level: INFO + formatter: simple + filename: info.log + maxBytes: 10485760 # 10MB + backupCount: 20 + encoding: utf8 + delay: True + + rich: + class: kedro.logging.RichHandler + rich_tracebacks: True + # Advance options for customisation. + # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration + # tracebacks_show_locals: False + +loggers: + kedro: + level: INFO + + your_python_package: + level: INFO + +root: + handlers: [rich] +``` + +
+ + +```diff + root: + your_python_package: +- level: INFO ++ level: DEBUG +``` + +```{note} +The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. ``` + +By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. + +## Advanced logging + +In addition to the `rich` handler defined in Kedro's framework, we provide two additional handlers in the template. + +* `console`: show logs on standard output (typically your terminal screen) without any rich formatting +* `info_file_handler`: write logs of level `INFO` and above to `info.log` + +The following section illustrates some common examples of how to change your project's logging configuration. +## How to customise the `rich` handler + +Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. + +```{note} +If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/logging.yml`. +``` + +When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings. + +For instance, you can enable the display of local variables inside `logging.yml` to aid with debugging. + +```diff + rich: + class: kedro.logging.RichHandler + rich_tracebacks: True ++ tracebacks_show_locals: True +``` + +A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). + +## How to enable file-based logging + +File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behaviour and it's easy to search. However, it does not work well with read-only systems such as [Databricks Repos](https://docs.databricks.com/repos/index.html). + +To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follows: + +```diff + root: +- handlers: [rich] ++ handlers: [rich, info_file_handler] +``` + +By default it only tracks `INFO` level messages, but it can be configured to capture any level of logs. + +## How to use plain console logging + +To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows: + +```diff + root: +- handlers: [rich] ++ handlers: [console] +``` + +## How to enable rich logging in a dumb terminal + +Rich [detects whether your terminal is capable](https://rich.readthedocs.io/en/stable/console.html#terminal-detection) of displaying richly formatted messages. If your terminal is "dumb" then formatting is automatically stripped out so that the logs are just plain text. This is likely to happen if you perform `kedro run` on CI (e.g. GitHub Actions or CircleCI). + +If you find that the default wrapping of the log messages is too narrow but do not wish to switch to using the `console` logger on CI then the simplest way to control the log message wrapping is through altering the `COLUMNS` and `LINES` environment variables. For example: + +```bash +export COLUMNS=120 LINES=25 +``` + +```{note} +You must provide a value for both `COLUMNS` and `LINES` even if you only wish to change the width of the log message. Rich's default values for these variables are `COLUMNS=80` and `LINE=25`. +``` + +## How to enable rich logging in Jupyter + +Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`. diff --git a/docs/source/logging/logging.md b/docs/source/logging/logging.md deleted file mode 100644 index 84cd1c5e87..0000000000 --- a/docs/source/logging/logging.md +++ /dev/null @@ -1,178 +0,0 @@ -By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. - -# Default logging configuration -Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. - -## How to perform logging in your Kedro project -To add logging to your own code (e.g. in a node): - -```python -import logging - -logger = logging.getLogger(__name__) -logger.warning("Issue warning") -logger.info("Send information") -logger.debug("Useful information for debugging") -``` - - - -You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: - -```python -log.error("[bold red blink]Important error message![/]", extra={"markup": True}) -``` - -## How to customise Kedro logging - -To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. - -For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into your terminal: - -```bash -export KEDRO_LOGGING_CONFIG=/conf/logging.yml -``` - -After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path. - -```{note} -If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. -``` - -### How to show DEBUG level messages -To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template.: - -
-Click to expand the `logging.yml` template - - -```yaml -version: 1 - -disable_existing_loggers: False - -formatters: - simple: - format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" - -handlers: - console: - class: logging.StreamHandler - level: INFO - formatter: simple - stream: ext://sys.stdout - - info_file_handler: - class: logging.handlers.RotatingFileHandler - level: INFO - formatter: simple - filename: info.log - maxBytes: 10485760 # 10MB - backupCount: 20 - encoding: utf8 - delay: True - - rich: - class: kedro.logging.RichHandler - rich_tracebacks: True - # Advance options for customisation. - # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration - # tracebacks_show_locals: False - -loggers: - kedro: - level: INFO - - your_python_package: - level: INFO - -root: - handlers: [rich] -``` - -
- - -```diff - root: - your_python_package: -- level: INFO -+ level: DEBUG -``` - -```{note} -The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. -``` - -By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. - -## Advanced logging - -In addition to the `rich` handler defined in Kedro's framework, we provide two additional handlers in the template. - -* `console`: show logs on standard output (typically your terminal screen) without any rich formatting -* `info_file_handler`: write logs of level `INFO` and above to `info.log` - -The following section illustrates some common examples of how to change your project's logging configuration. -## How to customise the `rich` handler - -Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. - -```{note} -If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/logging.yml`. -``` - -When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings. - -For instance, you can enable the display of local variables inside `logging.yml` to aid with debugging. - -```diff - rich: - class: kedro.logging.RichHandler - rich_tracebacks: True -+ tracebacks_show_locals: True -``` - -A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler). - -## How to enable file-based logging - -File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behaviour and it's easy to search. However, it does not work well with read-only systems such as [Databricks Repos](https://docs.databricks.com/repos/index.html). - -To enable file-based logging, add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follows: - -```diff - root: -- handlers: [rich] -+ handlers: [rich, info_file_handler] -``` - -By default it only tracks `INFO` level messages, but it can be configured to capture any level of logs. - -## How to use plain console logging - -To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows: - -```diff - root: -- handlers: [rich] -+ handlers: [console] -``` - -## How to enable rich logging in a dumb terminal - -Rich [detects whether your terminal is capable](https://rich.readthedocs.io/en/stable/console.html#terminal-detection) of displaying richly formatted messages. If your terminal is "dumb" then formatting is automatically stripped out so that the logs are just plain text. This is likely to happen if you perform `kedro run` on CI (e.g. GitHub Actions or CircleCI). - -If you find that the default wrapping of the log messages is too narrow but do not wish to switch to using the `console` logger on CI then the simplest way to control the log message wrapping is through altering the `COLUMNS` and `LINES` environment variables. For example: - -```bash -export COLUMNS=120 LINES=25 -``` - -```{note} -You must provide a value for both `COLUMNS` and `LINES` even if you only wish to change the width of the log message. Rich's default values for these variables are `COLUMNS=80` and `LINE=25`. -``` - -## How to enable rich logging in Jupyter - -Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`. From 4400f0f5f82e14de377b75f88b26807d0d879009 Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 28 Jun 2023 10:32:44 +0100 Subject: [PATCH 19/23] Few minor tweaks Signed-off-by: Jo Stichbury --- docs/source/logging/index.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index a836302a1a..7fcedf1c0c 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -6,7 +6,7 @@ Kedro uses [Python's `logging` library](https://docs.python.org/3/library/loggin By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`. # Default logging configuration -Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions. +Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the [Rich logging handler](https://rich.readthedocs.io) to format messages. We also use the Rich traceback handler to render exceptions. ## How to perform logging in your Kedro project To add logging to your own code (e.g. in a node): @@ -20,9 +20,7 @@ logger.info("Send information") logger.debug("Useful information for debugging") ``` - - -You can take advantage of Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: +You can use Rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) in your logging calls: ```python log.error("[bold red blink]Important error message![/]", extra={"markup": True}) @@ -30,7 +28,7 @@ log.error("[bold red blink]Important error message![/]", extra={"markup": True}) ## How to customise Kedro logging -To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration. Point the variable instead to your project-specific configuration, which we recommend you name `logging.yml` and store inside the project's`conf` folder. +To customise logging in your Kedro project, you need to specify the path to a project-specific logging configuration file. Change the environment variable `KEDRO_LOGGING_CONFIG` to override the default logging configuration. Point the variable instead to your project-specific configuration, which we recommend you store inside the project's`conf` folder, and name `logging.yml`. For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into your terminal: @@ -38,14 +36,14 @@ For example, you can set `KEDRO_LOGGING_CONFIG` by typing the following into you export KEDRO_LOGGING_CONFIG=/conf/logging.yml ``` -After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path. +After setting the environment variable, any subsequent Kedro commands use the logging configuration file at the specified path. ```{note} -If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. +If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. ``` ### How to show DEBUG level messages -To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template.: +To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template:
Click to expand the logging.yml template @@ -109,7 +107,7 @@ root: The name of a logger corresponds to a key in the `loggers` section of the logging configuration file (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information. ``` -By changing the level value to `DEBUG` for the desired logger (e.g., ``), you will start seeing `DEBUG` level messages in the log output. +By changing the level value to `DEBUG` for the desired logger (e.g. ``), you will start seeing `DEBUG` level messages in the log output. ## Advanced logging @@ -119,6 +117,7 @@ In addition to the `rich` handler defined in Kedro's framework, we provide two a * `info_file_handler`: write logs of level `INFO` and above to `info.log` The following section illustrates some common examples of how to change your project's logging configuration. + ## How to customise the `rich` handler Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. From 605b5e15e986e60345c31968d6af222069b1237e Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 28 Jun 2023 10:41:18 +0100 Subject: [PATCH 20/23] Fix broken internal linking Signed-off-by: Jo Stichbury --- docs/source/configuration/configuration_basics.md | 2 +- docs/source/development/set_up_pycharm.md | 2 +- docs/source/tutorial/tutorial_template.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index e2575f1fd6..56af25cf29 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -1,6 +1,6 @@ # Configuration -This section contains detailed information about Kedro project configuration, which you can use to store settings for your project such as [parameters](./parameters.md), [credentials](./credentials.md), the [data catalog](../data/data_catalog.md), and [logging information](../logging/logging.md). +This section contains detailed information about Kedro project configuration, which you can use to store settings for your project such as [parameters](./parameters.md), [credentials](./credentials.md), the [data catalog](../data/data_catalog.md), and [logging information](../logging/index.md). Kedro makes use of a configuration loader to load any project configuration files, and the available configuration loader classes are: diff --git a/docs/source/development/set_up_pycharm.md b/docs/source/development/set_up_pycharm.md index b7a34d5c8b..28d936bf22 100644 --- a/docs/source/development/set_up_pycharm.md +++ b/docs/source/development/set_up_pycharm.md @@ -67,7 +67,7 @@ Edit the new Run configuration as follows: Replace **Script path** with path obtained above and **Working directory** with the path of your project directory and then click **OK**. ```{note} -**Emulate terminal in output console** enables PyCharm to show [rich terminal output](../logging/logging.md). +**Emulate terminal in output console** enables PyCharm to show [rich terminal output](../logging/index.md). ``` To execute the Run configuration, select it from the **Run / Debug Configurations** dropdown in the toolbar (if that toolbar is not visible, you can enable it by going to **View > Toolbar**). Click the green triangle: diff --git a/docs/source/tutorial/tutorial_template.md b/docs/source/tutorial/tutorial_template.md index 99b75cb031..af23f15cfd 100644 --- a/docs/source/tutorial/tutorial_template.md +++ b/docs/source/tutorial/tutorial_template.md @@ -68,7 +68,7 @@ pip install -r src/requirements.txt ## Optional: logging and configuration -You might want to [set up logging](../logging/logging.md) at this stage of the workflow, but we do not use it in this tutorial. +You might want to [set up logging](../logging/index.md) at this stage of the workflow, but we do not use it in this tutorial. You may also want to store credentials such as usernames and passwords if they are needed for specific data sources used by the project. From ad1fe98ca31e382e344f5d3a6c827f65d2af0b2a Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 28 Jun 2023 13:27:04 +0000 Subject: [PATCH 21/23] fix default logging mention Signed-off-by: Nok --- docs/source/logging/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index 7fcedf1c0c..1d876c746c 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -39,7 +39,7 @@ export KEDRO_LOGGING_CONFIG=/conf/logging.yml After setting the environment variable, any subsequent Kedro commands use the logging configuration file at the specified path. ```{note} -If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of Kedro's `default_logging.yml`. +If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will use the [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). ``` ### How to show DEBUG level messages From 78b29797373a01466c6e0e8a87c7a3140a46fcfe Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 29 Jun 2023 13:44:11 +0000 Subject: [PATCH 22/23] add more description to the change needed Signed-off-by: Nok --- docs/source/logging/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index 1d876c746c..96d8e606d7 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -95,7 +95,7 @@ root:
- +You need to change the line: ```diff root: your_python_package: From 16557e6f2397cc82e20910d955070a4f0c1dd495 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 29 Jun 2023 14:26:20 +0000 Subject: [PATCH 23/23] Fix the wrong `diff` to show debug logging Signed-off-by: Nok --- docs/source/logging/index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/source/logging/index.md b/docs/source/logging/index.md index 96d8e606d7..23fb20d26d 100644 --- a/docs/source/logging/index.md +++ b/docs/source/logging/index.md @@ -97,8 +97,11 @@ root: You need to change the line: ```diff - root: - your_python_package: +loggers: + kedro: + level: INFO + + your_python_package: - level: INFO + level: DEBUG ```