kedro-org · noklam · Jun 29, 2023 · Jun 9, 2023 · Jun 9, 2023 · Jun 9, 2023
@@ -6,6 +6,7 @@
 
 ## Breaking changes to the API
 
+
 ## Migration guide from Kedro 0.18.* to 0.19.*
 
 # Upcoming Release 0.18.11
@@ -15,7 +16,7 @@
 ## Bug fixes and other changes
 
 ## Breaking changes to the API
-
+* Logging is decoupled from `ConfigLoader`, use `KEDRO_LOGGING_CONFIG` to configure logging.
 ## Upcoming deprecations for Kedro 0.19.0
 
 # Release 0.18.10

diff --git a/docs/source/deployment/databricks/databricks_workspace.md b/docs/source/deployment/databricks/databricks_workspace.md
@@ -2,9 +2,6 @@
 
 This tutorial uses the [PySpark Iris Kedro Starter](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris) to illustrate how to bootstrap a Kedro project using Spark and deploy it to a [Databricks cluster on AWS](https://databricks.com/aws).
 
-```{note}
-If you are using [Databricks Repos](https://docs.databricks.com/repos/index.html) to run a Kedro project then you should [disable file-based logging](../../logging/logging.md#disable-file-based-logging). This prevents Kedro from attempting to write to the read-only file system.
-```
 
 ```{note}
 If you are a Kedro contributor looking for information on deploying a custom build of Kedro to Databricks, see the [development guide](../../contribution/development_for_databricks.md).

@@ -1,9 +1,9 @@
 # Logging
 
 
-Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in two places:
-1. [Default configuration built into the Kedro framework](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml). This cannot be altered.
-2. Your project-side logging configuration. Every project generated using Kedro's CLI `kedro new` command includes a file `conf/base/logging.yml`. You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md).
+Kedro uses [Python's `logging` library](https://docs.python.org/3/library/logging.html). Configuration is provided as a dictionary according to the [Python logging configuration schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) in [Kedro's default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml).
+
+You can alter this configuration and provide different configurations for different run environment according to the [standard Kedro mechanism for handling configuration](../configuration/configuration_basics.md).
 
 ```{note}
 Providing project-side logging configuration is entirely optional. You can delete the `conf/base/logging.yml` file and Kedro will run using the framework's built in configuration.

@@ -1,54 +1,124 @@
+# Default logging configuration
+Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the Rich logging handler to format messages. We also use the Rich traceback handler to render exceptions.
 
-# Default framework-side logging configuration
+By default, Python only shows logging messages at level WARNING and above. Kedro's logging configuration specifies that INFO level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a kedro run.
 
-Kedro's [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml) defines a handler called `rich` that uses the [Rich logging handler](https://rich.readthedocs.io/en/stable/logging.html) to format messages. We also use the [Rich traceback handler](https://rich.readthedocs.io/en/stable/traceback.html) to render exceptions.
+## Perform logging in your project
+To add logging to your own code (e.g. in a node), you are advised to do as follows:
+
+```python
+import logging
 
-By default, Python only shows logging messages at level `WARNING` and above. Kedro's logging configuration specifies that `INFO` level messages from Kedro should also be emitted. This makes it easier to track the progress of your pipeline when you perform a `kedro run`.
+log = logging.getLogger(__name__)
+log.warning("Issue warning")
+log.info("Send information")
+log.debug("Useful information for debugging")
+```
 
-## Project-side logging configuration
+```{note}
+The name of a logger corresponds to a key in the `loggers`  section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information.
+```
 
-In addition to the `rich` handler defined in Kedro's framework, the [project-side `conf/base/logging.yml`](https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml) defines two further logging handlers:
-* `console`: show logs on standard output (typically your terminal screen) without any rich formatting
-* `info_file_handler`: write logs of level `INFO` and above to `info.log`
+You can take advantage of rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls:
+```python
+log.error("[bold red blink]Important error message![/]", extra={"markup": True})
+```
 
-The logging handlers that are actually used by default are `rich` and `info_file_handler`.
+### Show DEBUG level messages
+To see your `DEBUG` level message, you can change the level of log messages that you wish to see in `logging.yml`.
 
-The project-side logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`).
+```yml
+loggers:
+  kedro:
+    level: INFO
 
-We now give some common examples of how you might like to change your project's logging configuration.
+  your_python_pacakge:
+    level: DEBUG  # Change this to DEBUG
+```
 
-### Using `KEDRO_LOGGING_CONFIG` environment variable
+By changing the level value to `DEBUG` for the desired logger (e.g., <your_python_package>), you will start seeing `DEBUG` level messages in the log output.
 
-`KEDRO_LOGGING_CONFIG` is an optional environment variable that you can use to specify the path of your logging configuration file, overriding the default Kedro's `default_logging.yml`.
+## Customise Logging
 
-To use this environment variable, set it to the path of your desired logging configuration file before running any Kedro commands. For example, if you have a logging configuration file located at `/path/to/logging.yml`, you can set `KEDRO_LOGGING_CONFIG` as follows:
+### Using `KEDRO_LOGGING_CONFIG` environment variable
+In order to customise logging, you need to specify the path of your logging configuration file via setting the environment variable `KEDRO_LOGGING_CONFIG`, which overrides the default Kedro's `default_logging.yml`. We recommend to put your `logging.yml` inside the `conf` folder. For example, you can set `KEDRO_LOGGING_CONFIG` as follows:
 
 ```bash
-export KEDRO_LOGGING_CONFIG=/path/to/logging.yml
+export KEDRO_LOGGING_CONFIG=<project_root>/conf/logging.yml
 ```
 
 After setting the environment variable, any subsequent Kedro commands will use the logging configuration file at the specified path.
 
 ```{note}
 If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will default to using the logging configuration file at the project's default location of  Kedro's `default_logging.yml`.
 ```
-### Disable file-based logging
 
-You might sometimes need to disable file-based logging, e.g. if you are running Kedro on a read-only file system such as [Databricks Repos](https://docs.databricks.com/repos/index.html). The simplest way to do this is to delete your `conf/base/logging.yml` file. With no project-side logging configuration specified, Kedro uses the default framework-side logging configuration, which does not include any file-based handlers.
 
-Alternatively, if you would like to keep other configuration in `conf/base/logging.yml` and just disable file-based logging, then you can remove the file-based handlers from the `root` logger as follows:
-```diff
- root:
--  handlers: [console, info_file_handler]
-+  handlers: [console]
+# Advance Logging
+In addition to the `rich` handler defined in Kedro's framework, we provide a `logging.yml` template that you can use to start with.
+
+<details>
+<summary><b>Click to expand the `logging.yml` template</b></summary>
+```yml
+version: 1
+
+disable_existing_loggers: False
+
+formatters:
+  simple:
+    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+
+handlers:
+  console:
+    class: logging.StreamHandler
+    level: INFO
+    formatter: simple
+    stream: ext://sys.stdout
+
+  info_file_handler:
+    class: logging.handlers.RotatingFileHandler
+    level: INFO
+    formatter: simple
+    filename: info.log
+    maxBytes: 10485760 # 10MB
+    backupCount: 20
+    encoding: utf8
+    delay: True
+
+  rich:
+    class: kedro.logging.RichHandler
+    rich_tracebacks: True
+    # Advance options for customisation.
+    # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration
+    # tracebacks_show_locals: False
+
+loggers:
+  kedro:
+    level: INFO
+
+  {{ cookiecutter.python_package }}:
+    level: INFO
+
+root:
+  handlers: [rich]
 ```
+</details>
+
+* `console`: show logs on standard output (typically your terminal screen) without any rich formatting
+* `info_file_handler`: write logs of level `INFO` and above to `info.log`
+
+The logging handlers that are actually used by default is `rich`.
+
+The default logging configuration also ensures that [logs emitted from your project's logger](#perform-logging-in-your-project) should be shown if they are `INFO` level or above (as opposed to the Python default of `WARNING`).
 
-### Customise the `rich` Handler
+Now, let's provide some common examples of how you might like to change your project's logging configuration.
 
-Kedro's `kedro.extras.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`.
+## Customise the `rich` Handler
+
+Kedro's `kedro.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`.
 
 ```{note}
-If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/base/logging.yml`.
+If you want to disable `rich`'s tracebacks, you must set `KEDRO_LOGGING_CONFIG` to point to your local config i.e. `conf/logging.yml`.
 ```
 
 When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings.
@@ -57,25 +127,37 @@ For instance, you can enable the display of local variables inside `logging.yml`
 
 ```yaml
 rich:
-  class: kedro.extras.logging.RichHandler
+  class: kedro.logging.RichHandler
   rich_tracebacks: True
   tracebacks_show_locals: True
 ```
 
 A comprehensive list of available options can be found in the [RichHandler documentation](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler).
 
+## Enable file-based logging
+
+File-based logging in Python projects aids troubleshooting and debugging. It offers better visibility into application's behavior and it's easy to search. However, it does not work well with read-only system such as [Databricks Repos](https://docs.databricks.com/repos/index.html).
+
+To enable file-based logging,  add `info_file_handler` in your `root` logger as follows in your `conf/logging.yml` as follow:
+```diff
+ root:
+-  handlers: [rich]
++  handlers: [rich, info_file_handler]
+```
+
+By default it only tracks `INFO` level message, but it can be configured to capture any level of logs.
 
-### Use plain console logging
+## Use plain console logging
 
 To use plain rather than rich logging, swap the `rich` handler for the `console` one as follows:
 
 ```diff
  root:
--  handlers: [rich, info_file_handler]
-+  handlers: [console, info_file_handler]
+-  handlers: [rich]
++  handlers: [console]
 ```
 
-### Rich logging in a dumb terminal
+## Rich logging in a dumb terminal
 
 Rich [detects whether your terminal is capable](https://rich.readthedocs.io/en/stable/console.html#terminal-detection) of displaying richly formatted messages. If your terminal is "dumb" then formatting is automatically stripped out so that the logs are just plain text. This is likely to happen if you perform `kedro run` on CI (e.g. GitHub Actions or CircleCI).
 
@@ -89,27 +171,6 @@ export COLUMNS=120 LINES=25
 You must provide a value for both `COLUMNS` and `LINES` even if you only wish to change the width of the log message. Rich's default values for these variables are `COLUMNS=80` and `LINE=25`.
 ```
 
-### Rich logging in Jupyter
+## Rich logging in Jupyter
 
 Rich also formats the logs in JupyterLab and Jupyter Notebook. The size of the output console does not adapt to your window but can be controlled through the `JUPYTER_COLUMNS` and `JUPYTER_LINES` environment variables. The default values (115 and 100 respectively) should be suitable for most users, but if you require a different output console size then you should alter the values of `JUPYTER_COLUMNS` and `JUPYTER_LINES`.
-
-## Perform logging in your project
-
-To perform logging in your own code (e.g. in a node), you are advised to do as follows:
-
-```python
-import logging
-
-log = logging.getLogger(__name__)
-log.warning("Issue warning")
-log.info("Send information")
-```
-
-```{note}
-The name of a logger corresponds to a key in the `loggers`  section in `logging.yml` (e.g. `kedro`). See [Python's logging documentation](https://docs.python.org/3/library/logging.html#logger-objects) for more information.
-```
-
-You can take advantage of rich's [console markup](https://rich.readthedocs.io/en/stable/markup.html) when enabled in your logging calls:
-```python
-log.error("[bold red blink]Important error message![/]", extra={"markup": True})
-```