Document how to properly use Logfire in Kedro

[astrojuanlu]

Description

I was trying to use Logfire in Kedro, and it mostly worked. However, the results aren't optimal.

Context

Steps to Reproduce

1. Create a new spaceflights project
2. Add logfire.configure() in settings.py (for lack of a better place)
3. Add a logfire handler to the logging. For example:

--- spaceflights-pandas-viz2/conf/logging.yml	2024-07-02 14:47:47
+++ spaceflights-pandas-viz/conf/logging.yml	2024-07-02 14:47:10
@@ -32,12 +32,15 @@
     # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration
     # tracebacks_show_locals: False
 
+  logfire:
+    class: logfire.LogfireLoggingHandler
+
 loggers:
   kedro:
     level: INFO
 
 root:
-  handlers: [rich, info_file_handler]
+  handlers: [rich, info_file_handler, logfire]

Expected Result

The logs should display flawlessly on the Logfire web UI.

Actual Result

Note

The shortcodes shouldn't leak any longer, see below

With this configuration, the shortcodes for rich coloring "leak" onto the logs:

(notice the [dark_orange] shortcodes)

If I disable the rich handler in logging.yml, the CLI output changes but the result on Logfire UI is the same.

And finally, I tried rich.uninstall() but it turns out logfire depends on rich 🙃

---

On top of that, I observe this big fat warning:

$ kedro run
[07/02/24 14:51:01] INFO     Using `conf/logging.yml` as logging configuration. You can change    __init__.py:249
                             this by setting the KEDRO_LOGGING_CONFIG environment variable                       
                             accordingly.                                                                        
                    WARNING  /Users/juan_cano/Projects/QuantumBlackLabs/workshop-from-zero-to-mlo warnings.py:110
                             ps/.venv/lib/python3.11/site-packages/kedro/framework/cli/catalog.py                
                             :15: LogfireNotConfiguredWarning: No logs or spans will be created                  
                             until `logfire.configure()` has been called. Set the environment                    
                             variable LOGFIRE_IGNORE_NO_CONFIG=1 or add ignore_no_config=true in                 
                             pyproject.toml to suppress this warning.                                            
                               from kedro.framework.project import pipelines, settings                           
                                                                                                                 
Logfire project URL: https://logfire.pydantic.dev/astrojuanlu/test-simple
[07/02/24 14:51:03] INFO     Kedro project spaceflights-pandas-viz                                 session.py:324

Apart from that LOGFIRE_IGNORE_NO_CONFIG, is there a better way to achieve this?

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.19.6
• Python version used (python -V): 3.11.9
• Operating system and version: macOS Sonoma 14.5

[astrojuanlu]

The shortcodes are because of #3626, fixed by #3682 (thanks for the heads-up @noklam)

My question on LogfireNotConfiguredWarning still stands

[astrojuanlu]

FTR Logfire is leaving beta soon https://pydantic.notion.site/Pydantic-Logfire-Pricing-f719ee768d10457f970f78eedbff976e

[noklam]

If I disable the rich handler in logging.yml, the CLI output changes but the result on Logfire UI is the same.

How did you disable rich handler? If I remove rich from the list of handlers, it works correctly for me.

  # rich:
  #   class: kedro.logging.RichHandler
  #   rich_tracebacks: True
  #   # Advance options for customisation.
  #   # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration
  #   # tracebacks_show_locals: False

[astrojuanlu]

https://noklam.github.io/blog/posts/kedro_logfire/2024-12-01-kedro-with-logfire.html I guess the spike is done? 🙃 @noklam do you have any more thoughts you want to add?

[noklam]

@astrojuanlu I didn't spend too much time on logfire specifically (feel free to close this ). Tangentially, I did some experiments with Otel and struggled a bit with Kedro Hooks. It could be me not know Otel well enough. But basically what I want to do is to have a span tracking how long a node/pipeline run. The code is not the best but leave it here for reference.

The code is weird because most of the time Otel use a context manager with xxx to define a span, but with hook the context is splitted across multiple hook so I have to do this manually. It's likely that this won't work in distributed run (ParallelRunner etc).

(kedro) Nok_Lam_Chan@ spaceflights_pandas % cat settings.py
"""Project settings. There is no need to edit this file unless you want to change values
from the Kedro defaults. For further information, including these default values, see
https://docs.kedro.org/en/stable/kedro_project_setup/settings.html."""

# Instantiated project hooks.
# For example, after creating a hooks.py and defining a ProjectHooks class there, do
# from spaceflights_pandas.hooks import ProjectHooks

# Hooks are executed in a Last-In-First-Out (LIFO) order.
import time

from kedro.framework.hooks import hook_impl
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.context import attach, detach, get_current
# Acquire a tracer
tracer = trace.get_tracer("kedro_app.tracer")
# Set up tracing with service name 'kedro_app'
resource = Resource.create({"service.name": "kedro_node_trace"})
trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter to send traces to the collector
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))


class OtelHook:
    @hook_impl
    def before_node_run(self):
      self.node_span = tracer.start_as_current_span("trace")
      self.node_span.__enter__()
      print("Send something to Otel before node run")
      time.sleep(1)
      print("sleep for 1 sec")

    @hook_impl
    def after_node_run(self):
      with tracer.start_as_current_span("after node run trace") as span:
        print("Send something to Otel after node run")
        time.sleep(0.5)
        print("sleep for 0.5 sec")
      self.node_span.__exit__(None, None, None) # call __exit__ for @contextmanager


HOOKS = (OtelHook(),)