Hooks can slow down execution in some cases

[astrojuanlu]

Description

I'm talking to a user who reported that their hooks are incredibly slow, taking 82 % of the total kedro run time, which amounts to several minutes. I attach the profiling.

CPU_profile_for_Program_kedro_run_env_ai_nodes_multivariate_predict_sales_at_2025_02_20T08_44_27.speedscope.json

Context

Massively slow running times.

Steps to Reproduce

(TBC)

Expected Result

Actual Result

-- If you received an error, place it here.

-- Separate them if you have more than one.

Your Environment

• Kedro version used (pip show kedro or kedro -V): kedro 0.18.4
• Python version used (python -V):
• Operating system and version:

[astrojuanlu]

As far as I interpret it, most of the time is being spent rendering some very large pandas DataFrame inside Pluggy:

I suspect, but cannot prove, that it might be related to #2630, which was introduced by #2596

in other words, Pluggy tracing is producing messages that, despite being invisible to the user, require huge pandas dataframes to be turned into strings.

I could be wrong here though, but hope it serves as a place to start.

[ElenaKhaustova]

@astrojuanlu do you mean this issue?
#4504

[astrojuanlu]

Wow, perfect timing. Yes, that.

[ElenaKhaustova]

Wow, perfect timing. Yes, that.

Then we don't need two tickets, right? So, if you don't mind, I'm going to close this one in favour of the original one.

[astrojuanlu]

Yes 🙏🏼