SequentialRunner might fail with non thread-safe code

[astrojuanlu]

Description

Since Kedro 0.19.11, SequentialRunner was refactored and is known to occasionally fail or produce unexpected behavior with code that is not thread-safe. See #4353 for more details.

Existing issues related to this:

• Kedro-mlflow with kedro 0.19.11 produces multiple runs Galileo-Galilei/kedro-mlflow#624
• Ugly crash when using MatplotlibWriter in Kedro 0.19.11 kedro-plugins#1012

This is technically a breaking change, and we are trying to gauge user impact. If this is a problem for you, please leave a comment below.

[astrojuanlu]

@merelcht I think we should treat this as a breaking change and try to address it or revert it ahead of the next release, since it's causing disruption that's often difficult to spot and holding users back from upgrading to 0.19.11.

[astrojuanlu]

Another way I've seen this manifest is that kedro run doesn't immediately react to SIGINT. Randomly found it when trying a hook that did time.sleep(10).

[jakepenzak]

We noticed this issue for our team when running our pipeline in databricks, where it manifested as a failure of the spark session to be accessed in certain nodes/imported modules. Admittedly, I don't fully understand the exact mechanics behind this failure (outside of spark session not being fully thread safe?), but I brought it up in a slack convo and was able to link it to this specific issue.

Here is the excerpt for those not on slack:

Hello! I am running into issues with Kedro 0.19.11 release while running pipelines in databricks. Specifically, I am running into an error where an imported python module for a node is unable to find active SparkSession via SparkSession.getActiveSession(). Our pipeline is comprised entirely of Ibis.TableDataset datasets & I/O with pyspark backend. What is throwing me is that other nodes use the pyspark connection and are able to perform operations properly across the spark session, but fails on this single node when leveraging an imported module that it is unable to find the spark session. This issue is not present in Kedro 0.19.10. My best guess is that it has something to do with the updated code in kedro/runner/sequential_runner.py using ThreadPoolExecutor and possible scoping issues? Apologies on the somewhat scattered explanation, there is quite a bit I don't fully understand here, so appreciate any help or guidance. Lmk if I can provide any additional info as well.