reinstall + fileinstall: async clash

[oliver-sanders]

Due to a recent change in cylc-flow, reinstallation is now async (note unreleased bug).

Unfortunately, this puts cylc-rose in an awkward situation because of the following call chain:

• cylc-flow - reinstall (async)
• cylc-rose - fileinstall
• rose - JobRunner.run (async)

Because cylc-rose is sync, we can't bridge the gap between the two async islands of code.

Try reinstalling a workflow which uses a rose fileinstallation with cylc-rose installed, bang!

Example

# rose-suite.conf
[file:foo]
source=bar

# bar
whatever

# flow.cylc
whatever

$ cylc reinstall reload --yes
REINSTALLED reload/run1 from /net/home/h06/osanders/cylc-tmp/reload
PluginError: Error in plugin cylc.post_install.rose_opts
RuntimeError: This event loop is already running

Possible Solutions

1 - Propagate async/await

Bridge the gap by propagating async/await right the way from cylc-flow, through cylc-rose into rose.

Problem! If we try to make cylc-rose async (by making the install plugin interfaces async), then we end up making the glbl_cfg async (because the install plugins are called during loading of the global config). This would be prohibitively disruptive as every single glbl_cfg call would need to be wrapped in async which would cascade to the point of making pretty much all of the cylc-flow codebase async.

Dropping plugin support for the global config (not the worst idea, although arguably preprocessors should be plugins) solves the issue, but would require duplicate sync/async parsec implementations, messy.

2 - Push fileinstallation into a separate process

Out of sight out of mind.

If the async part of rose happened in a different process, then there wouldn't be any clash with cylc-flow.

But this means forking another process. Too expensive?

3 - Try to drop back into the event loop

We can access the running event loop via asyncio.get_event_loop(), so we can schedule the file installation to run in this loop.

The problem is that we can't await it which means that we have no way of knowing when it has completed.

You can't poll for completion because sleep is blocking to asyncio. You can't poll from a thread, because of the GIL.

With no way of knowing when fileinstallation is complete, cylc play would follow on from cylc install in a cylc vip command call, potentially whilst file installation is still taking place.

This patch makes the error go away, but it would introduce a subtle new bug

diff --git a/metomi/rose/job_runner.py b/metomi/rose/job_runner.py
index 6c8d1610..6627080e 100644
--- a/metomi/rose/job_runner.py
+++ b/metomi/rose/job_runner.py
@@ -175,19 +175,22 @@ class JobRunner:
                 The maximum number of jobs to run concurrently.
 
         """
-        running = []
         loop = asyncio.get_event_loop()
         loop.set_exception_handler(self.job_processor.handle_event)
-        loop.run_until_complete(
-            asyncio.gather(
-                self._run_jobs(running, job_manager, args, concurrency),
-                self._post_process_jobs(running, job_manager, args),
-            )
+        loop.create_task(
+            self._run(job_manager, *args, concurrency=concurrency)
         )
         dead_jobs = job_manager.get_dead_jobs()
         if dead_jobs:
             raise JobRunnerNotCompletedError(dead_jobs)
 
+    async def _run(self, job_manager, *args, concurrency=6):
+        running = []
+        asyncio.gather(
+            self._run_jobs(running, job_manager, args, concurrency),
+            self._post_process_jobs(running, job_manager, args),
+        )
+
     async def _run_jobs(self, running, job_manager, args, concurrency):
         """Run pending jobs subject to the concurrency limit.
 

But we can shift the awaiting part of the problem back to cylc-flow, icky, but, hopefully functional...

[oliver-sanders]

I'm got a solution along the lines of (3) where we create the async tasks in the Rose code, and await them in the cylc-flow code.

It's slightly icky but it's simple and effective. The main caveat to be aware of is that file installation will happen after the cylc-rose plugin returns, so don't go poking at any files from within the cylc-rose plugin becuase they might not be there yet.