Add commands to create training materials #861

bebatut · 2018-08-08T15:46:05Z

Hi,

This PR adds new commands to planemo (as discussed during GCCBOSC: galaxyproject/training-material#941):

training_init: a command to create new training material for the GTN material given the template there:
1. It creates a topic if not present (with metadata filled)
2. It creates a tutorial if not present (with all needed files)
3. It generates the skeleton of a tutorial if a workflow is given and add also the workflow
4. It creates the data_library.yaml file by parsing the Zenodo API and add the link in the tutorial
5. It updates the metadata if needed
training_generate_tuto_from_wf to generate the tutorial.md of an existing tutorial from a workflow
training_fill_data_library to fill the data_library.yaml file by parsing the Zenodo API of an existing tutorial

I tested it by generating several tutorial skeletons. @abretaud tested it too.

I added tests to cover most of the functions (I have some doubts about the tests of the commands).
I am not sure about the code quality, particularly the training.py script if not so straightforward.
I hope I could get your help there 😄

ping @shiltemann, @jmchilton

Bérénice

abretaud · 2018-08-09T13:21:24Z

Oops, I get this exception now:

Traceback (most recent call last):
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/bin/planemo", line 11, in <module>
    load_entry_point('planemo==0.55.0.dev0', 'console_scripts', 'planemo')()
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/cli.py", line 195, in handle_blended_options
    return f(*args, **kwds)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/commands/cmd_training_init.py", line 19, in cli
    training.init(ctx, kwds)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 855, in init
    create_tutorial(kwds, tuto_dir, topic_dir, tuto_template_dir, ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 824, in create_tutorial
    create_tutorial_from_workflow(kwds, z_file_links, tuto_dir, ctx)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 770, in create_tutorial_from_workflow
    body += get_handson_box(step, wf['steps'], tools)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 747, in get_handson_box
    paramlist = get_param_desc(step_params, step_inputs, tp_desc, 1, steps, should_be_there=True)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 731, in get_param_desc
    paramlist += format_param_desc(step_param, step_inputs, tp_d, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 704, in format_param_desc
    paramlist += format_section_param_desc(step_params, step_inputs, tp_desc, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 613, in format_section_param_desc
    sub_param_desc = get_param_desc(params, inputs, get_tool_input(tp_desc), level+1, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 731, in get_param_desc
    paramlist += format_param_desc(step_param, step_inputs, tp_d, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 706, in format_param_desc
    paramlist += format_conditional_param_desc(step_params, step_inputs, tp_desc, level, wf_steps)
  File "/home/abretaud/miniconda3/envs/galaxy_training_material/lib/python3.6/site-packages/planemo/training.py", line 628, in format_conditional_param_desc
    params[test_param['name']],
TypeError: string indices must be integers

bebatut · 2018-08-09T14:14:42Z

😟
Can you give me your workflow?

abretaud · 2018-08-09T14:42:14Z

It's this one: https://github.com/abretaud/training-material/blob/d5c75ade7ef436b6fc93dd40cf5fa90e3a939dbf/topics/genome-annotation/tutorials/annotation-with-maker/workflows/init_workflow.ga

bebatut · 2018-08-16T15:50:16Z

@abretaud it should be fixed. can you test? thanks

abretaud · 2018-08-20T12:58:30Z

@bebatut I confirm! thanks

abretaud · 2018-08-20T15:28:54Z

I've just re-checked the tutorial.md generated with the latest code on the maker workflow:

some select boxes are still absent from the tutorial: e.g. "Organism type" for Maker tool
the same for a multiple select box: "Select fasta outputs" for gffread
The rest looks good :)

jmchilton · 2018-08-21T14:06:04Z

Some of the tests just don't pass for me:

    n = 'save_db'
    tp_desc = tools[step['name']][n]
    step_param = training.get_lower_params(step_params, n)
    print(tp_desc)
    print(n)
    print(step_params)
    print(step_param)
    paramlist = training.format_param_desc(
        step_param,
        step_inputs,
        tp_desc,
        0,
        wf['steps'],
        force_default=False)
    assert '`Yes`' in paramlist, paramlist

That parameter appears to be false not true for instance. Does that work for you locally - does it require Galaxy modifications?

abretaud · 2018-08-23T12:55:36Z

planemo/training.py

+> ### {{ '{%' }} icon hands_on {{ '%}' }} Hands-on: Data upload
+>
+> 1. Import the following files from [Zenodo]({{ zenodo_link }}) or from a data
+>    library named `TODO` if available (ask your instructor)


This should probably be the same as L147?

abretaud · 2018-08-23T13:00:57Z

project_templates/training/tutorials/tutorial1/data-library.yaml

@@ -0,0 +1,7 @@
+---


Is this file still needed? It looks its format changed

abretaud · 2018-08-23T13:01:06Z

planemo/training.py

+>    > ### {{ '{%' }} icon tip {{ '%}' }} Tip: Importing data from a data library
+>    >
+>    > * Go into "Shared data" (top panel) then "Data libraries"
+>    > * Click on "Training data" and then "{{ topic_title }}"


Is it only topic_title or topic_title/tutorial_title?

abretaud · 2018-08-23T13:08:54Z

planemo/training.py

+        wf, tools = get_wf_tools_from_running_galaxy(kwds)
+    else:
+        wf, tools = get_wf_tool_from_local_galaxy(kwds, kwds["workflow"], ctx)
+    save_to_yaml(tools, 'tools.yaml')


Hum, don't know if it's intended but now I get a tools.yaml at the root of the gtn dir (but it doesn't look the same as the tutorial tools.yaml)

bebatut · 2018-08-29T16:14:48Z

I updated to fit with the new metadata definition

@jmchilton The test should be fixed now
@abretaud I fixed some of your comments. I need to test again your workflow to investigate the issue with the boxes... Next week

bebatut · 2018-09-07T12:33:23Z

@abretaud I checked your comments. If the value of a parameter is set to the default value in the tool description, the corresponding parameter description does not appear (to avoid all parameters to be described, specially for big tool). It explain why "Organism type" for Maker tool does not appear
Does it make sense?

abretaud · 2018-09-07T12:35:42Z

@bebatut ah yes, it makes sense, I thought it was not the default value

bebatut · 2018-09-07T15:41:24Z

I restructured the code and split it into classes to try to make it easier to maintain.
I extracted the doc for it.
I think it is ready for review. ping @jmchilton, @shiltemann, @erasche

mblue9 · 2018-09-10T06:35:55Z

Hi @bebatut !

I'm trying training_generate_from_wf out at the moment and getting an error TypeError: string indices must be integers. This is the workflow I'm using (I just changed the file ending to .txt from .ga to add it here)
Galaxy-Workflow-limma-voom_tutorial_fastqs_to_counts.txt

Btw I'm using python 2.7 with this but I see python 3.6 in @abretaud's error above, should I be using python 3 or it doesn't matter?

bebatut · 2018-09-10T13:28:41Z

@mblue9 I could not reproduce your error (with Python 2.7 or 3.6) 😟
Did you use the latest version from this PR? What command did you use?

jmchilton · 2018-09-10T18:23:07Z

That error happens in the Travis tests as well https://travis-ci.org/galaxyproject/planemo/jobs/425797918.

======================================================================
ERROR: Test :func:`planemo.training.tutorial.tutorial.create_tutorial`.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/galaxyproject/planemo/.tox/py27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/galaxyproject/planemo/tests/test_training_tutorial.py", line 432, in test_tutorial_create_tutorial
    tuto.create_tutorial(CTX)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 463, in create_tutorial
    self.create_hands_on_tutorial(ctx)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 444, in create_hands_on_tutorial
    self.body = get_hands_on_boxes_from_local_galaxy(self.training.kwds, self.init_wf_fp, ctx)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 592, in get_hands_on_boxes_from_local_galaxy
    tuto_body = format_wf_steps(wf, config.gi)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tutorial.py", line 574, in format_wf_steps
    tool_inp = ToolInput(inp, wf_param_values, steps, 1, should_be_there=True)
  File "/home/travis/build/galaxyproject/planemo/planemo/training/tool_input.py", line 48, in __init__
    self.wf_param_values = self.wf_param_values[self.name]
TypeError: string indices must be integers
-------------------- >> begin captured stdout << ----------

mblue9 · 2018-09-10T23:38:38Z

@bebatut and @jmchilton thanks for taking a look! Yes I used the version from this PR and the command I used was: planemo training_generate_from_wf --topic_name transcriptomics --tutorial_name "limma-voom_fastqs_to_counts" --workflow ~/training/pmac/galaxy/limma-voom/limma-voom_fastqs_to_counts/Galaxy-Workflow-limma-voom_tutorial_fastqs_to_counts.ga --galaxy_root ~/galaxy/galaxy/

(The training_init command seemed to work)

@bebatut does it work for you with my workflow? If so do you think you could send me the tutorial.md and I could keep going with that? as I'm running this tutorial in a couple of weeks. Otherwise I'll just manually create it.

bebatut · 2018-09-11T09:21:23Z

@mblue9 It works for me with your workflow: the tutorial.md (with changed extension)
Does it work if you do not specify the --galaxy_root? And with the training-init with the workflow?

bebatut · 2018-09-11T09:33:21Z

@jmchilton All the tests implying to start a Galaxy instance (though the call of get_hands_on_boxes_from_local_galaxy or in which I started a Galaxy instance) and then getting the workflow and the tools from this Galaxy instancne seems to fail on Travis (but locally for me):

Test :func:`planemo.training.generate_tuto_from_wf`. ... ERROR
Test :func:`planemo.training.tutorial.format_wf_steps`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_local_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_running_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_local_galaxy`. ... ERROR
Test :func:`planemo.training.tutorial.get_hands_on_boxes_from_running_galaxy`. ... ERROR

Are you aware of any similar issue? I will investigate how the workflow and the extracted tools from these Galaxy instance look like (empty or not).

mblue9 · 2018-09-11T10:51:13Z

@bebatut thanks for the file!!! That auto-generation is so cool 😎 🙌and going to save me a chunk of time (and tedium) 😄!!!

I tried training_generate_from_wf without --galaxy_root and I get the same error. I had run training-init without --galaxy_root.

jmchilton · 2018-09-11T21:41:08Z

Fixed some linting issues and I added test decorators so the tests requiring Galaxy don't slow the "quick" version of the test suite. All the tests will still run with the full suite for both Python 2 and Python 3.

I couldn't love this anymore - it is absolutely amazing.

The wide scope and stellar quality of the new training material commands, library functionality, and tests clearly warrant this. See #861. This is both to say thanks for the great contribution and done in hopes of encouraging continued use of Planemo for this awesome functionality.

bebatut mentioned this pull request Aug 9, 2018

Add yaml file with correspondance between Zenodo ext and Galaxy datatype galaxyproject/training-material#951

Merged

abretaud reviewed Aug 23, 2018

View reviewed changes

jmchilton and others added 17 commits August 29, 2018 11:23

Template to command to launch a workflow for training...

16c6c0f

Add command to create training topic/tutorial

69a9f7a

Add tutorial skeleton creation from workflow

5c30316

Add parameter description in the tutorial skeleton

065702a

Create data_library.yaml file from Zenodo

e0f43da

Extract workflow and tool also from running Galaxy instance

f318da3

Add command to fill a data library of an existing tutorial

2c6c6bd

Remove template tutorial from metadata

bbf9473

Fix tutorial top metadata

d803f32

Copy or extract workflow into a file

5b5d4b7

Fix flake8 linting and parameter description

58f5d35

Add template option and directory creation if not there

206bab9

Add tests for training init command

d9368a1

Add tests for training fille data library command

185e070

Add tests for training and fix training code

daba767

Add command to generate skeleton of existing tutorial

07b8fe3

Formatting of command testing

64ac652

bebatut added 5 commits August 29, 2018 11:23

Fix failing test

1052267

Fix function docstrings

83d2c7d

Fix parameters with extra quotes and sub inputs for conditionals

0415b76

Add test for none or empty parameters

cf02ea8

Update metadata description, remove templates and fix tests

1d902b9

bebatut force-pushed the training_init branch from 55f6d47 to 1d902b9 Compare August 29, 2018 16:12

bebatut added 2 commits September 7, 2018 17:35

Restructuration of the code with classes

6713909

Update the documentation

de98fbb

bebatut force-pushed the training_init branch 2 times, most recently from c385d2d to ff390ff Compare September 11, 2018 14:21

Fix test of workflow formatting and test_get_wf_param_values comparison

45cd23a

bebatut force-pushed the training_init branch from ff390ff to 45cd23a Compare September 11, 2018 15:24

Fix linting, annotate training tests requiring Galaxy.

f383233

jmchilton merged commit fef52a2 into master Sep 11, 2018

martenson deleted the training_init branch November 7, 2018 20:45

martenson restored the training_init branch November 7, 2018 20:46

martenson deleted the training_init branch November 7, 2018 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add commands to create training materials #861

Add commands to create training materials #861

bebatut commented Aug 8, 2018 •

edited

Loading

abretaud commented Aug 9, 2018

bebatut commented Aug 9, 2018

abretaud commented Aug 9, 2018

bebatut commented Aug 16, 2018

abretaud commented Aug 20, 2018

abretaud commented Aug 20, 2018

jmchilton commented Aug 21, 2018

abretaud Aug 23, 2018

abretaud Aug 23, 2018

abretaud Aug 23, 2018

abretaud Aug 23, 2018

bebatut commented Aug 29, 2018

bebatut commented Sep 7, 2018

abretaud commented Sep 7, 2018

bebatut commented Sep 7, 2018

mblue9 commented Sep 10, 2018

bebatut commented Sep 10, 2018

jmchilton commented Sep 10, 2018

mblue9 commented Sep 10, 2018

bebatut commented Sep 11, 2018

bebatut commented Sep 11, 2018

mblue9 commented Sep 11, 2018

jmchilton commented Sep 11, 2018

Add commands to create training materials #861

Add commands to create training materials #861

Conversation

bebatut commented Aug 8, 2018 • edited Loading

abretaud commented Aug 9, 2018

bebatut commented Aug 9, 2018

abretaud commented Aug 9, 2018

bebatut commented Aug 16, 2018

abretaud commented Aug 20, 2018

abretaud commented Aug 20, 2018

jmchilton commented Aug 21, 2018

abretaud Aug 23, 2018

Choose a reason for hiding this comment

abretaud Aug 23, 2018

Choose a reason for hiding this comment

abretaud Aug 23, 2018

Choose a reason for hiding this comment

abretaud Aug 23, 2018

Choose a reason for hiding this comment

bebatut commented Aug 29, 2018

bebatut commented Sep 7, 2018

abretaud commented Sep 7, 2018

bebatut commented Sep 7, 2018

mblue9 commented Sep 10, 2018

bebatut commented Sep 10, 2018

jmchilton commented Sep 10, 2018

mblue9 commented Sep 10, 2018

bebatut commented Sep 11, 2018

bebatut commented Sep 11, 2018

mblue9 commented Sep 11, 2018

jmchilton commented Sep 11, 2018

bebatut commented Aug 8, 2018 •

edited

Loading