testing against latest conda version

[bgruening]

In order to test if the latest conda version is actually working for us right now, I started to test all IUC tools against latest miniconda3.

 % /home/bag/miniconda3/bin/conda --version                         
conda 4.2.9

With this command I installed all deps:
planemo conda_install --conda_prefix /home/bag/miniconda3 tools/*

And tested with:

 % planemo test --conda_dependency_resolution --conda_prefix /home/bag/miniconda3 tools/<tool>

Please try to test your tools as well and extend the list. It would be nice to get some confidence that the newer version is actually working for us right now.

A list of already tested repos:

• abricate (failed partially, does not look like conda problems)
• abyss
• art
• bam_to_scidx
• bcftools
• bedtools
• biom_convert
• bioperl
• blastxml_to_gapped_gff3
• bwameth
• collection_column_join
• deseq2
• extract_genomic_dna
• goseq
• fasta_nucleotide_color_plot
• jbrowse (ok with Jbrowse: finish python3 conversion #1076)
• kobas
• nugen_nudup
• prinseq
• prokka
• raxml
• rgdifferentialcount (does not look like conda problems)
• seqtk
• sickle
• stacks
• stringtie
• trinity (same remark as trinotate + dep conflict problem)
• trinotate (impacted by option to use soft links instead of hard links conda/conda#3308, pass if conda_prefix is on same volume as job files)
• ucsc_tools
• vegan
• vsearch
• weather_app
• xpath

Not yet tested:

• cwpair2
• datamash
• data_source_iris_tcga
• dexseq
• differential_count_models
• ebi_tools
• featurecounts
• gatk2
• gemini
• genetrack
• gff3_rebase
• hisat2
• hmmer3
• htseq
• htseq_count
• khmer
• kraken_taxonomy_report
• length_and_gc_content
• macs2
• meme
• mothur
• ncbi_entrez_eutils
• ngsutils
• pear
• pe_histogram
• picrust
• pilon
• progressivemauve
• quast
• repmatch_gff3
• resize_coordinate_window
• rglasso
• rgrnastar
• scikit-bio
• sra-tools
• star_fusion
• tag_pileup_frequency
• taxonomy_krona_chart
• tool_factory_2
• transdecoder
• transtermhp

ping @galaxyproject/iuc

[lecorguille]

About Trinotate, I guess this PR will fix this soft links Trinotate/Trinotate#14

Meanwhile, I will test so of them ASAP :)

[abretaud]

@lecorguille I think/fear the PR will not be enough (ie RealBin can be wrong)

Just tested Trinity, it looks ok, except the same remark as Trinotate + this error (not quite sure how to investigate it now):

UnsatisfiableError: The following specifications were found to be in conflict:

• bioconductor-annotate 1.50.0 0
• bioconductor-edger -> r 3.3.1*
  Use "conda info " to see the dependencies for each package.

galaxy.jobs.runners ERROR 2016-12-14 15:12:51,066 (6) Failure preparing job
Traceback (most recent call last):
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/jobs/runners/init.py", line 170, in prepare_job
job_wrapper.prepare()
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/jobs/init.py", line 901, in prepare
self.dependency_shell_commands = self.tool.build_dependency_shell_commands(job_directory=self.working_directory)
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/tools/init.py", line 1291, in build_dependency_shell_commands
metadata=metadata,
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/tools/deps/init.py", line 98, in dependency_shell_commands
**kwds )
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/tools/deps/init.py", line 116, in find_dep
dependency = resolver.resolve( name, version, type, **kwds )
File "/tmp/tmplQp3El/galaxy-dev/lib/galaxy/tools/deps/resolvers/conda.py", line 203, in resolve
raise Exception("Conda dependency seemingly installed but failed to build job environment.")
Exception: Conda dependency seemingly installed but failed to build job environment.

[mvdbeek]

UnsatisfiableError: The following specifications were found to be in conflict:

bioconductor-annotate 1.50.0 0
bioconductor-edger -> r 3.3.1*
Use "conda info " to see the dependencies for each package.

that's probably due to galaxyproject/galaxy#3299

[abretaud]

Oh, that looks tricky :/

[lecorguille]

I know there are not within the IUC tools repository.
But I can validate for our Workflow4Metabolomics tools: essentially R libraries (cran and bioconductor)

[bgruening]

Any help is welcome and as more tools we can test as more confident we are to switch to the latest conda version.

[lecorguille]

Sorry, I should have write I have (my English)

But I have validate for our Workflow4Metabolomics tools: essentially R libraries (cran and bioconductor)

[bgruening]

So it is working for the entire flavor? This is good to know! Thanks a lot!

[lecorguille]

ERRATUM:
It was a mistake (a bit lost in all my miniconda version).
The test actually fails using the last version of Conda.
I summarized that in this issue: workflow4metabolomics/tools-metabolomics#26

I'm really disappointed to have to stop all Pull Requests and developments for a bunch of our tools. 😞

I can send a mug with a beautiful logo drawn by me to any of you who can help me on that!

[nsoranzo]

@lecorguille Does this work with miniconda2 3.19.3?

[lecorguille]

Yes, it works, all green on our ressources (laptop, HTC)

But I run on this issues: galaxyproject/planemo#604

• 2 of my deps come with R 3.2.2 and 2 with R 3.3.1. Actually, it's not a big deal, they don't induce bugs
• 1 of my deps takes ages to be installed (bioconductor-camera). TravisCI reaches the timeout. My guess is it's some internal conflicts. It really better with the last version of conda 😕

So I couldn't wait to test with conda-4.

I test the different version between 3.19.3 and 4.2.9. It solve the timeout with bioconductor-camera but I still got the deps 3.2.2 and finally the DependencyException: Conda dependency seemingly installed but failed to build job environment.

[lecorguille]

@nsoranzo ERRATUM: it work now with the 3.19.3 under TravisCI. Installing bioconductor-camera still take time but less than previously.
Sorry to have spoil this thread

Now focus on conda-4.2.9

[lecorguille]

All good for me, now!
Thanks @bgruening and John and @mvdbeek

[bgruening]

Yeah! Thanks!

[abretaud]

I guess we can close this now