18.09 update #464

abretaud · 2018-12-12T16:40:09Z

Various fixes for the 18.09 compose images
It's still failing currently until galaxyproject/ansible-galaxy-extras#220 is merged

compose/build-orchestration-images.sh

pcm32 · 2018-12-12T16:44:56Z

compose/galaxy-web/Dockerfile

@@ -27,7 +27,7 @@ RUN echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/02apt-speedup && \
    # apt-get update -qq && apt-get upgrade -y && \
    # apt-get update -qq && \
    apt-get install --no-install-recommends -y \
-    nginx-extras nginx-common supervisor autofs libslurm32 libslurmdb32 && \


Although beyond this PR, it would be really great if things like slurm dependencies could be installable via ansible and then that could be enabled or disabled via injection of arguments on build time.

@pcm32 are you using the pre-build images or are you building the images on your own again? Or what is your preferred use-case? We could also define those libs via an ENV and create different web containers during build-time - not sure this is a use-case.

Maybe ourselves building different flavours might be too much trouble to maintain, but I would at least leave the flexibility for the advanced user who knows what is doing. I like the model of the labels that we have, where to add the support for certain things, you need to add that label (ie. k8s,cvmfs, etc). So I would extend that, as man power allows, to other sections (like telegraf, slumr, condor, etc) that are optional requirements depending on how you are using this.

@bgruening I'm probably still building my own because I need to point to a version of ansible galaxy extras that includes this: galaxyproject/ansible-galaxy-extras#213
On my setup, it fails because uswgi process I think points to that galaxy.yml but that file doesn't exists at that point... the test not passed seems to be unrelated to the logic added on that PR.

@bgruening in your compose setup works without my PR ansible-galaxy-extras due to https://github.com/bgruening/docker-galaxy-stable/blob/dev/compose/docker-compose.yml#L327 but the current PR (464) is actually removing that line, so the compose setup will need the solution proposed in https://github.com/bgruening/docker-galaxy-stable/blob/dev/compose/docker-compose.yml#L327 I think....

I removed the mount of galaxy.yml because when you launch docker-compose up, the file doesn't exist yet, so Docker decides to create an empty directory named "galaxy.yml". This makes it impossible later to copy the good galaxy.yml file during startup. So galaxy fails to start because it can't read galaxy.yml.
Mounting a non existing file will never work as it will be considered a directory by docker

pcm32 · 2018-12-12T16:46:10Z

compose/galaxy-postgres/init-galaxy-db.sql.in

@@ -8237,7 +8126,7 @@ GalaxyTools	lib/tool_shed/galaxy_install/migrate	1
 --

 COPY migrate_version (repository_id, repository_path, version) FROM stdin;
-Galaxy	lib/galaxy/model/migrate	144


My bad, I must have been testing a few things on 19.01 when I pushed this... sorry.

I have been thinking for a while on incorporating

https://github.com/bgruening/docker-galaxy-stable/blob/dev/compose/dumpsql.sh#L9-L36

inside the build-orchestration-images script, right after the init container creation (using that init container created). Then this whole file would be generated on the fly when building the containers, based on the version that it was used of Galaxy for those containers (so fairly failure-proof). Would you consider adding that to your PR @abretaud ?

Yep, that would be cool. I'm not sure how to do it properly though: do you mean that at the end of build-orchestration-images we should:

run a docker-compose with an empty postgresql database

run the install_db.sh in galaxy-web

run the dumpsql.sh script

(re)build the galaxy-postgres image

Not sure I'll have time to write this now, maybe it could be another PR?

Yes, don't worry, I'll PR it soon.

compose/galaxy-htcondor/Dockerfile

abretaud · 2018-12-13T10:51:58Z

There are a few errors left, I'm looking into them.
For https://travis-ci.org/bgruening/docker-galaxy-stable/jobs/467400898, the problem is that nginx is not listening on port 443 as it fails to load the config when reading the "upload_store" instruction in /etc/nginx/nginx.conf.
So the problem is that the nginx-upload-module is not installed, and probably not in any ubuntu 18.04 package. I've found https://launchpad.net/~galaxyproject/+archive/ubuntu/nginx, but it doesn't seem to be up-to-date.
Any help on this?

bgruening · 2018-12-13T11:32:29Z

Let's remove the upload module, it is not needed in recent Galaxy versions afaik.

…ode is not available yet at build time)

…ted later by galaxy-init). It was mounted as an empty dir, so galaxy refused to run. The /usr/bin/startup script will copy the good file from $GALAXY_ROOT a bit later during galaxy-web startup, after galaxy-init has created it (https://github.com/galaxyproject/ansible-galaxy-extras/blob/18.09/templates/startup.sh.j2#L42).

abretaud · 2018-12-14T16:34:02Z

Still a few errors, I'll try to continue next week
Is it intended that the client is not built in compose? I guess it should now in 18.09?

pcm32 · 2018-12-14T21:55:38Z

mmm... weird, I remember adding some commits around a month ago or so to have the client built in these containers. But it needs to be built, it was only when I had it built that these images worked for me (I was a bit ahead of 18.09 though, but the same should apply).

pcm32 · 2018-12-14T22:00:38Z

Yes, look, we are building the client in the init (and that is why we kept the --skip-build-client on web):

5915886

and then this gets copied to the web container on runtime. Due to this I would also say that you don't need to install nodejs/yarn either on web (and the whole setup works for me without installing those on web AFAIK).

abretaud · 2018-12-18T10:40:58Z

@pcm32 for some reason, the static/scripts dir is missing in the galaxy-web container when I run it locally, I'll investigate.
For nodejs, we still need it galaxy-web to run the GIE proxy (got a supervisor error without it when it's not there)

abretaud · 2018-12-19T11:27:56Z

I think it's ok now, the last failing test was probably a temporary error from the toolshed. Rerunning it should solve it (I don't have the perm to trigger it)
The k8s test is still failing, but I guess it's more or less expected

bgruening

Super cool! I owe you a couple of beers :)

Just two small comments inline.

bgruening · 2018-12-20T22:51:57Z

compose/dumpsql.sh

@@ -1,6 +1,6 @@
 #!/bin/bash

-TAG=18.09
+TAG=v18.09


should that be 18.09?

I tested, but no, it failed (=empty dump) without the v, because all images are tagged with this v

mh, ok I might remove this at some point. We should standardize it ...

bgruening · 2018-12-20T22:52:25Z

compose/galaxy-htcondor-base/Dockerfile

@@ -18,3 +18,8 @@ RUN wget https://dl.influxdata.com/telegraf/releases/telegraf-1.5.0_linux_amd64.
    rm telegraf-1.5.0_linux_amd64.tar.gz

 ADD telegraf.conf /etc/telegraf/telegraf.conf
+
+
+RUN touch /var/log/condor/StartLog /var/log/condor/StarterLog /var/log/condor/CollectorLog /var/log/condor/NegotiatorLog && \


This can be removed now I think.

Just tested it: I think it's still needed as this image is not based on galaxy-base. I get errors at startup without this line

bgruening · 2018-12-21T09:17:41Z

Thanks a lot!!!!

18.09 update

abretaud added 17 commits December 11, 2018 13:42

update galaxyextras 18.09

7d834ec

need munge

9998785

fix slurm install

beb8440

fix requirements install

1be3274

try hack for ansible rule

2efd525

fix for the hack

c194c51

hack for the hack

1481493

supervisor stuff

5bd1ba5

no need to mkdir

9efc04c

slurm startup fix

2312f4f

For 18.09 uses migration 142, not 144

b533b20

condor fixes

d456e53

avoid harmless error message

716b859

slurm install

8f42276

fix missing ansible tag

57dcde9

kube fix?

933b13a

not needed

05be0b7

pcm32 reviewed Dec 12, 2018

View reviewed changes

compose/build-orchestration-images.sh Show resolved Hide resolved

pcm32 reviewed Dec 12, 2018

View reviewed changes

bgruening reviewed Dec 12, 2018

View reviewed changes

compose/galaxy-htcondor/Dockerfile Outdated Show resolved Hide resolved

abretaud added 2 commits December 13, 2018 09:40

move to htcondor-base

209b057

need 18.09 branch of ansible-galaxy-extras

edbef97

abretaud mentioned this pull request Dec 13, 2018

Add missing imports galaxy-iuc/parsec#22

Merged

abretaud mentioned this pull request Dec 13, 2018

[18.09] small fixes galaxyproject/ansible-galaxy-extras#221

Merged

abretaud added 3 commits December 13, 2018 15:55

need nodejs installed (but can't use the ansible task as the galaxy c…

b50f1d1

…ode is not available yet at build time)

trigger rebuild

95a5d7c

abretaud added 2 commits December 14, 2018 15:33

update bioblend testing

52f92d4

fix bioblend test env

e938b86

abretaud and others added 13 commits December 18, 2018 11:48

fix image tag

7e53dc0

dump fix

5516786

reenable client building

72866ed

fix slurm_singularity test

c21cfcc

Need to delay setting of GALAXY_CONFIG_TOOL_CONFIG_FILE

7ec499e

create needed dirs/files for condor

27ee4ab

update tag

2c0be52

latest kubernetes needs go >=1.11.1

40b74ac

fix path problem (not sure if sudo is really needed)

7beba07

longer timeout

bb3083d

Longer timeout

83e2f24

fix ignored BIOBLEND_TEST_JOB_TIMEOUT

1cf3215

wait more!?

76185aa

abretaud changed the title ~~[WIP] 18.09 update~~ 18.09 update Dec 19, 2018

update kompose

e5b586f

bgruening approved these changes Dec 20, 2018

View reviewed changes

bgruening merged commit 3eabd62 into bgruening:dev Dec 21, 2018

AndreasSko pushed a commit to AndreasSko/docker-galaxy-stable that referenced this pull request May 22, 2020

Merge pull request bgruening#464 from abretaud/1804

5feeca7

18.09 update

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

18.09 update #464

18.09 update #464

abretaud commented Dec 12, 2018

pcm32 Dec 12, 2018

bgruening Dec 12, 2018

pcm32 Dec 13, 2018

pcm32 Dec 14, 2018

pcm32 Dec 14, 2018

abretaud Dec 18, 2018

pcm32 Dec 12, 2018

pcm32 Dec 12, 2018

abretaud Dec 13, 2018

pcm32 Dec 13, 2018

abretaud commented Dec 13, 2018

bgruening commented Dec 13, 2018

abretaud commented Dec 14, 2018

pcm32 commented Dec 14, 2018

pcm32 commented Dec 14, 2018 •

edited

Loading

abretaud commented Dec 18, 2018

abretaud commented Dec 19, 2018

bgruening left a comment

bgruening Dec 20, 2018

abretaud Dec 21, 2018

bgruening Dec 21, 2018

bgruening Dec 20, 2018

abretaud Dec 21, 2018

bgruening commented Dec 21, 2018

18.09 update #464

18.09 update #464

Conversation

abretaud commented Dec 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abretaud commented Dec 13, 2018

bgruening commented Dec 13, 2018

abretaud commented Dec 14, 2018

pcm32 commented Dec 14, 2018

pcm32 commented Dec 14, 2018 • edited Loading

abretaud commented Dec 18, 2018

abretaud commented Dec 19, 2018

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening commented Dec 21, 2018

pcm32 commented Dec 14, 2018 •

edited

Loading