Upload cylc-run artifact on test fail.

[hjoliver]

@oliver-sanders this just adds an artifact upload section to the bash test config. Hoping it'll allow us to diagnose the broadcast/00-simple test problem. See comments from here: #3933 (comment) The change looks simple but all I did was copy your section from the other file, so you'd best check it.

[UPDATE: now for bash and main functional test actions]

[kinow]

Nice! 🤓 wasn't aware of the upload-artifact action, #TIL

[hjoliver]

Well this test failed here 🎉

But no artifact uploaded 😢

Warning: No files were found with the provided path: cylc-run. No artifacts will be uploaded.

[kinow]

@hjoliver I think path is relative to where the action is running. In the log lines next to the error uploading file in the last run, we have

Test Summary Report
-------------------
tests/functional/broadcast/00-simple.t              (Wstat: 0 Tests: 5 Failed: 4)
  Failed tests:  2-5

I think tests/functional/broadcast/00-simple.t is relative to the working directory of the action. So the cylc-run would be probably somewhere like /root/cylc-run/ (from the error log line suite logs can be found under: /root/cylc-run/cylctb-20201112T101740Z/functional/broadcast/00-simple/log/suite/).

WDYT?

[oliver-sanders]

Ah yes, this functionality hadn't been tested, yet.

Yes, my bad, looks like it's a relative path so ~/cylc-run, /root/cylc-run or possibly ../cylc-run.

[hjoliver]

Run actions/upload-artifact@v2
  with:
    name: Upload cylc-run artifact
    path: /root/cylc-run
    if-no-files-found: warn
Error: EACCES: permission denied, lstat '/root/cylc-run'

[kinow]

Run actions/upload-artifact@v2
  with:
    name: Upload cylc-run artifact
    path: /root/cylc-run
    if-no-files-found: warn
Error: EACCES: permission denied, lstat '/root/cylc-run'

actions/upload-artifact#130

Maybe for now we just cp /root/cylc-flow /elsewhere followed by chmod 0777 /elsewhere? Then upload from /elsewhere instead?

[hjoliver]

actions/upload-artifact#130

[kinow]

Warning: No files were found with the provided path: cylc-run. No artifacts will be uploaded.

😥 this issue is giving some trouble to fix. I bet it will be something really simple, a one-liner fix somewhere in our code base. A typical Friday 🐛 haha

[hjoliver]

Yeah I don't think my cp -r command at the end of the Run step is being executed. If there are multiple commands and one fails, does the stop abort?

[kinow]

Yeah I don't think my cp -r command at the end of the Run step is being executed. If there are multiple commands and one fails, does the stop abort?

I think so. Maybe

tests/functional/broadcast/00-simple.t \
  & cp -r /root/cylc-run cylc-run

?

(ah, just saw your last commit defining the working directory 👍 )

[hjoliver]

Ah, I screwed up the docker exec for cp command, and got this:

Error: No such container: cp
So I know it is being executed; trying again...

[hjoliver]

Still no joy.

Ah, it's probably because the tests run inside a Docker container, but the Upload step doesn't.

[kinow]

Still no joy.

Ah, it's probably because the tests run inside a Docker container, but the Upload step doesn't.

I think you are right. But in that case you can just drop the docker exec.

The container has a volume mounted on /root/cylc-flow. The /root/cylc-flow inside the container, is actually the GitHub action clone of cylc-flow directory.

I think we just need to adjust the permissions, and try to upload it (files created by Docker are created as root). What about this one @hjoliver ?

docker exec -w /root/cylc-flow bash tar -zcvf cylc-run.tar.gz /root/cylc-run

I think that would tar the /root/cylc-run directory inside /root/cylc-flow/cylc-run.tar.gz. Then in the upload you could just tell it to upload cylc-run.tar.gz (I think a single file being uploaded might be faster, but never used that upload artifact action)

[hjoliver]

My latest failure attempts to use docker cp to copy cylc-run out of the container:

   run: |                                                                                                                                                                                                     
          docker exec -w /root/cylc-flow bash \                                                                                                                                                                    
            ./etc/bin/run-functional-tests -v \                                                                                                                                                                    
            tests/functional/broadcast/00-simple.t                                                                                                                                                                 
          docker cp bash:/root/cylc-run .         

But the upload step still doesn't find cylc-run.

I thought that would work 😖 I'll try what you suggested now...

[hjoliver]

But there's no evidence in the log that the docker cp command ran, weirdly.

[kinow]

Yup, no way of copying a file out of the container in Docker (that I am aware of). That's normally done via mounted volumes. That was one of the things that took me a while to get used with Singularity.

[hjoliver]

Yup, no way of copying a file out of the container in Docker (that I am aware of). That's normally done via mounted volumes. That was one of the things that took me a while to get used with Singularity.

No, there is! I tried it locally and it worked.

$ docker cp --help

Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
        docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

Copy files/folders between a container and the local filesystem

Use '-' as the source to read a tar archive from stdin
and extract it to a directory destination in a container.
Use '-' as the destination to stream a tar archive of a
container source to stdout.

Options:
  -a, --archive       Archive mode (copy all uid/gid information)
  -L, --follow-link   Always follow symbol link in SRC_PATH

[hjoliver]

🎉 😂

Copy cylc-run out of container:

Run docker cp bash:/root/cylc-run .
  docker cp bash:/root/cylc-run .
  shell: /bin/bash -e {0}
1s

Upload

Run actions/upload-artifact@v2
With the provided path, there will be 23 file(s) uploaded
Total size of all the files uploaded is 23409 bytes
Finished uploading artifact Upload cylc-run artifact. Reported size is 23409 bytes. There were 0 items that failed to upload
Artifact Upload cylc-run artifact has been successfully uploaded!

[oliver-sanders]

If really stuck you could try sharing the cylc-run directory with the container to avoid having to copy stuff back.

cylc-flow/etc/bin/swarm

Line 248 in bc8c48c

'-v' "$HOME/cylc-run:/root/cylc-run"

[hjoliver]

Not stuck, I just got it working (although your way might be a bit more efficient).

[kinow]

And #TIL about the docker cp. I've always mapped volumes to have access to the files of containers. No idea why I never learned about this command! Thanks @hjoliver !

[hjoliver]

@kinow - tests all passed, and I've verified that artifact upload works now. One review should do on this; if you agree, maybe you could merge this we can start using it to diagnose test failures better.

[hjoliver]

(Follow up improvement might be to use a mapped volume instead of docker cp, but I don't think it matters much).