Issue when sourcing `shipyard.envlist` #314

stity · 2019-09-16T13:56:59Z

Problem Description

When sourcing the .shipyard.envlist, I got an the following errors :

/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 4: --name: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^SHLVL=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^HOSTNAME=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^PATH=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^PWD=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^HOME=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^USER=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 16: -c: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 20: source /mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist && run_freesurfer.sh : No such file or directory
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 29: -c: command not found

Batch Shipyard Version

We recently upgraded from batch-shipyard 3.5.0 to batch-shipyard 3.8.1

Steps to Reproduce

Submit a job with batch-shipyard 3.8.1 with a task sourcing the shipyard.envlist

Expected Results

No errorwhen sourcing the shipyard.envlist

Actual Results

Error when sourcing the shipyard.envlist

Redacted Configuration

batch_shipyard:
  storage_account_settings: shipyardstorage

global_resources:
  additional_registries:
    docker:
    - micliqs.azurecr.io

Additional Logs

Content of the shipard.envlist on the node

AZ_BATCH_TASK_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer
SHIPYARD_ENV_FILE=.shipyard.envlist
NB_CORE=2
SHIPYARD_RUNTIME_CMD_OPTS=--rm --name freesurfer_CADASIL_1001_Baseline_3DT1-Freesurfer --entrypoint /bin/bash -w $AZ_BATCH_TASK_WORKING_DIR -v $AZ_BATCH_NODE_ROOT_DIR:$AZ_BATCH_NODE_ROOT_DIR --env-file .shipyard.envlist
AZ_BATCH_NODE_STARTUP_DIR=/mnt/batch/tasks/startup
DOCKER_LOGIN_PASSWORD==redacted
AZ_BATCH_CERTIFICATES_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/certs
AZ_BATCH_ACCOUNT_URL=https://****.westeurope.batch.azure.com/
AZ_BATCH_TASK_WORKING_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd
AZ_BATCH_NODE_SHARED_DIR=/mnt/batch/tasks/shared
SHIPYARD_ENV_EXCLUDE=^_=|^SHLVL=|^HOSTNAME=|^PATH=|^PWD=|^HOME=|^USER=
AZ_BATCH_JOB_PREP_WORKING_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/jobpreparation/wd
DOCKER_LOGIN_SERVER=****.azurecr.io
AZ_BATCH_TASK_USER=_azbatch
AZ_BATCH_NODE_ROOT_DIR=/mnt/batch/tasks
SHIPYARD_SYSTEM_PROLOGUE_CMD=/bin/bash -c 'set -e; set -o pipefail; pushd /mnt/batch/tasks/startup/wd; ./registry_login.sh; popd; set -f; $AZ_BATCH_NODE_STARTUP_DIR/wd/shipyard_blobxfer.sh "1.9.1,i,False,redacted,$AZ_BATCH_TASK_WORKING_DIR,--include "*.dcm" "; set +f; wait'
AZ_BATCH_JOB_ID=freesurfer_CADASIL_1001_Baseline_3DT1
AZ_BATCH_NODE_IS_DEDICATED=true
INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/cadasil/3dd5e7dd-29d4-4699-97d2-c73b2bcdf434
SHIPYARD_USER_CMD=-c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist && run_freesurfer.sh "
AZ_BATCH_JOB_PREP_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/jobpreparation
SUBJECT_NAME=3dd5e7dd-29d4-4699-97d2-c73b2bcdf434
SHIPYARD_CONTAINER_IMAGE_NAME=****.azurecr.io/freesurfer:1.4.1
AZ_BATCH_NODE_MOUNTS_DIR=/mnt/batch/tasks/fsmounts
AZ_BATCH_NODE_ID=tvm-605534940_8-20190916t100009z
SHIPYARD_RUNTIME=docker
AZ_BATCH_POOL_ID=iqsfreesurfer
AZ_BATCH_TASK_ID=Freesurfer
SHIPYARD_SYSTEM_EPILOGUE_CMD=/bin/bash -c 'set -e; set -o pipefail; set -f; $AZ_BATCH_NODE_STARTUP_DIR/wd/shipyard_blobxfer.sh "1.9.1,e,False,redacted,$AZ_BATCH_TASK_WORKING_DIR/3dd5e7dd-29d4-4699-97d2-c73b2bcdf434,--include "*.tar.gz" ,tasksuccess"; set +f; wait'
DOCKER_LOGIN_USERNAME=***
AZ_BATCH_ACCOUNT_NAME=***
SHIPYARD_RUNTIME_CMD=run
AZ_BATCH_TASK_USER_IDENTITY=PoolAdmin
OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR

Additonal Comments

Thank you for your help

The text was updated successfully, but these errors were encountered:

alfpark · 2019-09-16T14:51:51Z

The envlist file is an internal implementation detail. Why do you need to source it when it is "sourced" by the container runtime?

f-roche · 2019-09-16T15:05:18Z

At the time we were using 3.5.0, we were having issues with using environment variables using Shipyard variables (like $AZ_BATCH_TASK_WORKING_DIR) so we followed your recommendation in the doc:

Note that environment variables are not expanded and are passed as-is. You will need to source the environment file $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist in a shell within the Docker command or entrypoint if you want any environment variables to be expanded.

Basically we need the Shipyard variables to be evaluated in the environment variables. Do you have any workaround for this purpose?

alfpark · 2019-09-16T15:18:50Z

I believe starting with 3.8.0, that limitation does not apply due to how the non-native task execution works now. Can you please try your command without sourcing?

Also, notabene, 3.8.2 was released recently; if you're upgrading, it's recommended to jump to that version.

f-roche · 2019-09-16T16:12:29Z

Following your recommendations when using 3.8.2 and removing sourcing $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist , I am getting this error:

So I guess the $AZ_BATCH_TASK_WORKING_DIR variable is not replaced by the actual value in the shell.

alfpark · 2019-09-16T16:19:55Z

Are you using a native or non-native pool?

I cannot repro this error in my local env var expansion test.

job_specifications:
  - id: test
    tasks:
      - environment_variables:
          exptest: $AZ_BATCH_TASK_WORKING_DIR/hello/world
        docker_image: busybox
        command: /bin/sh -c "echo $exptest"

This returns:

/mnt/resource/batch/tasks/workitems/test/job-1/task-00000/wd/hello/world

Also, it's recommended to rotate your credentials (Docker login password, storage accounts) referenced in the envlist output above. I redacted and deleted the history, but they have been leaked.

f-roche · 2019-09-16T16:27:06Z

We are using non-native pool, should we set the attribute to true when creating the pool?
What is the difference between native and non-native?
Thanks

alfpark · 2019-09-16T16:29:40Z

It should not matter in this case, but I wanted to eliminate a potential possible side-effect. In the case of non-native, it should work as above.

The differences are explained here: https://github.com/Azure/batch-shipyard/blob/master/docs/97-faq.md#what-is-native-under-pool-platform_image-and-custom_image

f-roche · 2019-09-17T14:14:32Z

The issue is that we are using Shipyard environment variables (like $AZ_BATCH_TASK_WORKING_DIR) in a shell script so they are not replaced by actual value by the shell first.

In your example your are writing the environment variable in a shell command echo $exptest which is replaced by echo $AZ_BATCH_TASK_WORKING_DIR/hello/world and the shell is interpreting the Shipyard variable. But if you use this variable in a script it won't work since the shell won't replace it first.

alfpark · 2019-09-17T15:35:15Z

All values in the .shipyard_envlist will be in your docker container process context. You can easily test this by simply running /bin/bash -c "env" as your command.

$AZ_BATCH_TASK_WORKING_DIR is not set by Shipyard. That is set by Azure Batch when the process is forked. What you described is precisely what should happen in the example I provided. The shell is what typically does the expansion, or your program would do it (you would need to explicitly do it yourself) if you are not invoking a shell.

If you still believe that you need to source the envlist, then you can simply strip the SHIPYARD_* environment variables out of the file and export the rest:

export $(grep -v "^SHIPYARD_" .shipyard.envlist | xargs)

- Environment variables are already sourced. Further expansion of environment variables should be performed by shell invocation or typical conventions by the invoking program (#314)

f-roche · 2019-09-18T07:49:04Z

Thanks for your help, we decided to change it like this at the end:

previous

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
      INPUT_DICOM_DIR: $AZ_BATCH_TASK_WORKING_DIR/foo/bar
      OUTPUT_DIR: $AZ_BATCH_TASK_WORKING_DIR
    entrypoint: /bin/bash
    command: -c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist &&
                 run.sh {options}"

now

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
    entrypoint: /bin/bash
    command: -c "export INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/foo/bar &&
                 export OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR &&
                 run.sh {options}"

We are using the shell to do the expansion before assigning environment variables.

0zzzhhui · 2022-07-08T02:50:58Z

Thanks for your help, we decided to change it like this at the end:

previous

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
      INPUT_DICOM_DIR: $AZ_BATCH_TASK_WORKING_DIR/foo/bar
      OUTPUT_DIR: $AZ_BATCH_TASK_WORKING_DIR
    entrypoint: /bin/bash
    command: -c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist &&
                 run.sh {options}"

now

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
    entrypoint: /bin/bash
    command: -c "export INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/foo/bar &&
                 export OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR &&
                 run.sh {options}"

We are using the shell to do the expansion before assigning environment variables.

I am a newer for develop batch shipyard, can you tell me where the {version} is defined?
I need to define a docker image variable in my job config yaml file.

alfpark added the question label Sep 16, 2019

alfpark added a commit that referenced this issue Sep 17, 2019

Update jobs doc regarding envlist

e252f4b

- Environment variables are already sourced. Further expansion of environment variables should be performed by shell invocation or typical conventions by the invoking program (#314)

alfpark closed this as completed Sep 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue when sourcing `shipyard.envlist` #314

Issue when sourcing `shipyard.envlist` #314

stity commented Sep 16, 2019 •

edited by alfpark

Loading

alfpark commented Sep 16, 2019

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019 •

edited

Loading

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019

f-roche commented Sep 17, 2019

alfpark commented Sep 17, 2019 •

edited

Loading

f-roche commented Sep 18, 2019

0zzzhhui commented Jul 8, 2022

Issue when sourcing shipyard.envlist #314

Issue when sourcing shipyard.envlist #314

Comments

stity commented Sep 16, 2019 • edited by alfpark Loading

Problem Description

Batch Shipyard Version

Steps to Reproduce

Expected Results

Actual Results

Redacted Configuration

Additional Logs

Additonal Comments

alfpark commented Sep 16, 2019

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019 • edited Loading

f-roche commented Sep 16, 2019

alfpark commented Sep 16, 2019

f-roche commented Sep 17, 2019

alfpark commented Sep 17, 2019 • edited Loading

f-roche commented Sep 18, 2019

0zzzhhui commented Jul 8, 2022

Issue when sourcing `shipyard.envlist` #314

Issue when sourcing `shipyard.envlist` #314

stity commented Sep 16, 2019 •

edited by alfpark

Loading

alfpark commented Sep 16, 2019 •

edited

Loading

alfpark commented Sep 17, 2019 •

edited

Loading