input_data pointing to Azure File Share fails

[sebastko]

Problem Description

Downloading folders or files from Azure File Share, as specified in the Task's input_data property, has started failing in the most recent version of Shipyard.

Batch Shipyard Version

3.6.0b1

Steps to Reproduce

• Upload application packages to an Azure File Share
• Use the following in Task's properties to get the app packages:

    input_data:
      azure_storage:
      - storage_account_settings: mystorageaccount
        remote_path: scripts/app
        local_path: $AZ_BATCH_TASK_WORKING_DIR
        is_file_share: true

where "scripts" is a File Share on mystorageaccount and "app" is a folder (containing binaries) on that share.

Expected Results

The scenario works fine up to tag 3.6.0a1:

2018-10-16 00:15:01.581 INFO - blobxfer start time: 2018-10-16 00:15:01.581286+00:00
2018-10-16 00:15:01.616 DEBUG - dest is_dir=True for 1 specs
2018-10-16 00:15:01.617 INFO - downloading blobs/files to local path: /mnt/batch/tasks/workitems/[...]/job-1/datasplit/wd/app
2018-10-16 00:15:01.617 DEBUG - spawning 3 transfer threads
2018-10-16 00:15:01.633 DEBUG - spawning 4 disk threads
2018-10-16 00:15:01.701 DEBUG - 0 files 0.0000 MiB filesize and/or lmt_ge skipped
2018-10-16 00:15:01.701 DEBUG - 2 remote files processed, waiting for download completion of approx. 0.0556 MiB
2018-10-16 00:15:01.828 INFO - MD5: SKIPPED, scripts/app/myapp.tar.gz None <L..R> None
2018-10-16 00:15:01.828 INFO - MD5: SKIPPED, scripts/app/mylib.tar.gz None <L..R> None
2018-10-16 00:15:01.829 INFO - attempting to delete 0 extraneous files
2018-10-16 00:15:01.829 INFO - elapsed download + verify time and throughput of 0.0001 GiB: 0.137 sec, 3.2518 Mbps (0.406 MiB/sec)
2018-10-16 00:15:01.829 INFO - blobxfer end time: 2018-10-16 00:15:01.829806+00:00 (elapsed: 0.249 sec)

Actual Results

2018-10-16 00:11:40.891 INFO - blobxfer start time: 2018-10-16 00:11:40.891901+00:00
2018-10-16 00:11:40.927 DEBUG - dest is_dir=True for 1 specs
2018-10-16 00:11:40.927 INFO - downloading blobs/files to local path: /mnt/batch/tasks/workitems/[...]/job-1/datasplit/wd/app
2018-10-16 00:11:40.927 DEBUG - spawning 3 transfer threads
2018-10-16 00:11:40.953 DEBUG - spawning 4 disk threads
2018-10-16 00:11:40.983 ERROR - file scripts/app not found in storage account [...]
2018-10-16 00:11:40.983 DEBUG - 0 files 0.0000 MiB filesize and/or lmt_ge skipped
2018-10-16 00:11:40.983 DEBUG - 0 remote files processed, waiting for download completion of approx. 0.0000 MiB
2018-10-16 00:11:40.984 INFO - attempting to delete 0 extraneous files
2018-10-16 00:11:40.984 INFO - blobxfer end time: 2018-10-16 00:11:40.984534+00:00 (elapsed: 0.093 sec)

Additonal Comments

If there is any simpler/more reliable way to upload application binaries to use inside a job, please let me know.

[alfpark]

For the Actual Results, are there log lines before the first one that you posted? The latest blobxfer fixed regressions with SAS keys permission scoping of the previous release and there still may be some latent ones with file shares.

[alfpark]

Issue has been identified in blobxfer and the dependency will be updated in the next release: Azure/blobxfer#88