This repository has been archived by the owner on Oct 12, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 49
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Sample - Add a sample for using SAS resource files (#253)
* resubmit sas resource files example * fixed typos and grammar * remove unnecessary github reference
- Loading branch information
Showing
5 changed files
with
104 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Name,Age | ||
Julie,16 | ||
John,19 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Name,Age | ||
Julie,17 | ||
John,20 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# SAS Resource Files | ||
|
||
The following sample shows how to transfer data using secure [SAS blob tokens](https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1). This allows secure transfer between cloud storage from either your local computer or the nodes in the cluster. | ||
|
||
As part of this example you will see how to create a secure write-only SAS and upload files to the cloud. Then create a secure read-only SAS and download those files to the nodes in your cluster. Finally, you will enumerate the files on each node in the cluster and can operate against them however you choose. | ||
|
||
Make sure to replace the storage account you want to use. The the storage account listed in the credentials.json file must be used for this sample to work. | ||
|
||
```R | ||
storageAccountName <- "<YOUR_STORAGE_ACCOUNT>" | ||
``` |
22 changes: 22 additions & 0 deletions
22
samples/sas_resource_files/sas_resource_files_cluster.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
{ | ||
"name": "sas_resource_files", | ||
"vmSize": "Standard_D11_v2", | ||
"maxTasksPerNode": 1, | ||
"poolSize": { | ||
"dedicatedNodes": { | ||
"min": 0, | ||
"max": 0 | ||
}, | ||
"lowPriorityNodes": { | ||
"min": 3, | ||
"max": 3 | ||
}, | ||
"autoscaleFormula": "QUEUE" | ||
}, | ||
"rPackages": { | ||
"cran": [], | ||
"github": [], | ||
"bioconductor": [] | ||
}, | ||
"commandLine": [] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
library(doAzureParallel) | ||
|
||
doAzureParallel::setCredentials("credentials.json") | ||
storageAccountName <- "<YOUR_STORAGE_ACCOUNT>" | ||
inputContainerName <- "datasets" | ||
|
||
# Generate a sas tokens with the createSasToken function | ||
|
||
# Write-only SAS. Will be used for uploading files to storage. | ||
writeSasToken <- rAzureBatch::createSasToken(permission = "w", sr = "c", path = inputContainerName) | ||
|
||
# Read-only SAS. Will be used for downloading files from storage. | ||
readSasToken <- rAzureBatch::createSasToken(permission = "r", sr = "c", path = inputContainerName) | ||
|
||
# Create a Storage container in the Azure Storage account | ||
rAzureBatch::createContainer(inputContainerName) | ||
|
||
# Upload blobs with a write sasToken | ||
rAzureBatch::uploadBlob(inputContainerName, | ||
fileDirectory = "1989.csv", | ||
sasToken = writeSasToken, | ||
accountName = storageAccountName) | ||
|
||
rAzureBatch::uploadBlob(inputContainerName, | ||
fileDirectory = "1990.csv", | ||
sasToken = writeSasToken, | ||
accountName = storageAccountName) | ||
|
||
# Create URL paths with read-only permissions | ||
csvFileUrl1 <- rAzureBatch::createBlobUrl(storageAccount = storageAccountName, | ||
containerName = inputContainerName, | ||
sasToken = readSasToken, | ||
fileName = "1989.csv") | ||
|
||
|
||
csvFileUrl2 <- rAzureBatch::createBlobUrl(storageAccount = storageAccountName, | ||
containerName = inputContainerName, | ||
sasToken = readSasToken, | ||
fileName = "1990.csv") | ||
|
||
# Create a list of files to download to the cluster using read-only permissions | ||
# Place the files in a directory called 'data' | ||
resource_files = list( | ||
rAzureBatch::createResourceFile(url = csvFileUrl1, fileName = "data/1989.csv"), | ||
rAzureBatch::createResourceFile(url = csvFileUrl2, fileName = "data/1990.csv") | ||
) | ||
|
||
# Create the cluster | ||
cluster <- makeCluster("sas_resource_files_cluster.json", resourceFiles = resource_files) | ||
registerDoAzureParallel(cluster) | ||
workers <- getDoParWorkers() | ||
|
||
# Files downloaded to the cluster are placed in a specific directory on each node called 'wd' | ||
# Use the pre-defined environment variable 'AZ_BATCH_NODE_STARTUP_DIR' to find the path to the directory | ||
listFiles <- foreach(i = 1:workers, .combine='rbind') %dopar% { | ||
fileDirectory <- paste0(Sys.getenv("AZ_BATCH_NODE_STARTUP_DIR"), "/wd", "/data") | ||
files <- list.files(fileDirectory) | ||
df = data.frame("node" = i, "files" = files) | ||
return(df) | ||
} | ||
|
||
# List the files downloaded to each node in the cluster | ||
listFiles | ||
|
||
stopCluster(cluster) |