Use key vault task in CI yaml (#1037) (#1166)

Currently our integration tests expect a certificate to be installed on the test machine. The certificate gives us access to a key vault containing the secrets we need for the tests. The certificate itself is also in the key vault, so we have to give the pipeline access to get the certificate. All this was done early in the project, before the key vault task existed in Azure Pipelines.

Using the Azure Pipelines Key Vault task allows us to remove key vault credentials from the pipeline. It also simplifies the build machine setup: we no longer need to install the certificate, and on Linux we don't need to install things like Python, the `az` CLI, and PowerShell that our scripts require to get/install the certificate. Instead we use the task to list exactly the secrets we need, and they are available to the pipeline. For script tasks, we map the secrets into the environment so they don't get passed as arguments and potentially exposed.

Other changes:
- Use the .NET Core installer task to pin the version of .NET Core for builds
- Remove "Install dependencies" and "install test dependencies" tasks since they're no longer needed. For Linux I had to add a script task to install libsnappy, which installPrereqs.sh was doing. It's needed for the rocksdb integration tests. I also had to add a task to install the protocol head certificate, which several integration tests need. This could arguably be checked in as a PS1 script rather than inline script in the YAML file, but this works for now.
- Remove tasks that copy and publish artifacts for the end-to-end tests (IotEdgeQuickstart and friends). No other pipelines curently consume it right now (they rely on the Build Images pipeline). We can add these back if/when they're needed.
- Move the Windows job onto a hosted agent
- Update SecretsHelper to look in the environment first for the needed secrets. When the integration tests are run in the CI pipeline, the KeyVaultHelper is never invoked. Elsewhere, things continue to work like they always have.
- Our build scripts expect dotnet.exe to live in a certain path, but the .NET Core Installer task puts it somewhere funny, and then prepends that location to the PATH. So I updated the build/test scripts to (1) look for an environment variable, then (2) look on the PATH, then (3) look in some default locations.

builds/ci/dotnet.yaml

@@ -5,8 +5,9 @@ trigger:
       - master
 pr: none
 variables:
-  test.filter: "Category=Integration&Category!=Stress"
-  build.configuration: "Release"
+  test.filter: Category=Integration&Category!=Stress
+  build.configuration: Release
+  dotnet.version: 2.1.302
 jobs:
   - job: linux
     displayName: Linux
@@ -15,101 +16,98 @@ jobs:
       name: Hosted Ubuntu 1604
       vmImage: ubuntu-16.04
     variables:
-      TestEnvironment: linux
-      NetCorePackageUri: https://download.microsoft.com/download/4/0/9/40920432-3302-47a8-b13c-bbc4848ad114/dotnet-sdk-2.1.302-linux-x64.tar.gz
+      testEnvironment: linux
     steps:
-      - script: scripts/linux/installPrereqs.sh -u $(NetCorePackageUri)
-        name: install_dependencies
-        displayName: Install dependencies
-      - script: scripts/linux/installKvPrereqs_Ubuntu.sh
-        name: install_test_dependencies
-        displayName: 'Install test dependencies'
-      - bash: |
-          az login --service-principal --username $(kv.sp.user) --tenant $(kv.sp.tenant) --password $(kv.sp.password)
-          scripts/linux/downloadAndInstallCert.sh -c IoTEdgeTestCert -v $(kv.name)
-          scripts/linux/downloadAndInstallCert.sh -c IotHubMqttHeadCert -v $(kv.name) -s 'Root'
-        name: install_certs
-        displayName: 'Install test certificate'
-      - script: scripts/linux/buildBranch.sh -c "$BUILD_CONFIGURATION" --no-rocksdb-bin
-        name: build
-        displayName: 'Build'
-      - script: sudo -E bash -c './scripts/linux/runTests.sh "--filter $TEST_FILTER" "$BUILD_CONFIGURATION"'
-        name: test
+      - task: AzureKeyVault@1
+        displayName: Get secrets
+        inputs:
+          azureSubscription: $(azure.subscription)
+          keyVaultName: $(kv.name)
+          secretsFilter: >-
+            DummySecret1,
+            EdgeCapableDevice2,
+            EventHubConnStr2,
+            IotDevice2ConnStr2,
+            IotDevice3ConnStr2,
+            IotHubConnStr2,
+            IotHubMqttHeadCert
+      - task: DotNetCoreInstaller@0
+        displayName: Install .NET Core
+        inputs:
+          packageType: sdk
+          version: $(dotnet.version)
+      - script: |
+          sudo apt-get update
+          sudo apt-get install -y libsnappy1v5
+        displayName: Install libsnappy1v5 # Needed for rocksdb integration tests
+      - pwsh: |
+          $store = [System.Security.Cryptography.X509Certificates.X509Store]::new('Root', 'CurrentUser')
+          $store.Open('ReadWrite')
+          $bytes = [System.Convert]::FromBase64String($env:CERT_VALUE)
+          $cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($bytes)
+          $store.Add($cert)
+        displayName: Install protocol head certificate # Needed for integration tests
+        env:
+          CERT_VALUE: $(IotHubMqttHeadCert)
+      - script: scripts/linux/buildBranch.sh -c $(Build.Configuration) --no-rocksdb-bin
+        displayName: Build
+      - script: sudo -E bash -c './scripts/linux/runTests.sh "--filter $(test.filter)" $(Build.Configuration)'
         displayName: Test
+        env:
+          DummySecret1: $(DummySecret1)
+          EdgeCapableDevice2: $(EdgeCapableDevice2)
+          EventHubConnStr2: $(EventHubConnStr2)
+          IotDevice2ConnStr2: $(IotDevice2ConnStr2)
+          IotDevice3ConnStr2: $(IotDevice3ConnStr2)
+          IotHubConnStr2: $(IotHubConnStr2)
+          IotHubMqttHeadCert: $(IotHubMqttHeadCert)
       - task: PublishTestResults@1
-        displayName: 'Publish test results'
+        displayName: Publish test results
         inputs:
           testRunner: VSTest
           testResultsFiles: '**/TestResults/result.trx'
         condition: succeededOrFailed()
-      - task: PublishBuildArtifacts@1
-        displayName: 'Publish Artifact: e2e deployment files'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish/e2e_deployment_files'
-          ArtifactName: 'core-linux/e2e_deployment_files'
-      - task: PublishBuildArtifacts@1
-        displayName: 'Publish Artifact: e2e test files'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish/e2e_test_files'
-          ArtifactName: 'core-linux/e2e_test_files'
-      - task: PublishBuildArtifacts@1
-        displayName: 'Publish Artifact: IotEdgeQuickStart'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish/IotEdgeQuickstart.linux-x64.tar.gz'
-          ArtifactName: 'core-linux'
   - job: windows
     displayName: Windows
     timeoutInMinutes: 120
     pool:
-      name: $(windows.pool.name)
-      demands:
-        - azureps
-        - Run-CI -equals true
+      vmImage: windows-2019
     variables:
-      NetCorePackageUri: https://download.microsoft.com/download/4/0/9/40920432-3302-47a8-b13c-bbc4848ad114/dotnet-sdk-2.1.302-win-x64.zip
-      TestEnvironment: windows
+      testEnvironment: windows
     steps:
-      - task: AzurePowerShell@2
-        displayName: 'Install test certificate'
+      - task: AzureKeyVault@1
+        displayName: Get secrets
+        inputs:
+          azureSubscription: $(azure.subscription)
+          keyVaultName: $(kv.name)
+          secretsFilter: >-
+            DummySecret1,
+            EdgeCapableDevice2,
+            EventHubConnStr2,
+            IotDevice2ConnStr2,
+            IotDevice3ConnStr2,
+            IotHubConnStr2,
+            IotHubMqttHeadCert
+      - task: DotNetCoreInstaller@0
+        displayName: Install .NET Core
         inputs:
-          azureSubscription: "$(azure.subscription)"
-          ScriptPath: 'scripts/windows/setup/Install-VaultCertificate.ps1'
-          ScriptArguments: '-VaultName $(kv.name) -CertificateName IoTEdgeTestCert'
-          azurePowerShellVersion: LatestVersion
-      - powershell: scripts/windows/setup/Install-Prerequisites.ps1 -DotnetSdkUrl $(NetCorePackageUri)
-        name: install_dependencies
-        displayName: Install dependencies
-      - powershell: scripts/windows/build/Publish-Branch.ps1 -Configuration:$env:BUILD_CONFIGURATION -UpdateVersion
-        name: build
+          packageType: sdk
+          version: $(dotnet.version)
+      - powershell: scripts/windows/build/Publish-Branch.ps1 -Configuration $(Build.Configuration) -UpdateVersion
         displayName: Build
-      - powershell: scripts/windows/test/Test-Branch.ps1 -Filter "$env:TEST_FILTER" -BuildConfig "$env:BUILD_CONFIGURATION"
-        name: test
+      - powershell: scripts/windows/test/Test-Branch.ps1 -Filter "$(test.filter)" -BuildConfig $(Build.Configuration)
         displayName: Test
         env:
-          TEST_FILTER: $(test.filter)
-      - task: PublishBuildArtifacts@1
-        displayName: 'Publish Artifact: e2e deployment files'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish/e2e_deployment_files'
-          ArtifactName: 'core-windows/e2e_deployment_files'
-      - task: PublishBuildArtifacts@1
-        displayName: 'Publish Artifact: IotEdgeQuickStart'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish/IotEdgeQuickStart'
-          ArtifactName: 'core-windows/IotEdgeQuickStart'
-      - task: PublishBuildArtifacts@1
-        name: publish_artifacts
-        displayName: 'Publish Artifacts'
-        inputs:
-          PathtoPublish: '$(Build.BinariesDirectory)/publish'
-          ArtifactName: 'core-windows'
+          DummySecret1: $(DummySecret1)
+          EdgeCapableDevice2: $(EdgeCapableDevice2)
+          EventHubConnStr2: $(EventHubConnStr2)
+          IotDevice2ConnStr2: $(IotDevice2ConnStr2)
+          IotDevice3ConnStr2: $(IotDevice3ConnStr2)
+          IotHubConnStr2: $(IotHubConnStr2)
+          IotHubMqttHeadCert: $(IotHubMqttHeadCert)
       - task: PublishTestResults@1
-        name: publish_results
-        displayName: 'Publish test results'
+        displayName: Publish test results
         condition: succeededOrFailed()
         inputs:
           testRunner: VSTest
           testResultsFiles: '**\TestResults\*.trx'
-      - powershell: docker system prune -a -f
-        name: clean_machine
-        displayName: Clean machine

edge-util/test/Microsoft.Azure.Devices.Edge.Util.Test.Common/SecretsHelper.cs

@@ -19,9 +19,13 @@ public static class SecretsHelper
         public static Task<string> GetSecret(string secret)
         {
             Preconditions.CheckNonWhiteSpace(secret, nameof(secret));
-            return Uri.TryCreate(secret, UriKind.Absolute, out Uri secretUri)
-                ? KeyVaultHelperLazy.Value.GetSecret(secretUri.AbsoluteUri)
-                : KeyVaultHelperLazy.Value.GetSecret(KeyVaultBaseUrl, secret);
+            // Get the secret from the environment first, otherwise go to key vault
+            string value = ConfigHelper.TestConfig[secret];
+            return string.IsNullOrWhiteSpace(value)
+                ? Uri.TryCreate(secret, UriKind.Absolute, out Uri secretUri)
+                    ? KeyVaultHelperLazy.Value.GetSecret(secretUri.AbsoluteUri)
+                    : KeyVaultHelperLazy.Value.GetSecret(KeyVaultBaseUrl, secret)
+                : Task.Run(() => value);
         }
 
         public static Task<string> GetSecretFromConfigKey(string configName)

scripts/linux/buildBranch.sh

@@ -67,15 +67,20 @@ process_args()
         exit 1
     fi
 
-    if [ -f "$AGENT_WORKFOLDER/dotnet/dotnet" ]; then   # VSTS Linux
-        DOTNET_ROOT_PATH="$AGENT_WORKFOLDER/dotnet"
-    elif [ -f "/usr/share/dotnet/dotnet" ]; then        # default Linux
-        DOTNET_ROOT_PATH="/usr/share/dotnet"
-    elif [ -f "/usr/local/share/dotnet/dotnet" ]; then  # default macOS
-        DOTNET_ROOT_PATH="/usr/local/share/dotnet"
-    else
-        echo "dotnet not found" 1>&2
-        exit 1
+    if [ ! -f "$DOTNET_ROOT_PATH" ]; then
+        local dotnet_path=$(command -v dotnet)
+        if [ $? -eq 0 ]; then
+            DOTNET_ROOT_PATH=$(dirname $dotnet_path)
+        elif [ -f "$AGENT_WORKFOLDER/dotnet/dotnet" ]; then # VSTS Linux
+            DOTNET_ROOT_PATH="$AGENT_WORKFOLDER/dotnet"
+        elif [ -f "/usr/share/dotnet/dotnet" ]; then        # default Linux
+            DOTNET_ROOT_PATH="/usr/share/dotnet"
+        elif [ -f "/usr/local/share/dotnet/dotnet" ]; then  # default macOS
+            DOTNET_ROOT_PATH="/usr/local/share/dotnet"
+        else
+            echo "dotnet not found" 1>&2
+            exit 1
+        fi
     fi
 
     if [ ! -d "$BUILD_BINARIESDIRECTORY" ]; then

scripts/linux/runTests.sh

@@ -18,12 +18,22 @@ TEST_FILTER="$1"
 BUILD_CONFIG="$2"
 
 SUFFIX='Microsoft.Azure*test.dll'
-DOTNET_ROOT_PATH=$AGENT_WORKFOLDER/dotnet
 OUTPUT_FOLDER=$BUILD_BINARIESDIRECTORY
 
-if [ ! -f "$DOTNET_ROOT_PATH/dotnet" ]; then
-  echo "Path $DOTNET_ROOT_PATH/dotnet does not exist" 1>&2
-  exit 1
+if [ ! -f "$DOTNET_ROOT_PATH" ]; then
+    dotnet_path=$(command -v dotnet)
+    if [ $? -eq 0 ]; then
+        DOTNET_ROOT_PATH=$(dirname $dotnet_path)
+    elif [ -f "$AGENT_WORKFOLDER/dotnet/dotnet" ]; then # VSTS Linux
+        DOTNET_ROOT_PATH="$AGENT_WORKFOLDER/dotnet"
+    elif [ -f "/usr/share/dotnet/dotnet" ]; then        # default Linux
+        DOTNET_ROOT_PATH="/usr/share/dotnet"
+    elif [ -f "/usr/local/share/dotnet/dotnet" ]; then  # default macOS
+        DOTNET_ROOT_PATH="/usr/local/share/dotnet"
+    else
+        echo "dotnet not found" 1>&2
+        exit 1
+    fi
 fi
 
 if [ ! -d "$BUILD_BINARIESDIRECTORY" ]; then

scripts/windows/build/Publish-Branch.ps1

@@ -59,31 +59,37 @@ if (-not $BuildSourceVersion) {
     $BuildSourceVersion = DefaultBuildSourceVersion
 }
 
-$SLN_PATTERN = "Microsoft.Azure.*.sln"
-$TEST_CSPROJ_PATTERN = "Microsoft.Azure*Test.csproj"
+if (Test-Path env:DOTNET_PATH) {
+    $DOTNET_PATH = $env:DOTNET_PATH
+} else {
+    $dotnetCmd = Get-Command dotnet -ErrorAction SilentlyContinue
+    if ($?) {
+        $DOTNET_PATH = $dotnetCmd.Path
+    } else {
+        $DOTNET_PATH = [IO.Path]::Combine($AgentWorkFolder, "dotnet", "dotnet.exe")
+    }
+}
+
+if (Test-Path $DOTNET_PATH -PathType Leaf) {
+    Write-Host "Found '$DOTNET_PATH'"
+} else {
+    throw "$DOTNET_PATH not found"
+}
 
-$DOTNET_PATH = [IO.Path]::Combine($AgentWorkFolder, "dotnet", "dotnet.exe")
 $PUBLISH_FOLDER = Join-Path $BuildBinariesDirectory "publish"
-$RELEASE_TESTS_FOLDER = Join-Path $BuildBinariesDirectory "release-tests"
 $VERSIONINFO_FILE_PATH = Join-Path $BuildRepositoryLocalPath "versionInfo.json"
 
 $SRC_SCRIPTS_DIR = Join-Path $BuildRepositoryLocalPath "scripts"
 $PUB_SCRIPTS_DIR = Join-Path $PUBLISH_FOLDER "scripts"
-$PUB_STRESS_DIR = Join-Path $PUBLISH_FOLDER "stress"
 $SRC_BIN_DIR = Join-Path $BuildRepositoryLocalPath "bin"
 $PUB_BIN_DIR = Join-Path $PUBLISH_FOLDER "bin"
 $SRC_E2E_TEMPLATES_DIR = Join-Path $BuildRepositoryLocalPath "e2e_deployment_files"
 $PUB_E2E_TEMPLATES_DIR = Join-Path $PUBLISH_FOLDER "e2e_deployment_files"
 $SRC_E2E_TEST_FILES_DIR = Join-Path $BuildRepositoryLocalPath "e2e_test_files"
 $PUB_E2E_TEST_FILES_DIR = Join-Path $PUBLISH_FOLDER "e2e_test_files"
-$TEST_SCRIPTS_DIR = Join-Path $RELEASE_TESTS_FOLDER "scripts"
 $SRC_CERT_TOOLS_DIR = Join-Path $BuildRepositoryLocalPath "tools/CACertificates"
 $PUB_CERT_TOOLS_DIR = Join-Path $PUBLISH_FOLDER "CACertificates"
 
-if (-not (Test-Path $DOTNET_PATH -PathType Leaf)) {
-    throw "$DOTNET_PATH not found"
-}
-
 if (Test-Path $BuildBinariesDirectory -PathType Container) {
     Remove-Item $BuildBinariesDirectory -Force -Recurse
 }
@@ -229,17 +235,17 @@ $IoTEdgeQuickstartPublishBaseFolder = Join-Path $PUBLISH_FOLDER "IoTEdgeQuicksta
 Write-Host "Publishing - IoTEdgeQuickstart x64"
 $ProjectPublishPath = Join-Path $IoTEdgeQuickstartPublishBaseFolder "x64"
 &$DOTNET_PATH publish -f netcoreapp2.1 -r "win10-x64" -c $Configuration -o $ProjectPublishPath $IoTEdgeQuickstartProjectFolder |
-	Write-Host
+    Write-Host
 if ($LASTEXITCODE -ne 0) {
-	throw "Failed publishing IoTEdgeQuickstart x64."
+    throw "Failed publishing IoTEdgeQuickstart x64."
 }
 
 Write-Host "Publishing - IoTEdgeQuickstart arm32"
 $ProjectPublishPath = Join-Path $IoTEdgeQuickstartPublishBaseFolder "arm32v7"
 &$DOTNET_PATH publish -f netcoreapp2.1 -r "win10-arm" -c $Configuration -o $ProjectPublishPath $IoTEdgeQuickstartProjectFolder |
-	Write-Host
+    Write-Host
 if ($LASTEXITCODE -ne 0) {
-	throw "Failed publishing IoTEdgeQuickstart arm32."
+    throw "Failed publishing IoTEdgeQuickstart arm32."
 }
 
 <#
@@ -251,15 +257,15 @@ $LeafDevicePublishBaseFolder = Join-Path $PUBLISH_FOLDER "LeafDevice"
 Write-Host "Publishing - LeafDevice x64"
 $ProjectPublishPath = Join-Path $LeafDevicePublishBaseFolder "x64"
 &$DOTNET_PATH publish -f netcoreapp2.1 -r "win10-x64" -c $Configuration -o $ProjectPublishPath $LeafDeviceProjectFolder |
-	Write-Host
+    Write-Host
 if ($LASTEXITCODE -ne 0) {
-	throw "Failed publishing LeafDevice x64."
+    throw "Failed publishing LeafDevice x64."
 }
 
 Write-Host "Publishing - LeafDevice arm32"
 $ProjectPublishPath = Join-Path $LeafDevicePublishBaseFolder "arm32v7"
 &$DOTNET_PATH publish -f netcoreapp2.1 -r "win10-arm" -c $Configuration -o $ProjectPublishPath $LeafDeviceProjectFolder |
-	Write-Host
+    Write-Host
 if ($LASTEXITCODE -ne 0) {
-	throw "Failed publishing LeafDevice arm32."
-}
+    throw "Failed publishing LeafDevice arm32."
+}

scripts/windows/test/Test-Branch.ps1

@@ -44,10 +44,22 @@ if (-not $BuildBinariesDirectory) {
 
 $SUFFIX = "Microsoft.Azure*test.dll"
 $LOGGER_ARG = "trx;LogFileName=result.trx"
-$DOTNET_PATH = [IO.Path]::Combine($AgentWorkFolder, "dotnet", "dotnet.exe")
 
-if (-not (Test-Path $DOTNET_PATH -PathType Leaf)) {
-    throw "$DOTNET_PATH not found."
+if (Test-Path env:DOTNET_PATH) {
+    $DOTNET_PATH = $env:DOTNET_PATH
+} else {
+    $dotnetCmd = Get-Command dotnet -ErrorAction SilentlyContinue
+    if ($?) {
+        $DOTNET_PATH = $dotnetCmd.Path
+    } else {
+        $DOTNET_PATH = [IO.Path]::Combine($AgentWorkFolder, "dotnet", "dotnet.exe")
+    }
+}
+
+if (Test-Path $DOTNET_PATH -PathType Leaf) {
+    Write-Host "Found '$DOTNET_PATH'"
+} else {
+    throw "$DOTNET_PATH not found"
 }
 
 if (-not $BuildConfig) {
@@ -76,7 +88,7 @@ foreach ($testDll in (Get-ChildItem $BuildBinariesDirectory -Include $SUFFIX -Re
 	}
 }
 
-$testCommandPrefix = "$DOTNET_PATH vstest /Logger:`"$LOGGER_ARG`" /TestAdapterPath:`"$BuildBinariesDirectory`" /Parallel /InIsolation"
+$testCommandPrefix = "& '$DOTNET_PATH' vstest /Logger:`"$LOGGER_ARG`" /TestAdapterPath:`"$BuildBinariesDirectory`" /Parallel /InIsolation"
 
 if ($Filter) {
     $testCommandPrefix += " /TestCaseFilter:`"$Filter`""