Update latest from master branch (#1785)

* Tweaks for vcpkg use

* Bring up to date with main

* Sync eng/common directory with azure-sdk-tools for PR 1579 (#1724)

* Allow propogation of SetDevVersion.

* Allow use of SetDevVersion locally and across stages.

Co-authored-by: Mitch Denny <midenn@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 1524 (#1728)

* Make RELEASE_TITLE_REGEX more specific

* Update eng/common/scripts/ChangeLog-Operations.ps1

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Update release REGEX

* tighten up changelog verification

* Ensure releaseDate is in the appriopriate format.

* Update error messages.

* Retrun false after error

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Update samples to include log of mqtt client username (#1731)

* Typo "Github"→"GitHub" (#1738)

* Revert changes to SetDevVersion. (#1740)

Co-authored-by: Mitch Denny <midenn@microsoft.com>

* Add better error handling in Prepare-Release (#1746)

Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 1594 (#1747)

* Update sparse checkout create directory step name

* plumb BaseBranchName to create pull request

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Fix sample readme urls (#1752)

* allow multiple invocations of archetype-sdk-tests-generate (#1754)

Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>

* Sync eng/common directory with azure-sdk-tools for PR 1585 (#1758)

* Sparse checkout docs repository in release step

* Detect default branch for custom sparse checkout repository

* Use sparse checkout paths parameter in docs metadata release

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 1615 (#1761)

* Move docindex.yml from steps -> jobs

* Add docindex.yml

Co-authored-by: Daniel Jurek <djurek@microsoft.com>

* add clang format version (#1755)

* Sync eng/common directory with azure-sdk-tools for PR 1627 (#1763)

* Refactor sparse checkout template: idempotency and no yaml loops

* Add logging, re-organize sparse checkout script

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Add Get-ChangeLogEntriesFromContent (#1764)

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>

* Sync eng/common directory with azure-sdk-tools for PR 1635 (#1767)

* Update common engineering system for docs onboarding

* Add updates from review feedback

* Space nit

Co-authored-by: Daniel Jurek <djurek@microsoft.com>

* add empty section for compile options (#1741)

* Enable Caching of PS Modules (#1769)

- Remove copied AzPowershell utilities
- Add latest AZ module path already on hosted agents to PSModulePath
- Rename setup-az-modules template setup-environments to reflect what is is doing
- Add support for Caching the current user PS Module folder
- Add support for install-module if not already present in module folder
- Organize the live test clean-up script to be in the standard location

Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>

* Adding new CA trusted certificates (#1765)

Adding new CA certificates and updating the documentation to create a store when testing in Windows or OSX.

* Sync eng/common directory with azure-sdk-tools for PR 1611 (#1766)

* Add API status check

* Revert temp change

* Update as per review comments

* Removed blank line

* Review comments to use az cli

* Changes to move az cli commands to caller script and other review comments

* Skip languages that's not supported in APIView

* Fix bug in language mapping

Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
Co-authored-by: praveenkuttappan <55455725+praveenkuttappan@users.noreply.github.com>

* Fix for Issue #1768  - Response Types (#1770)

* Add the ability to check for open pull request to a different repo. (#1774)

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>

* Fetch specific branch name only in git-branch-push script (#1771)

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Use generate matrix job name parameter as display name (#1776)

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* Updating documentation for ECC server certificate chain support. (#1773)

* Sync eng/common directory with azure-sdk-tools for PR 1633 (#1778)

* Update format of new Changelog Entry

* Add parsing of changelog sections

* Update ChangeLog Logic

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>

* Strict mode needs the variables initialized (#1779)

Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>

* Fix of depricated BearSSL method call of WiFiClientSecure class in sample for ESP8266. (#1780)

* Fix of depricated BearSSL method call in sample for ESP8266.

* Fix of depricated BearSSL method call of WiFiClientSecure class in sample for ESP8266.

Co-authored-by: ts\bozicekm <miha.bozicek@telekom.si>

* Sync eng/common directory with azure-sdk-tools for PR 1688 (#1782)

* Add verification of changelog sections

* Update eng/common/scripts/ChangeLog-Operations.ps1

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Mitch Denny <midenn@microsoft.com>
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Mollie Munoz <mollie.munoz@microsoft.com>
Co-authored-by: Hiroshi Yoshioka <40815708+hyoshioka0128@users.noreply.github.com>
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
Co-authored-by: Dane Walton <dane.walton@microsoft.com>
Co-authored-by: Cristian Pop <CIPop@users.noreply.github.com>
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
Co-authored-by: praveenkuttappan <55455725+praveenkuttappan@users.noreply.github.com>
Co-authored-by: Miha Božiček <miha.bozicek@gmail.com>
Co-authored-by: ts\bozicekm <miha.bozicek@telekom.si>

README.md

@@ -302,7 +302,7 @@ git clone https://github.com/Microsoft/vcpkg.git
 # (consider this path as PATH_TO_VCPKG)
 cd vcpkg
 # Checkout the vcpkg commit from the vcpkg-commit.txt file (link above)
-git checkout <vcpkg commit>
+git checkout <vcpkg commit> 
 
 # build vcpkg
 .\bootstrap-vcpkg.bat

eng/cmake/global_compile_options.txt

@@ -17,10 +17,12 @@ elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
   endif()
 
   add_compile_options(-Xclang -Wall -Wextra -pedantic  ${WARNINGS_AS_ERRORS_FLAG} -Wdocumentation -Wdocumentation-unknown-command -fcomment-block-commands=retval -Wcast-qual -Wunused -Wuninitialized -Wmissing-declarations -Wconversion -Wpointer-arith -Wshadow -Wfloat-equal)
-else()
+elseif(CMAKE_C_COMPILER_ID MATCHES "GNU")
   if(WARNINGS_AS_ERRORS)
     set(WARNINGS_AS_ERRORS_FLAG "-Werror")
   endif()
 
   add_compile_options(-Wall -Wextra -pedantic  ${WARNINGS_AS_ERRORS_FLAG} -Wcast-qual -Wunused -Wuninitialized -Wmissing-declarations -Wconversion -Wpointer-arith -Wshadow -Wlogical-op -Wfloat-equal)
+else()
+  message(WARNING "Using an unsupported compiler. Disabling stricter compiler flags.")
 endif()

eng/common/TestResources/deploy-test-resources.yml

@@ -33,10 +33,12 @@ parameters:
 
 
 steps:
-  - template: /eng/common/TestResources/setup-az-modules.yml
+  - template: /eng/common/pipelines/templates/steps/cache-ps-modules.yml
+
+  - template: /eng/common/TestResources/setup-environments.yml
 
   - pwsh: |
-      eng/common/TestResources/Import-AzModules.ps1
+      eng/common/scripts/Import-AzModules.ps1
 
       $subscriptionConfiguration = @'
         ${{ parameters.SubscriptionConfiguration }}

eng/common/TestResources/remove-test-resources.yml

@@ -21,7 +21,7 @@ parameters:
 
 steps:
   - pwsh: |
-      eng/common/TestResources/Import-AzModules.ps1
+      eng/common/scripts/Import-AzModules.ps1
 
       $subscriptionConfiguration = @"
         ${{ parameters.SubscriptionConfiguration }}

eng/common/TestResources/setup-az-modules.yml → eng/common/TestResources/setup-environments.yml

@@ -21,12 +21,12 @@ steps:
     condition: contains(variables['OSVmImage'], 'mac')
 
   - task: Powershell@2
+    displayName: Register Dogfood environment
     inputs:
-      displayName: Register Dogfood environment
       targetType: inline
       pwsh: true
       script: |
-        eng/common/TestResources/Import-AzModules.ps1
+        eng/common/scripts/Import-AzModules.ps1
 
         $environmentSpec = @"
           $(env-config-dogfood)

eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml

@@ -31,15 +31,19 @@ parameters:
 - name: OsVmImage
   type: string
   default: MMSUbuntu18.04
+# This parameter is only necessary if there are multiple invocations of this template within the SAME STAGE.
+# When that occurs, provide a name other than the default value.
+- name: GenerateJobName
+  type: string
+  default: 'generate_job_matrix'
 
 jobs:
-- job: generate_matrix
+- job: ${{ parameters.GenerateJobName }}
   variables:
     displayNameFilter: $[ coalesce(variables.jobMatrixFilter, '.*') ]
   pool:
     name: ${{ parameters.Pool }}
     vmImage: ${{ parameters.OsVmImage }}
-  displayName: Generate Job Matrix
   ${{ if parameters.DependsOn }}:
     dependsOn: ${{ parameters.DependsOn }}
   steps:
@@ -89,8 +93,8 @@ jobs:
     - template: ${{ parameters.JobTemplatePath }}
       parameters:
         UsePlatformContainer: false
-        Matrix: dependencies.generate_matrix.outputs['generate_vm_job_matrix_${{ config.Name }}.matrix']
-        DependsOn: generate_matrix
+        Matrix: dependencies.${{ parameters.GenerateJobName }}.outputs['generate_vm_job_matrix_${{ config.Name }}.matrix']
+        DependsOn: ${{ parameters.GenerateJobName }}
         CloudConfig: ${{ parameters.CloudConfig }}
         ${{ each param in parameters.AdditionalParameters }}:
           ${{ param.key }}: ${{ param.value }}
@@ -99,8 +103,8 @@ jobs:
     - template: ${{ parameters.JobTemplatePath }}
       parameters:
         UsePlatformContainer: true
-        Matrix: dependencies.generate_matrix.outputs['generate_container_job_matrix_${{ config.Name }}.matrix']
-        DependsOn: generate_matrix
+        Matrix: dependencies.${{ parameters.GenerateJobName }}.outputs['generate_container_job_matrix_${{ config.Name }}.matrix']
+        DependsOn: ${{ parameters.GenerateJobName }}
         CloudConfig: ${{ parameters.CloudConfig }}
         ${{ each param in parameters.AdditionalParameters }}:
           ${{ param.key }}: ${{ param.value }}

eng/common/pipelines/templates/steps/cache-ps-modules.yml

@@ -0,0 +1,10 @@
+steps:
+- pwsh: |
+    . ./eng/common/scripts/Helpers/PSModule-Helpers.ps1
+    Write-Host "##vso[task.setvariable variable=CachedPSModulePath]$global:CurrentUserModulePath"
+  displayName: Set PS Modules Cache Directory
+- task: Cache@2
+  inputs:
+    key: 'PSModulePath | $(CacheSalt) | $(Agent.OS) | $(Build.SourcesDirectory)/eng/common/scripts/Import-AzModules.ps1'
+    path: $(CachedPSModulePath)
+  displayName: Cache PS Modules

eng/common/pipelines/templates/steps/create-pull-request.yml

@@ -13,7 +13,7 @@ parameters:
   PRTitle: not-specified
   PRBody: ''
   ScriptDirectory: eng/common/scripts
-  GHReviewersVariable: '' 
+  GHReviewersVariable: ''
   GHTeamReviewersVariable: ''
   GHAssignessVariable: ''
   # Multiple labels seperated by comma, e.g. "bug, APIView"
@@ -33,7 +33,7 @@ steps:
     PushArgs: ${{ parameters.PushArgs }}
     WorkingDirectory: ${{ parameters.WorkingDirectory }}
     ScriptDirectory: ${{ parameters.ScriptDirectory }}
-    SkipCheckingForChanges: ${{ parameters.SkipCheckingForChanges }} 
+    SkipCheckingForChanges: ${{ parameters.SkipCheckingForChanges }}
 
 - task: PowerShell@2
   displayName: Create pull request

eng/common/pipelines/templates/steps/daily-dev-build-variable.yml

@@ -16,7 +16,7 @@ steps:
     condition: succeeded()
 - pwsh: |
     $setDailyDevBuild = "false"
-    if (('$(Build.Reason)' -in 'Schedule', 'IndividualCI') -and ('$(System.TeamProject)' -eq 'internal')) {
+    if (('$(Build.Reason)' -eq 'Schedule') -and ('$(System.TeamProject)' -eq 'internal')) {
       $setDailyDevBuild = "true"
     }
     echo "##vso[task.setvariable variable=SetDevVersion]$setDailyDevBuild"