[Eng] Added necessary tools/pipelines update to migrate main to future (

Azure#27104) * polish parameter for buildmodules.ps1 * polish * refine build and prepare autorest * remove unused changes * update csproj mapping to add folder generated/, and trim keys * Update CIFilter to remove src/ from mapping keys * add test-module.ps1 in gen2main * add check-dependencies.ps1 in gen2main * add psd1 in gen2main * update az.autorest.props to apply new folder structure * fix submodule name * fix trimmed submodule name * fix trimmed submodule name * add script to migrate generated folder * uncomment * Add readme.azure.md and readme.azure.noprofile.md * migrate readme from eng/archive-test branch * update migration script to add generated csproj in sln * fix previous commit * copy other script from eng/archive-test * fix script * refine build-module.ps1 and add some log * refine log message * color log output * use foreach instead of foreach-object to improve performance * improve write log * fix * question * apply new build process to dotnet msbuild /t:build * test pipeline 1 * test pipeline 2 * test pipeline 3 * test pipeline 4 * test pipeline 5 * test pipeline 6 * test pipeline 7 * test pipeline 8 * test pipeline 9 * test pipeline 10 * test pipeline 11 * test pipeline 12 * test pipeline 12 * test pipeline 13 * test pipeline 14 * test pipeline 15 * test pipeline 16 * test pipeline 17 * test pipeline 18 * test pipeline 19 * test pipeline 20 * test pipeline 21 * test pipeline 22 * test pipeline 23 * test pipeline 24 * test pipeline 25 * test pipeline 26 * test pipeline 27 * test pipeline 28 * test pipeline 29 * test pipeline 30 * test pipeline 31 * test pipeline 32 * test pipeline 33 * split build function to buildscripts.psm1 * Add generate id in generate-info * Add generate id in generate-info * Add generate id in generate-info * regenerate help during adaptautorestmodule * async generate * more async * move resolve tools back * update migrate script * fix path * generate info while migrating main * fix * Remove redundant dependencies for autogen modules * add log output * generate id * fix * revert async * fix * adjust generate id * exit when no generate id * fix * minor fix * fix * update readme.azure.noprofile.md * fixed after build * remove sln when build finish * add migration script * revert autorest dpendency changes * remove redundant * fix * fix * fix * fix * copy src/helpers from generation branch * fix * fix * fix * test * fix * fix * fix * fix * fix * fix * fix * fix * fix * update security tools * typo * fix * fix * fix * fix * update reference of csproj in resource.test.csproj * fix * update references of autorest modules for resource.sln * move removeunwanted and updatemodule to buildscript * polish parameter name for build script * add switch invokedbypipeline * trigger new pipeline for future branch by default * add accounts in build by default * fix * fix * build version controller during build script * fix * Update security-tools.yml * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * Update MigrateGenerated.ps1 * add accounts to build/test by default for CIPlan parameterset * Update AdaptAutorestModule.ps1 * Update AdaptAutorestModule.ps1 * do not add testfx specificly because it was already included for all test project * split build solution and test solution * add parent module csproj to module root sln for new module * fix incorrect parent module csproj name * fix incorrect help refresh for new module * fix * prepare autorest modules before calculate csproj to build * fix calculate generated files for new submodule * remove PR analyze that block PR has changes in both /src and /generated * remove files accidentally checked in * rename x.autorest/help -> x.autorest/docs * update pipeline * update build for daily-build.yml * new doc (unfinished) * finish new docs * update PR title * update * update * update * update * do not exit when pr analyze failed, write warning instead * remove generated ps1 script from autorest submodules during future migration * add assembly info attributes in readme.azure.noprofile.md * allow resources to be checked in * add git files to each autorest submodule during migration * disable assemblyinfo generate * add final block in after build task * replace module version for submodule assemblyinfo * reset assemblyinfo in finally block after re-compile * bugfix * polish file move * fix * skip livetests filder * fix * assemblyinfo * copy assemblyinfo separately * fix * check whether assemblyinfo existed before modifying it * fix child module path when add to sln * keep guid in psd1 under generated folder unchanged * update afterbuild task to take module-name instead of service-name * split archive pipeline from build-steps * refactor cifiltertask * refactor pipelines * move yml of archive pipeline * forgot function * pr validation pipeline won't be triggered by CI * fix path for util templates * fix array accidentally casted to string * fix branch name * fix parameter name for build-steps * add terminate condition when no files changed detected in archive pipeline * add helpers to not modules list * add helpers to not modules list * move helpers folder from generation branch * fix condition in yml * do use exit 0 when terminate * try to fix terminate condition * fix switch syntax * try to fix terminate condition * try to fix terminate condition * have to fail archive when no outdated modules found * remove duplicate warning message * fix * fix * enable new build process for windows-powershell.yml * remove duplicate warning message * use dotnet 8 for credscan * fix security tools pipeline * use batch ci trigger archive pipeline * refactor archive pipeline * add branch name filter, get token with internal method * add doc for future branch PR migration * add doc for future branch PR migration * removed migration docs * enable new ci/pr pipelines for all branches * add README.md for pipelinesteps scripts --------- Co-authored-by: Nori Zhang <norizhang@microsoft.com> Co-authored-by: NoriZC <110961157+NoriZC@users.noreply.github.com> Co-authored-by: Xiaogang Ding <xidi@microsoft.com>
azure-sdk · Feb 14, 2025 · ada2778 · ada2778
1 parent 715a440
commit ada2778
Show file tree

Hide file tree

Showing 41 changed files with 3,138 additions and 344 deletions.
diff --git a/.azure-pipelines/PipelineSteps/README.md b/.azure-pipelines/PipelineSteps/README.md
@@ -0,0 +1,7 @@
+# Pipeline Steps
+
+This directory contains the PowerShell scripts refactored from ".azure-pipelines/util".
+
+## Overview
+
+The pipeline steps are refactored from yml based templates from ".azure-pipelines/util" to PowerShell scripts. This is designed for converting some Azure Devops pipelines to github action.
diff --git a/.azure-pipelines/PipelineSteps/analyze-steps.ps1 b/.azure-pipelines/PipelineSteps/analyze-steps.ps1
@@ -0,0 +1,44 @@
+# Copyright Microsoft Corporation
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ----------------------------------------------------------------------------------
+[CmdletBinding(DefaultParameterSetName="AllSet")]
+param (
+    [string]$RepoRoot,
+    [string]$Configuration = 'Debug',
+    [string]$PowerShellPlatform
+)
+
+# Install PowerShell dependencies
+Write-Host -ForegroundColor Green "-------------------- Start installing PowerShell dependencies ... --------------------"
+Install-Module "platyPS", "PSScriptAnalyzer" -Repository PSGallery -Force -Confirm:$false -Scope CurrentUser
+Write-Host -ForegroundColor DarkGreen "-------------------- End installing PowerShell dependencies ... --------------------`n`n`n`n`n"
+
+# Install latest Az modules
+Write-Host -ForegroundColor Green "-------------------- Start installing latest Az modules ... --------------------"
+New-Item -ItemType Directory -Path "Az-Cmdlets-latest"
+Invoke-WebRequest -Uri "https://azpspackage.blob.core.windows.net/release/Az-Cmdlets-latest.tar.gz" -OutFile "Az-Cmdlets-latest/Az-Cmdlets-latest.tar.gz" -MaximumRetryCount 2 -RetryIntervalSec 1
+tar -xvzf "Az-Cmdlets-latest/Az-Cmdlets-latest.tar.gz" -C "Az-Cmdlets-latest"
+& Az-Cmdlets-latest/InstallModule.ps1
+Write-Host -ForegroundColor DarkGreen "-------------------- End installing latest Az modules ... --------------------`n`n`n`n`n"
+
+# Generate help
+Write-Host -ForegroundColor Green "-------------------- Start generating help ... --------------------"
+if ($PowerShellPlatform) {
+    $Env:PowerShellPlatform = $PowerShellPlatform
+}
+$buildProjPath = Join-Path $RepoRoot 'build.proj'
+dotnet msbuild $buildProjPath /t:GenerateHelp "/p:Configuration=$Configuration"
+Write-Host -ForegroundColor DarkGreen "-------------------- End generating help ... --------------------`n`n`n`n`n"
+
+# Static Analysis
+Write-Host -ForegroundColor Green "-------------------- Start static analysis ... --------------------"
+dotnet msbuild $buildProjPath /t:StaticAnalysis "/p:Configuration=$Configuration"
+Write-Host -ForegroundColor DarkGreen "-------------------- End static analysis ... --------------------`n`n`n`n`n"
diff --git a/.azure-pipelines/PipelineSteps/build-steps.ps1 b/.azure-pipelines/PipelineSteps/build-steps.ps1
@@ -0,0 +1,118 @@
+# ----------------------------------------------------------------------------------
+#
+# Copyright Microsoft Corporation
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ----------------------------------------------------------------------------------
+[CmdletBinding(DefaultParameterSetName="AllSet")]
+param (
+    [string]$RepoRoot,
+    [string]$Configuration = 'Debug',
+    [string]$PowerShellPlatform,
+    [string]$FilesChangedOutputPath,
+    [boolean]$ForceRegenerate = $false,
+    [string]$SubTasksFilePath,
+    [boolean]$IsSecurityCheck,
+    [string]$BuildReason,
+    [string]$Trigger
+)
+
+# filter changed files
+Write-Host -ForegroundColor Green "-------------------- Start filtering changed files ... --------------------"
+$buildProjPath = Join-Path $RepoRoot 'build.proj'
+if ($PowerShellPlatform) {
+    $Env:PowerShellPlatform = $PowerShellPlatform
+}
+dotnet msbuild $buildProjPath /t:FilterBuild "/p:FilesChangedOutputPath=$FilesChangedOutputPath;SubTasksFilePath=$SubTasksFilePath;IsSecurityCheck=$IsSecurityCheck"
+Write-Host -ForegroundColor DarkGreen "-------------------- End filtering changed files ... --------------------`n`n`n`n`n"
+
+# Set subtasks
+if (Test-Path $SubTasksFilePath) {
+    Write-Host -ForegroundColor Green "-------------------- Start setting subtasks ... --------------------"
+    Get-Content $SubTasksFilePath | ForEach-Object {
+      if ($_ && 'Predictor' -eq $_) {
+        $subTaskPredictor = $true
+      } elseif ($_ && 'Installer' -eq $_) {
+        Write-Host "##vso[task.setvariable variable=SubTaskInstaller]true"
+        $subTaskInstaller = $true
+      } elseif ($_ && 'all' -eq $_) {
+        $subTaskAll = $true
+      }
+    }
+    Write-Host -ForegroundColor DarkGreen "-------------------- End setting subtasks ... --------------------`n`n`n`n`n"
+}
+
+# Analyze changed files
+Write-Host -ForegroundColor Green "-------------------- Start Analyzing changed files ... --------------------"
+Write-Host "##1. Check generate-info.json for autorest generated modules" -ForegroundColor Magenta
+$noGenerateInfo = @()
+$modules = Get-Content $FilesChangedOutputPath -OutVariable paths | Foreach-Object { if($_ -match "^src.*\.Autorest"){ Write-Output $Matches[0] } } | Select-Object -Unique
+foreach ($module in $modules) {
+$hasGenerateInfo = $false
+$pattern = "^$module/generate-info.json"
+foreach ($path in $paths) {
+    if ($path -match $pattern) {
+    $hasGenerateInfo = $true
+    }
+}
+if (-not $hasGenerateInfo) {
+    $noGenerateInfo += $module
+}
+}
+if ($noGenerateInfo -and $noGenerateInfo.Count -gt 0) {
+$noGenerateInfo | Foreach-Object { Write-Warning "No generate-info.json detected for $_." }
+}
+
+# this might not be true, for example when release branch or preview branch merge back to main branch, it's possible changes present in both /src and /generated
+Write-Host "##2. PR should contain changes from either /src or /generated only" -ForegroundColor Magenta
+$content = Get-Content $FilesChangedOutputPath
+$srcFiles = $content | Where-Object { $_ -match "^src.*" }
+$generatedFiles = $content | Where-Object { $_ -match "^generated.*" }
+if ($srcFiles -and $srcFiles.Count -gt 0  -and $generatedFiles -and $generatedFiles.Count -gt 0) {
+Write-Warning "PR should contain changes from either /src or /generated only."
+}
+Write-Host -ForegroundColor DarkGreen "-------------------- End Analyzing changed files ... --------------------`n`n`n`n`n"
+
+# Check ignored files
+Write-Host -ForegroundColor Green "-------------------- Start checking ignored files ... --------------------"
+$checkIgnoreFilesScriptPath = Join-Path $RepoRoot 'tools' 'CheckIgnoredFile.ps1'
+& $checkIgnoreFilesScriptPath
+Write-Host -ForegroundColor DarkGreen "-------------------- End checking ignored files ... --------------------`n`n`n`n`n"
+
+# Build
+Write-Host -ForegroundColor Green "-------------------- Start building modules ... --------------------"
+$buildScriptPath = Join-Path $RepoRoot 'tools' 'BuildScripts' 'BuildModules.ps1'
+if ('Manual' -eq $BuildReason -and $ForceRegenerate) {
+& $buildScriptPath -CIPlan -RepoRoot $RepoRoot -Configuration $Configuration -ForceRegenerate -InvokedByPipeline
+} else {
+& $buildScriptPath -CIPlan -RepoRoot $RepoRoot -Configuration $Configuration -InvokedByPipeline
+}
+Write-Host -ForegroundColor DarkGreen "-------------------- End building modules ... --------------------`n`n`n`n`n"
+
+# Write pipeline result
+Write-Host -ForegroundColor Green "-------------------- Start writing pipeline result ... --------------------"
+$pipelineScript = Join-Path $RepoRoot 'tools' 'ExecuteCIStep.ps1'
+$repoArtifact = Join-Path $RepoRoot 'artifacts'
+& $pipelineScript -Build -TriggerType $BuildReason -Trigger $Trigger -RepoArtifacts $repoArtifact -Configuration $Configuration
+Write-Host -ForegroundColor DarkGreen "-------------------- End writing pipeline result ... --------------------`n`n`n`n`n"
+
+# Build Az.Tools.Predictor
+if ($subTaskPredictor -or $subTaskAll) {
+    Write-Host -ForegroundColor Green "-------------------- Start building Az.Tools.Predictor ... --------------------"
+    dotnet msbuild $buildProjPath /t:AzToolsPredictor
+    Write-Host -ForegroundColor DarkGreen "-------------------- End building Az.Tools.Predictor ... --------------------`n`n`n`n`n"
+}
+
+# Build Az.Tools.Installer
+if ($subTaskInstaller -or $subTaskAll) {
+    Write-Host -ForegroundColor Green "-------------------- Start building Az.Tools.Installer ... --------------------"
+    dotnet msbuild $buildProjPath /t:AzToolsInstaller
+    Write-Host -ForegroundColor DarkGreen "-------------------- End building Az.Tools.Installer ... --------------------`n`n`n`n`n"
+}
diff --git a/.azure-pipelines/PipelineSteps/test-steps.ps1 b/.azure-pipelines/PipelineSteps/test-steps.ps1
@@ -0,0 +1,77 @@
+# Copyright Microsoft Corporation
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ----------------------------------------------------------------------------------
+[CmdletBinding(DefaultParameterSetName="AllSet")]
+param (
+    [string]$RepoRoot,
+    [string]$PowerShellPlatform,
+    [string]$TestFramework,
+    [string]$Configuration = 'Debug'
+)
+
+# Remove pre-installed Az modules
+Write-Host -ForegroundColor Green "-------------------- Start removing pre-installed Az modules ... --------------------"
+$commonUtilityScriptPath = Join-Path $RepoRoot "tools" "TestFx" "Utilities" "CommonUtility.ps1"
+& $commonUtilityScriptPath
+Write-Host -ForegroundColor DarkGreen "-------------------- End removing pre-installed Az modules ... --------------------`n`n`n`n`n"
+
+# Test
+Write-Host -ForegroundColor Green "-------------------- Start testing ... --------------------"
+if ($PowerShellPlatform) {
+    $Env:PowerShellPlatform = $PowerShellPlatform
+}
+$preference = $ErrorActionPreference
+$ErrorActionPreference = 'Continue'
+$buildProjPath = Join-Path $RepoRoot 'build.proj'
+dotnet msbuild $buildProjPath /t:Test "/p:Configuration=$Configuration;TestFramework=$TestFramework"
+Write-Host -ForegroundColor DarkGreen "-------------------- End testing ... --------------------`n`n`n`n`n"
+
+# Test AutoGen Modules With PowerShell Core
+Write-Host -ForegroundColor Green "-------------------- Start testing AutoGen modules with PowerShell Core ... --------------------"
+$executeCIStepScriptPath = Join-Path $RepoRoot "tools" "ExecuteCIStep.ps1"
+$currentPath = $PWD
+$debugFolderPath = Join-Path $RepoRoot "artifacts" "Debug"
+Set-Location $debugFolderPath
+
+Install-Module -Name Pester -Repository PSGallery -RequiredVersion 4.10.1 -Force
+if ($IsWindows) { $sp = ";" } else { $sp = ":" }
+$env:PSModulePath = $env:PSModulePath + $sp + (pwd).Path
+Get-ChildItem -File -Recurse test-module.ps1 | ForEach-Object {
+Write-Host $_.Directory.FullName
+$repoArtifact = Join-Path $RepoRoot 'artifacts'
+& $executeCIStepScriptPath -TestAutorest -AutorestDirectory $_.Directory.FullName -RepoArtifacts $repoArtifact
+}
+
+$ErrorActionPreference = $preference
+Set-Location $currentPath
+Write-Host -ForegroundColor DarkGreen "-------------------- End testing AutoGen modules with PowerShell Core ... --------------------`n`n`n`n`n"
+
+# Analyze test coverage
+Write-Host -ForegroundColor Green "-------------------- Start analyzing test coverage ... --------------------"
+$validateTestCoverageScriptPath = Join-Path $RepoRoot 'tools' 'TestFx' 'Coverage' 'ValidateTestCoverage.ps1'
+& $validateTestCoverageScriptPath
+Write-Host -ForegroundColor DarkGreen "-------------------- End analyzing test coverage ... --------------------`n`n`n`n`n"
+
+# Check test status
+Write-Host -ForegroundColor Green "-------------------- Start checking test status ... --------------------"
+$currentPath = $PWD
+$pipelineResultPath = Join-Path $RepoRoot "artifacts" "PipelineResult"
+Set-Location $pipelineResultPath
+
+$PipelineResult = Get-Content PipelineResult.json | ConvertFrom-Json
+$FailedModuleList = $PipelineResult.test.Details[0].Modules | Where-Object { $_.Status -eq "Failed" } | ForEach-Object { $_.Module }
+if ($FailedModuleList.Length -ne 0)
+{
+    throw "test fails in module: $FailedModuleList"
+}
+
+Set-Location $currentPath
+Write-Host -ForegroundColor DarkGreen "-------------------- End checking test status ... --------------------`n`n`n`n`n"