Enable Vertical build PoC as a more first-class concept #14274
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,6 +10,7 @@ | |
| Configuration Build configuration: "Debug", "Release", etc. | ||
| DotNetBuildFromSource Building the entire stack from source with no external dependencies. | ||
| DotNetBuildVertical Building the entire stack in "Vertical" PoC mode. This is a superset of DotNetBuildFromSource. | ||
| DotNetOutputBlobFeedDir Directory to publish Source Build assets to (packages, symbol packages, installers, etc.). | ||
| DotNetPublishUsingPipelines Publish assets to Build Asset Registry. | ||
| DotNetSymbolServerTokenMsdl Personal access token for MSDL symbol server. Available from variable group DotNet-Symbol-Server-Pats. | ||
|
|
@@ -47,7 +48,7 @@ | |
|
|
||
| <Import Project="RepoLayout.props"/> | ||
|
|
||
| <Import Project="SourceBuild/SourceBuildArcadeBuild.targets" Condition="'$(ArcadeBuildFromSource)' == 'true' or '$(ArcadeBuildVertical)' == 'true'" /> | ||
|
|
||
| <!-- Allow for repo specific Build properties such as the list of Projects to build --> | ||
| <Import Project="$(RepositoryEngineeringDir)Build.props" Condition="Exists('$(RepositoryEngineeringDir)Build.props')" /> | ||
|
|
@@ -121,10 +122,10 @@ | |
| <_CommonProps Include="VersionsPropsPath=$(VersionsPropsPath)"/> | ||
|
|
||
| <!-- | ||
| When building from source we suppress restore for projects that set ExcludeFromSourceBuild=true or ExcludeFromVerticalBuild. | ||
| NuGet Restore task reports a warning for such projects, which we suppress here. | ||
| --> | ||
| <_CommonProps Include="DisableWarnForInvalidRestoreProjects=true" Condition="'$(DotNetBuildFromSource)' == 'true' or '$(DotNetBuildVertical)' == 'true'"/> | ||
|
|
||
| <!-- | ||
| C++ projects expect VCTargetsPath property to be set. MSBuild generates this property to solution | ||
|
|
@@ -296,14 +297,14 @@ | |
| --> | ||
| <MSBuild Projects="SourceBuild\AfterSourceBuild.proj" | ||
| Properties="@(_CommonProps);_NETCORE_ENGINEERING_TELEMETRY=AfterSourceBuild" | ||
| Condition="'$(ArcadeBuildFromSource)' == 'true' or '$(ArcadeBuildVertical)' == 'true'"/> | ||
|
There was a problem hiding this comment. This confuses me. Why would invoke source build infrastructure when not building from source? Which infrastructure from AfterSourceBuild.proj would you want a VMR build to execute? We don't create prebuilt usage reports in the VMR today when not building from source and I don't see that in scope for the PoCs. Similarly, I was under the impression that we don't want to create the source built intermediate nuget packages when not building from source. If there really is some common infrastructure (please explain which part), then we should rename the There was a problem hiding this comment. You need to create the intermediate nupkgs coming out of each repo build for use in downstream builds for the orchestrated build infra to work today. IMO, we should keep the prebuilt detection in place for VB (but only error in "strict" SB mode). Yes, we should rename AfterSourceBuild to something else, but I want to do this after .NET 9: Redesign source-build behavior controls is complete. What we want to do with the PoC, IMO, is use the source build infra, while not necessarily using the SB repo behavior. |
||
|
|
||
| <!-- | ||
| Publish artifacts. | ||
| --> | ||
| <MSBuild Projects="Publish.proj" | ||
| Properties="@(_PublishProps);_NETCORE_ENGINEERING_TELEMETRY=Publish" | ||
| Targets="Publish" | ||
| Condition="'$(Publish)' == 'true' and '$(DotNetBuildFromSource)' != 'true' and '$(DotNetBuildVertical)' != 'true'"/> | ||
|
Comment on lines
302
to
+308
There was a problem hiding this comment. Wouldn't it make more sense to just not pass the There was a problem hiding this comment. Maaybe? I'm not sure why that was put there in the first place so I'm skeptical of removing it at this point. |
||
| </Target> | ||
| </Project> | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| <# | ||
| .SYNOPSIS | ||
| Creates a git clone of <source> into <dest> directory, optionally saving WIP changes. | ||
| .PARAMETER Source | ||
| The source Git directory | ||
| .PARAMETER Dest | ||
| The destination Git directory. Created if doesn't exist. | ||
| .PARAMETER Clean | ||
| If dest directory already exists, delete it. | ||
| .PARAMETER CopyWip | ||
| Transfer most types of uncommitted change into the destination directory. Useful for dev workflows. | ||
| #> | ||
|
|
||
| param ( | ||
| [Parameter(Mandatory=$true)] | ||
| [string]$Source, | ||
| [Parameter(Mandatory=$true)] | ||
| [string]$Dest, | ||
| [switch]$Clean, | ||
| [switch]$CopyWip | ||
| ) | ||
|
|
||
| if (-not (Test-Path $Source -PathType Container)) { | ||
| Write-Host "-Source not a directory: $Source" | ||
| exit 1 | ||
| } | ||
|
|
||
| Write-Host "Cloning repository at: $Source -> $Dest ..." | ||
|
|
||
| if (Test-Path $Dest) { | ||
| Write-Host "Destination already exists!" | ||
| if (Test-Path $Dest -PathType Leaf) { | ||
| Write-Host "Existing destination is a regular file, not a directory. This is unusual: aborting." | ||
| exit 1 | ||
| } | ||
| elseif ($Clean) { | ||
| Write-Host "Clean is enabled: removing $dest and continuing..." | ||
| Remove-Item -Path $Dest -Recurse -Force | ||
| } | ||
| else { | ||
| Write-Host "Clean is not enabled: aborting." | ||
| exit 1 | ||
| } | ||
| } | ||
|
|
||
| Push-Location -Path $Source | ||
|
|
||
| if (-not (Test-Path $Dest)) { | ||
| New-Item -Path $Dest -ItemType Directory | Out-Null | ||
|
|
||
| if ($CopyWip) { | ||
| # Copy over changes that haven't been committed, for dev inner loop. | ||
| # This gets changes (whether staged or not) but misses untracked files. | ||
| $stashCommit = (git stash create) | ||
|
|
||
| if ($stashCommit) { | ||
| Write-Host "WIP changes detected: created temporary stash $stashCommit to transfer to inner repository..." | ||
| } | ||
| else { | ||
| Write-Host "No WIP changes detected..." | ||
| } | ||
| } | ||
|
|
||
| Write-Host "Creating empty clone at: $Dest" | ||
|
|
||
| # Combine the results of git rev-parse --git-dir with the "shallow" subdirectory | ||
| # to get the full path to the shallow file. | ||
|
|
||
| $shallowFile = Join-Path (git rev-parse --git-dir) "shallow" | ||
|
|
||
| if (Test-Path $shallowFile) { | ||
| Write-Host "Source repository is shallow..." | ||
| if ($stashCommit) { | ||
| Write-Host "WIP stash is not supported in a shallow repository: aborting." | ||
| exit 1 | ||
| } | ||
|
|
||
| # If source repo is shallow, old versions of Git refuse to clone it to another directory. First, | ||
| # remove the 'shallow' file to trick Git into allowing the clone. | ||
| $shallowContent = Get-Content -Path $shallowFile | ||
| Remove-Item -Path $shallowFile | ||
|
|
||
| # Then, run the clone: | ||
| # * 'depth=1' avoids encountering the leaf commit in the shallow repo that points to a parent | ||
| # that doesn't exist. (The commit marked "grafted".) Git would fail here, otherwise. | ||
| # * '--no-local' allows a shallow clone from a git dir on the same filesystem. This means the | ||
| # clone will not use hard links and takes up more space. However, since we're doing a shallow | ||
| # clone anyway, the difference is probably not significant. (This has not been measured.) | ||
| # * '-c protocol.file.allow=alwyas allows cloning from the local filesystem even on newer versions | ||
| # of git which have disabled this for security reasons. | ||
| git clone -c protocol.file.allow=always --depth=1 --no-local --no-checkout "$Source" "$Dest" | ||
|
There was a problem hiding this comment. When native executables calls fail, the error is not propagated in PowerShell (unless I'm missing some external setup) and $LASTEXITCODE has to be checked. Do we have a solution for this in arcade (such as a wrapper function for calling native binaries that checks the LASTEXITCODE and throws if non-zero) or that's something that we'd need to do yet? |
||
|
|
||
| # Restore the 'shallow' file | ||
| Set-Content -Path $shallowFile -Value $shallowContent | ||
| } else { | ||
| git clone -c protocol.file.allow=always --no-checkout "$Source" "$Dest" | ||
| } | ||
|
|
||
| echo "Checking out sources..." | ||
| # If no changes were stashed, stashCommit is empty string, and this is a simple checkout. | ||
| git -C $Dest checkout $stashCommit | ||
| echo "Clone complete: $Source -> $Dest" | ||
|
|
||
| } | ||
|
|
||
| Pop-Location | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't look right. The same switch exists in the build.sh file for Unix platforms but does something completely different:
arcade/eng/common/build.sh
Lines 22 to 24 in 7245f17
What does "source build" infra mode mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this for completeness. It would end up being "windows build without prebuilts". I can remove it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. Let's remove this until there's a need and until it works with the existing infrastructure.