Apply arcade-powered source-build patches

[crummel]

Bug

Fixes: NuGet/Home#11060
Fixes: dotnet/source-build#2311

Regression? Last working version:

Description

1. Don't use PublicApiAnalyzer on source-build

• We got a couple errors here about Microsoft.CodeAnalysis.PublicApiAnalyzers.DeclarePublicApiAnalyzer not being able to find src/NuGet.Core/NuGet.Common/PublicAPI/net6.0/PublicAPI.Shipped.txt. This patch removes the analyzer for source-build scenarios, which is in keeping in general for source-build - analyzers tend to bring in weird prebuilts or trigger on errors because of source-build's different environment.

2. Update netcoreapp TFMs to 6.0

• This is one that we're working toward not needing. Source-build has a historical problem of only having one SDK and runtime available, and things both above and below NuGet in the source-build stack depends on only building for the one corresponding TFM. In the long-term, we want all of these to use targeting packs and not have to change TFMs for anyone but this is the short-term fix that we've been using. This is just updated from net5 to net6.

3. Use source-built Microsoft.Web.Xdt & System.ComponentModel.Composition

• This is another one carried forward from 5.0. Source-build only produces one version of each package and expects that all the repos will use them, so this lets source-build override a couple package versions that were outside the normal PackageVersions.props flow.

4. Build script changes to support ArPow from tarball.

• This updates the source-build wrapper scripts to account for a couple issues we found bringing NuGet into the overall build. The overall build uses some different argument passing and dotnet and Arcade paths so we needed to adjust those.

5. Upgrade to .NET 6 SDK.

• Source-build builds with a 6.0 SDK and there are some new warnings (which become errors because of TreatWarningsAsErrors=true):
  • CS1998: This async method lacks 'await'
  • CA1416: Validate platform compatibility (some probably unreachable code isn't valid for all platforms)
• This also updates the "if NET5" checks to "if NET5 or NET6" to account for using the 6.0 SDK.

6. Build NuGet.Frameworks for netstandard as well for VSTest.

• VSTest depends on having netstandard2.0 bits for NuGet.Frameworks so we have to add it back in. I tried including netstandard2.0 as a TFM to build for in patch 2, but it doesn't work for all projects so I did this targeted fix instead. This is another one that will go away as we work toward eliminating all the TFM-finagling we currently do.

7. NuGet.Build.Tasks.Pack source-build support

• Source-build doesn't support ILMerge so we have to include the individual DLLs in this case. This was done in source-build 2.1 and 3.1 but lost somewhere between 5.0 and now so Michael restored this patch.

PR Checklist

• PR has a meaningful title

• PR has a linked issue.

• Described changes

• Tests

  • Automated tests added
  • OR
  • Test exception
  • OR
  • N/A

• Documentation

  • Documentation PR or issue filled
  • OR
  • N/A

[nkolev92]

Hey @crummel

Can you please add desriptions for the all the changes and the motivation?

[crummel]

@nkolev92 I updated the description. Let me know if you have any questions or want to chat about them. Thanks!

[nkolev92]

Thanks, the description made is super easy to follow @crummel

A couple of little things.

[crummel]

In addition to addressing your comments I added a patch from Michael's work to update source-build to 6.0 RC, ("NuGet.Build.Tasks.Pack source-build support") which handles source-build missing ILMerge by including the individual DLLs instead of the merged one. Thanks for the review!

[zivkan]

@crummel sorry for the slow response/review. Looking at the CI logs, it failed the source-build, linux, and mac jobs because .NET 6 SDK isn't installed, hence it can't target net6.0.

[zivkan]

Sorry again for the slow review. The build still doesn't work:

/home/vsts/work/1/s/eng/source-build/build.sh: line 72: ../../cli/dotnet-install.sh: No such file or directory

This doesn't immediately fail the build, so it goes on to try to build with the system-installed dotnet, which fails because the .NET 6 SDK isn't installed.

$repo/cli/dotnet-install.ps1 is downloaded by build.ps1, but not eng/source-build/build.sh

[zivkan]

The source build job is still failing: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5258934&view=logs&j=cb98efd6-7062-56ef-f642-0176540bbe0a&t=d42c45d2-f6c8-5ae2-d854-abf6f8d78853

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 54544  100 54544    0     0   352k      0 --:--:-- --:--:-- --:--:-- 2959k
/home/vsts/work/1/s/eng/source-build/build.sh: line 82: ../../cli/dotnet-install.sh: No such file or directory
Microsoft (R) Build Engine version 16.11.0+0538acc04 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/share/dotnet/sdk/5.0.401/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/5.0.401/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/5.0.401/dotnet.dll -maxcpucount -verbosity:m /bl:/home/vsts/work/1/s/eng/source-build/../../artifacts/source-build/self/log/source-build.binlog /p:Configuration= /p:DotNetBuildFromSource=true /p:ArcadeBuildFromSource=true /p:RepoRoot=/home/vsts/work/1/s/eng/source-build/../../ /home/vsts/work/1/s/eng/source-build/source-build.proj
  Check for patches in /home/vsts/work/1/s/eng/source-build/../../eng/source-build-patches/*.patch
  Running source-build PreBuild target
  Running source-build InnerBuild target
  Restoring for XPLAT
  Determining projects to restore...
/usr/share/dotnet/sdk/5.0.401/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(141,5): error NETSDK1045: The current .NET SDK does not support targeting .NET 6.0.  Either target .NET 5.0 or lower, or use a version of the .NET SDK that supports .NET 6.0. [/home/vsts/work/1/s/src/NuGet.Core/NuGet.Commands/NuGet.Commands.csproj]
/usr/share/dotnet/sdk/5.0.401/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(141,5): error NETSDK1045: The current .NET SDK does not support targeting .NET 6.0.  Either target .NET 5.0 or lower, or use a version of the .NET SDK that supports .NET 6.0. [/home/vsts/work/1/s/src/NuGet.Core/NuGet.Credentials/NuGet.Credentials.csproj]

[zivkan]

I don't know if it's intentional, but it's caused our linux test job to fail with the same error, .NET 6.0 SDK not installed. I'm guessing the Mac job would have failed as well, if/when it runs.

Ideally since NuGet is only targeting the .NET 6.0 SDK now, this branch will never be inserted into .NET 5.0 SDK, we (someone in the NuGet team) should update the build. We just have too many other priorities at the moment.

[nkolev92]

I don't know if it's intentional, but it's caused our linux test job to fail with the same error, .NET 6.0 SDK not installed. I'm guessing the Mac job would have failed as well, if/when it runs.

When I was reviewing the PR, I was under the impression that only source build should be setting net6.0.

Maybe we're missing something?

[zivkan]

Yeah, my brain kept thinking about it after I posted while trying to do other work, and I don't think it's good for this source-build requirement to cause our other jobs/tests to start using a different TFM, as it can cause builds to break when new versions of .NET enable new code analyzers (and we use WarningsAsErrors, so it will break us), plus our dotnet.integration.tests project patches a .NET SDK with the current build of NuGet and runs some dotnet commands to test the integration, so I can imagine there's significantly more risk if we patch a .NET 5 SDK with NuGet assemblies targeting .NET 6.

Therefore, I think it's a bug that the source-build TFM re-targeting to .NET 6 has "leaked" to our "normal" Linux and Mac CI jobs.

[crummel]

I think the latest set of changes addresses your concerns with the Linux non-source-build build. Windows tests failed on the PR but I restarted them and I don't think it's likely that my changes are the cause - I'll keep an eye on them.

[crummel]

@zivkan @nkolev92 wanted to make sure you saw this, I was hoping to get it in before the cutoff tomorrow.

[heng-liu]

I don't know if it's intentional, but it's caused our linux test job to fail with the same error, .NET 6.0 SDK not installed. I'm guessing the Mac job would have failed as well, if/when it runs.

Ideally since NuGet is only targeting the .NET 6.0 SDK now, this branch will never be inserted into .NET 5.0 SDK, we (someone in the NuGet team) should update the build. We just have too many other priorities at the moment.

Hi @zivkan, is this caused by the old copy of dotnet-install.sh we kept? Pls see #4289 (comment) for more details.

[zivkan]

@heng-liu no, the previous issue was caused by this PR changing the TFM for all "xplat" builds, rather than only source-build builds.

However, there are now 9 tests on Mac failing, all of them to do with signed packages. I don't see how it's related to this PR's changes, but it looks like @crummel re-ran the Mac tests 7 times, and it keeps failing with the 9 signing tests, and our other builds are not failing these tests on Mac, so it does not appear to be an infrastructure issue. As the team's signed packages expert, can you take a look?

[heng-liu]

Hi @zivkan and @crummel, sorry that I just saw this. The 9 tests on Mac are disabled in dev a few weeks ago (tracking issue: NuGet/Home#11178), so rebasing from dev should work.
Hi @crummel, could you try to rebase from dev and re-run the tests? Thanks!

[crummel]

I rebased and that seems to have fixed the Mac tests. The Apex tests failed so I kicked those, but the VS EndToEnd test machine deployment is failing, any ideas?

[zivkan]

I see a build for your other PR, the BAR publishing, but not this PR.

The first attempt for E2E tests failed for what I think is a Dartlab error. We already contacted the team, but haven't yet got a response. Attempts 2 onwards failed because you retried only the failed job, not the entire stage, and after the first attempt failed, the temporary assets were deleted, so trying to re-deploy the VM without running the configure VM job failed because the configuration file couldn't be found.

[crummel]

@zivkan What has NuGet been doing in regards to merging fixes to the release branch? Should I make a new PR to get this into release-6.0.x or will it get picked up automatically or is it too late? Thanks!

[zivkan]

@crummel I'll cherry-pick it into the release branch. We might not do any more insertions from that branch though, so the NuGet commit hash that corresponds to the binaries that ship in the .NET 6.0.100 SDK will not contain these changes.