DoNotShowDeprecationMessage_WhenNewCommandIsUsed failed in CI

[tmat]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=65642&view=ms.vss-test-web.build-test-results-tab&runId=1338148&resultId=100062&paneView=debug

Expected command to pass but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new install Microsoft.DotNet.Web.ItemTemplates::5.0.0 --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/DoNotShowDeprecationMessage_WhenNewCommandIsUsed/Home/20221027222053088\nExit Code: 103\nStdOut:\nThe following template packages will be installed:\n   Microsoft.DotNet.Web.ItemTemplates::5.0.0\n\nWarning: Microsoft.DotNet.Web.ItemTemplates::5.0.0 is not found in NuGet feeds https://pkgs.dev.azure.com/dnceng/public/_packaging/darc-pub-dotnet-aspnetcore-d50065c4/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/darc-pub-dotnet-aspnetcore-d50065c4-1/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/darc-pub-dotnet-runtime-cd2d8379/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6-transport/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7-transport/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools-transport/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json, https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries-transport/nuget/v3/index.json, https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json, https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-buildservices/nuget/v3/index.json.\nStdErr:\nMicrosoft.DotNet.Web.ItemTemplates::5.0.0 could not be installed, the package does not exist.\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n

[marcpopMSFT]

@dotnet/templating-engine-maintainers to take a look.

[vlada-shubina]

@GangWang01 could you please take a look? thank you

[GangWang01]

Actually the package could be found with the given package sources in the error message. And this issue didn't happen frequently.
From the directory path I understood the test failed on macOS. And the test failed to pull the package via Nuget. Considering recently some of our tests(that depend on Nuget to work) frequently failed on macOS with the error similar to below, it sounds like on macOS for CI Nuget client has some random connection issue.

StdErr:
Matches from template source: NuGet.org
Search failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)  

[vlada-shubina]

@dotnet/dotnet-cli I'm not sure if we are taking any action on that. These are random connectivity issues in the images. Any operation interacting with NuGet.org, Azure resources etc. may be affected. Setting up retries on all the tests that may touch external resources are not reasonable, and may not help.

I tend to not take any action at this point, unless the engineering can look into network connectivity issues for the images in general.

[marcpopMSFT]

@vlada-shubina do we know how common these connectivity issues are? Should we raise with dnceng and file an arcade issue to track it?

[nagilson]

It looks pretty common to me, I would do that.

[vlada-shubina]

@nagilson could you please clarify why do you think it's common?
I see failures only around Nov 14. During that time the problem accessing resources from Darwin occurred in many repos, and like was solved. There was no failures since that time.

[nagilson]

@vlada-shubina Ah, I said that because it had an impact on a large number of PRs at that time, and I have seen the Foo is not found in NuGet feeds issue several other times. I would try to dig up previously impacted builds but they've likely all been deleted. Anyway, it's not super high priority, but if they have capacity to investigate network connectivity issues, I think it would be nice to have.

[vlada-shubina]

We decided not to take any action on that. These are random connectivity issues in the images. Any operation interacting with NuGet.org, Azure resources etc. may be affected. Setting up retries on all the tests that may touch external resources are not reasonable, and may not help.

Closing for now, we might want to reopen if the issue will reappear.
Also we drastically decreased number of tests thanks to snapshot testing now, so issues hopefully are less likely.