Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub #42026

Open
akoeplinger opened this issue Jul 8, 2024 · 9 comments
Assignees
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Milestone

Comments

@akoeplinger
Copy link
Member

akoeplinger commented Jul 8, 2024

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build Definition Test Pull Request
957738 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46983
957373 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46965
957429 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46971
957428 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46972
957416 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46970
957432 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46973
957414 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46968
957415 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46969
956347 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46920
956314 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46233
956108 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46932
956261 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46945
956205 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46942
956123 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46938
956101 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46933
956116 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46935
956121 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46936
956238 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46944
956177 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46926
956120 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46937
955181 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46803
953803 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46873
953652 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46868
953660 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46837
953655 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46860
953649 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46871
953646 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46873
953106 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46759
953102 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46837
953098 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46863
952630 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46861
952389 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #42861
952195 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46846
952313 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46837
952219 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46851
952209 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46849
952181 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46845
952251 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46837
952214 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46759
936288 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46254
935366 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46429
936159 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46371
936153 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46281
936140 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #44953
936125 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46327
935929 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46353
935565 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46433
935902 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46441
935782 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46439
935623 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46431
935722 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46370
935828 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46261
935795 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46195
935787 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46356
935738 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46361
935687 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46353
935622 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46266
935616 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46435
935378 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46430
935549 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46428
934380 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46409
935525 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46408
934464 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46358
935373 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #44843
934459 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46417
934441 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46414
935118 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46313
935357 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46411
935124 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46387
934463 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46266
935035 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46370
934775 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46281
934766 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46327
934667 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46419
934531 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46290
934424 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46356
934417 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46410
934374 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46336
934486 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46261
934448 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.MultiArchStillAllowsSingleRID #46416
934469 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46319
934328 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46407
934435 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46415
934431 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #46413
934379 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46315
934416 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46411
934420 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46412
933380 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46361
932961 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46358
932962 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46319
933991 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46366
933804 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46346
933925 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46377
933902 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46394
933866 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46254
933593 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46392
933816 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46368
933795 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46401
933789 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46399
933506 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46390
Displaying 100 of 204 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 27 204
@akoeplinger
Copy link
Member Author

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@dotnet/sdk-container-builds-maintainers

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@nagilson
Copy link
Member

Bumping this thread since it's affecting more PRs

@nagilson
Copy link
Member

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

@baronfel
Copy link
Member

Could be fixed at a root cause level by dotnet/dnceng#3389

@MichaelSimons
Copy link
Member

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

@nagilson
Copy link
Member

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

@marcpopMSFT
Copy link
Member

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.

@marcpopMSFT marcpopMSFT added this to the 10.0.1xx milestone Dec 18, 2024
@marcpopMSFT marcpopMSFT added the Area-Containers Related to dotnet SDK containers functionality label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Projects
None yet
Development

No branches or pull requests

6 participants