Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Builds timing out on CI #38964

Closed
safern opened this issue Jul 8, 2020 · 3 comments
Closed

Builds timing out on CI #38964

safern opened this issue Jul 8, 2020 · 3 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
5.0.0

Comments

@safern
Copy link
Member

safern commented Jul 8, 2020

We're seeing multiple builds canceled due to "timeouts" when they're not really timeouts. This seems like a bug in AzDo that it is considering the "queue" time and time spent waiting for an agent as part of the timeout time.

##[error]The job running on agent NetCorePublic-Pool 172 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
Pool: NetCorePublic-Pool
Agent: NetCorePublic-Pool 172
Started: Today at 1:54 PM
Duration: 15m 2s

As seen, the duration was really 15m 2s which is not over 60 mins.

Runtime build

Build Kind Timeline Record
721340 PR #38938 CoreCLR Product Build Windows_NT x64 release
721340 PR #38938 CoreCLR Product Build Windows_NT arm64 checked
721340 PR #38938 CoreCLR Product Build Windows_NT x86 release
721425 PR #38952 CoreCLR Product Build Windows_NT x64 release
721425 PR #38952 CoreCLR Product Build Windows_NT arm release
721425 PR #38952 CoreCLR Product Build Windows_NT arm64 release
721425 PR #38952 CoreCLR Product Build Windows_NT x86 release
721535 PR #38765 CoreCLR Product Build OSX x64 checked
721294 PR #38948 Mono Product Build Linux x64 debug
721294 PR #38948 CoreCLR Product Build Linux_musl x64 release
721294 PR #38948 CoreCLR Product Build Linux_musl x64 checked
721294 PR #38948 CoreCLR Product Build Linux x64 release
721294 PR #38948 CoreCLR Product Build Linux arm release
721294 PR #38948 Formatting Linux x64
721294 PR #38948 CoreCLR Product Build FreeBSD x64 release
721294 PR #38948 CoreCLR Product Build Linux arm64 release
721294 PR #38948 CoreCLR Product Build Linux_musl arm64 release
721294 PR #38948 Build Linux arm Debug AllSubsets_Mono
721294 PR #38948 Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721294 PR #38948 Mono Product Build Linux arm64 debug
721294 PR #38948 Mono Product Build Linux x64 release
721294 PR #38948 Build Linux_musl x64 Release AllSubsets_Mono
721294 PR #38948 Mono Product Build Linux arm64 release
721294 PR #38948 CoreCLR Product Build Linux arm64 checked
721294 PR #38948 CoreCLR Product Build Linux_musl arm64 checked
721294 PR #38948 CoreCLR Product Build Linux x64 checked
721294 PR #38948 CoreCLR Product Build Linux arm checked
721294 PR #38948 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099 PR #38707 Mono Product Build Linux x64 debug
721099 PR #38707 CoreCLR Product Build Linux_musl x64 release
721099 PR #38707 CoreCLR Product Build Linux x64 release
721099 PR #38707 CoreCLR Product Build Linux arm release
721099 PR #38707 CoreCLR Product Build FreeBSD x64 release
721099 PR #38707 CoreCLR Product Build Linux arm64 release
721099 PR #38707 CoreCLR Product Build Linux_musl arm64 release
721099 PR #38707 Build Linux arm Debug AllSubsets_Mono
721099 PR #38707 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099 PR #38707 Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721099 PR #38707 Mono Product Build Linux arm64 debug
721099 PR #38707 Mono Product Build Linux x64 release
721099 PR #38707 Build Linux_musl x64 Release AllSubsets_Mono
721099 PR #38707 Mono Product Build Linux arm64 release
721062 PR #38935 CoreCLR Product Build Linux_musl x64 release
721062 PR #38935 CoreCLR Product Build Linux x64 release
721062 PR #38935 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721160 PR #37971 CoreCLR Product Build Windows_NT x86 checked
721161 PR #38884 CoreCLR Product Build Windows_NT x64 release
721161 PR #38884 CoreCLR Product Build Windows_NT arm release
721161 PR #38884 CoreCLR Product Build Windows_NT x86 release
721161 PR #38884 CoreCLR Product Build Windows_NT arm64 release
721009 PR #38815 Build Linux arm Debug AllSubsets_Mono
721042 PR #38933 CoreCLR Product Build Linux arm release
720484 PR #38893 CoreCLR Product Build OSX x64 release
719881 PR #38728 CoreCLR Product Build Windows_NT x86 checked

Evaluated 100 builds
Impacted 12 bulids
Impacted 54 jobs

This is impacting around 12% of builds of runtime and 12% of runtime-live-build.

Runtime-live-build

Build Kind Timeline Record
721591 PR #38547 Build OSX x64 release Runtime_Debug
721451 PR #38894 Build OSX x64 release Runtime_Debug
721456 PR #38616 Build Windows_NT x64 debug Mono_Libraries
721456 PR #38616 Build Windows_NT x86 release Runtime_Debug
721295 PR #38948 Build Linux x64 debug RuntimeFlavor_Mono
721295 PR #38948 Build Linux x64 debug Runtime_Release
721295 PR #38948 Build Linux x64 release SourceBuild
721264 PR #38547 Build Linux x64 debug Runtime_Release
721264 PR #38547 Build Linux x64 release SourceBuild
721270 PR #38946 Build Linux x64 debug RuntimeFlavor_Mono
721270 PR #38946 Build Linux x64 debug Runtime_Release
721241 PR #32520 Build Windows_NT x64 debug Mono_Libraries
721063 PR #38935 Build Windows_NT x64 debug Mono_Libraries
721063 PR #38935 Build Linux x64 debug RuntimeFlavor_Mono
721063 PR #38935 Build Linux x64 debug Runtime_Release
721063 PR #38935 Build Linux x64 release SourceBuild
721100 PR #38707 Build Linux x64 debug RuntimeFlavor_Mono
721100 PR #38707 Build Linux x64 debug Runtime_Release
721043 PR #38933 Build Linux x64 release SourceBuild
721039 PR #37159 Build Linux x64 debug Runtime_Release
720680 PR #38917 Build OSX x64 release Runtime_Debug

Evaluated 100 builds
Impacted 12 bulids
Impacted 21 jobs

@MattGal is looking into this and opening a ticket on AzDo

cc: @dotnet/runtime-infrastructure
@safern safern added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jul 8, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-Infrastructure-coreclr untriaged New issue has not been triaged by the area owner labels Jul 8, 2020
@ghost
Copy link

ghost commented Jul 8, 2020

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

@safern safern removed the untriaged New issue has not been triaged by the area owner label Jul 8, 2020
@safern safern added this to the 5.0.0 milestone Jul 8, 2020
@jashook jashook mentioned this issue Jul 8, 2020
Closed
@safern
Copy link
Member Author

safern commented Jul 8, 2020

FYI: @stephentoub @jkotas

@jakobbotsch jakobbotsch mentioned this issue Jul 9, 2020
Merged
@safern
Copy link
Member Author

safern commented Jul 17, 2020

I think I haven't seen this anymore. Closing

@safern safern closed this as completed Jul 17, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Projects
None yet
Milestone
5.0.0
Development

No branches or pull requests
2 participants
@safern @Dotnet-GitSync-Bot