This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
The runner has received a shutdown signal. #6709
Comments
|
@JakubMosakowski we cannot do any investigation without additional info. |
|
The interesting part is that it doesn't seem to be related to any of our changes. I created a branch that is reverted by the last X commits (to the point in history where our builds were smooth) and they are not passing anymore. |
|
We are also seeing this after upgrading our self hosted runners from 20.04 to 22.04 with no other seemingly related changes. Do the 22.04 runners have more conservative limits even when using self hosted? |
|
The same happens to us in private repo. We didn't do any significant changes to workflows. |
|
Hi @ihor-panasiuk95, please send me a links to workflow runs both with positive and negative results. |
|
@erik-bershel will you be able to visit them taking into account that they are in private repo? |
|
@ihor-panasiuk95 it's not a problem. There is no need to check what is going on in your private repository in the first step. I want to check the load on agents and compare successful and failed jobs. If the information is not enough, then we will discuss the repro-steps. For example: https://github.com/owner/repo/actions/runs/runID or https://github.com/erik-bershel/erik-tests/actions/runs/3680567148. |
|
@erik-bershel |
|
I find that this issue only occurs when using |
Previously Github Actions was killing us resulting in the Publish-Binary job failing with the following error message: The runner has received a shutdown signal. Related: actions/runner-images#6709
## Description Previously Github Actions was killing (probably because gorelease was using up quite a bit of resources) the process resulting in the Publish-Binary job failing with the following error message: ``` The runner has received a shutdown signal. ``` Related: actions/runner-images#6709 <!-- Insert PR description--> ## Does this PR need a docs update or release note? - [ ] ✅ Yes, it's included - [ ] 🕐 Yes, but in a later PR - [x] ⛔ No ## Type of change <!--- Please check the type of change your PR introduces: ---> - [ ] 🌻 Feature - [ ] 🐛 Bugfix - [ ] 🗺️ Documentation - [ ] 🤖 Test - [x] 💻 CI/Deployment - [ ] 🐹 Trivial/Minor ## Issue(s) <!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. --> * fixes #2014 ## Test Plan <!-- How will this be tested prior to merging.--> - [x] 💪 Manual - [ ] ⚡ Unit test - [ ] 💚 E2E
Workaround for actions/runner-images#6709 to address the flaky CI failures we've been experiencing.
Workaround for actions/runner-images#6709 to address the flaky CI failures we've been experiencing.
Try changing ubuntu version as per actions/runner-images#6709
Task/Issue URL: https://app.asana.com/0/488551667048375/1205793949269201/f ### Description Uses ubuntu-20.04 instead of latest to avoid issues on ci More info: actions/runner-images#6709 ### Steps to test this PR _Feature 1_ - [ ] - [ ] ### UI changes | Before | After | | ------ | ----- | !(Upload before screenshot)|(Upload after screenshot)|
Task/Issue URL: https://app.asana.com/0/488551667048375/1205793949269201/f ### Description Uses ubuntu-20.04 instead of latest to avoid issues on ci More info: actions/runner-images#6709 ### Steps to test this PR _Feature 1_ - [ ] - [ ] ### UI changes | Before | After | | ------ | ----- | !(Upload before screenshot)|(Upload after screenshot)|
## What changed? <!-- Describe what has changed in this PR --> Downgrading Github `All Tests` actions to use `ubuntu-20.04` instead of `ubuntu-latest` ## Why? <!-- Tell your future self why have you made these changes --> It seems we started hitting an issue with Github's action runners. These errors seem to appear most often when runners are killed because they are using too much CPU or memory, but we have been unable to conclusively say what might have caused an increase in resource usage. Downgrading the ubuntu version worked for others and seems to be working for us. For more info: * actions/runner-images#6709 * actions/runner-images#7188 * Users reported a similar issue with ubuntu-24.04 runners (not yet GA): actions/runner-images#9848 ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> Reran all PR check jobs several times ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
GitHub action builds started to randomly fail with a reason: "The runner has received a shutdown signal.". The issue [1] and discussion [2] points to a workaround: replace a "ubuntu-latest" with "ubuntu-22.04" and it started to work. The patch replaces "ubuntu-latest" with "ubuntu-22.04" in CFL workflows. 1. actions/runner-images#6709 2. actions/runner-images#7188
GitHub action builds started to randomly fail with a reason: "The runner has received a shutdown signal.". The issue [1] and discussion [2] points to a workaround: replace a "ubuntu-latest" with "ubuntu-22.04" and it started to work. The patch replaces "ubuntu-latest" with "ubuntu-22.04" in CFL workflows. 1. actions/runner-images#6709 2. actions/runner-images#7188
GitHub action builds started to randomly fail with a reason: "The runner has received a shutdown signal.". The issue [1] and discussion [2] points to a workaround: replace a "ubuntu-latest" with "ubuntu-20.04" and it started to work. Note that the problem reproduce with "ubuntu-22.04" as well. The patch replaces "ubuntu-latest" with "ubuntu-20.04" in CFL workflows. 1. actions/runner-images#6709 2. actions/runner-images#7188
GitHub action builds started to randomly fail with a reason: "The runner has received a shutdown signal.". The issue [1] and discussion [2] points to a workaround: replace a "ubuntu-latest" with "ubuntu-20.04" and it started to work. Note that the problem reproduce with "ubuntu-22.04" as well. The patch replaces "ubuntu-latest" with "ubuntu-20.04" in CFL workflows. 1. actions/runner-images#6709 2. actions/runner-images#7188
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Description
Since yesterday, our GitHub action builds started to randomly fail (we didn't change anything in our configuration). The error is not very precise, unfortunately.
The process is stopped in random stages of the build (but always after at least 15 minutes or so). Even if the build passes it takes much longer than before (~25 min clean build to ~35 min now).
Sometimes before the shutdown signal, there is also such log:
Idle daemon unexpectedly exit. This should not happen.Workflow passes normally on builds that are shorter (for example those from cache).
Platforms affected
Runner images affected
Image version and build link
Image: ubuntu-22.04
Version: 20221127.1
Current runner version: '2.299.1'
Unfortunately, it happens on the private repo.
Is it regression?
No
Expected behavior
Job should pass
Actual behavior
Job fails
Repro steps
Looks similar to: #6680
The text was updated successfully, but these errors were encountered: