Stop automatically deleting inactive runners

[pimhakkertgc]

Select Topic Area

Product Feedback

Body

We have runners stationed in 3 different servers. Normally we make use of these on a weekly basis, but this is not the case when I go on vacation. I normally do so for at least 15 days, twice a year. When I am not working there are usually no deployments, and as such the runners do not get activated.

We start our runners manually and do not leave them running in an automated process.

When runners are not ran for at least 14 days, Github removes them from the system and Github Actions. This is incredibly annoying and means that I have to delete the local runner directories and rebuild the runners from scratch twice a year.

Can you please not delete inactive runners? Or let us at least press a button to mark a runner to not be automatically deleted.

[ttolbol]

Agreed! This is very annoying in our company where we use self-hosted runners to deploy updates. This is triggered manually 1-2 times per month and has worked flawlessly for years, but now we have to waste time setting up runners every single time! Github have basically made runners single use for us now.

[rossbu]

make sure it's tracked, guys, and this should never be deleted for no reason, inactivity for a few month or even a bit longer is not a reason for github to do so

[CMM1305]

I agree, there should be a way of marking self-hosted runners to remain permanently unless we decide to manually delete them. I don't understand GitHub's reasoning for making this change. We should be in control of the runners we create and we decide when and if to delete them.

[JimMikola]

The automatic deletion of offline selfhosted runners needs to be configurable by the organization.

We have multiple runners that are staged to cover peaks in automation activities. The runners are in an offline state until required, at which point a low-cost runner spins up one of the staged high-cost runners. Some of the high-cost runners are offline for more than 14 days when automation workloads are low.

[tom-carbontrail]

Disappointing this hasn't been addressed by the GH team for over a year. This is also adding friction for no good reason. I agree this needs to be configurable, or at least allow a previously registered runner to "resurrect" its previous registration...

[nomeelnoj]

Also, we run self hosted code-scanning runners. We dynamically spin up the runners as needed, but if we dont have at least one runner registered with code-scanning as the label (even an offline runner will do), the process does not wait for our dynamic generation of runners and therefore it automatically picks up Github managed runners, and causes issues for us.

Can we please have a way of making sure offline runners do not get deleted after 14 days?!

[stuartjvnorton]

Even if they disabled the runner and it had to be manual re-enabled in the admin console, I wouldn't really have a problem with that. But this business of nuking it on 1 side and just leaving it hanging on the other is just a terrible solution.

[Martiix]

They even say they will be deleted after 14 days, but mine are even deleted long before that. I want to disable the whole removal part and instead remove them manually if needed, to have it controlled and on purpose. Now I just randomly discover missing runners, which is super annoying, don't even get notified that they are deleted.

This apparently took less than 3 days to be removed, and it wasn't even supposed to be inactive :( so in this case github's bug where runners become inactive without reason combined with auto-removal is very frustrating.

[Martiix]

We ended up making a hard-coded "runner-state" file with runners and labels and then made some tests in a nightly workflow that checks that the state is as expected. Annoying to have to do this, but at least now we get notified when our runners are suddenly removed. I hope that we can convert this into a stateful setup where a runner is just auto recreated if it is missing in the future. But honestly, why are they removed in the first place...

[ManuelSchenk]

Hi GH, please fix this issue! Its unacceptable behavior!

[FredeJ]

Just adding my voice to the choir here, that this is super annoying.

I've set up hardware-in-the-loop testing using GitHub runners. Just came back from Christmas vacation and suddenly one of the runners is missing, which completely stops our whole release flow as we validate firmware on all previous hardware versions and some of the old ones are in short supply.

I can't remove it, because it needs a removal token, that I can't access because it's been autoremoved. So now I may need to re-flash it or figure out where the configuration of the runner is stored. It's way more hassle than I would want.

This auto-removal "feature" basically cost me a day of work.

[MMalawski2]

Same here. To make it worse, it does not say anywhere that GH has removed runners due to inactivity. I was surprised when I came back from holidays and found half of my runners missing. Lost half day to investigate and then another to fix it.

[stuartjvnorton]

Our runner boxes are on AWS, so we have a schedule that starts them up once a week during quiet time and shuts down after an hour.
We thought that would be enough for the runner and GitHub to sort themselves out, but we still had runners removed after 2 weeks.
So now we force the issue.
Each runner has a unique label. A script is scheduled to run on machine start: it waits until the runner service is up, then fires a curl call to a GitHub action we wrote just for this, citing the unique label.
Then the action runs a Hello World type script on the agent matching the label and inactivity clock is reset.