Release 5.10.2 not starting any servers

[lionslair]

The latest release https://github.com/philips-labs/terraform-aws-github-runner/releases/tag/v5.10.2 is not starting any servers. 5.10.1 did but this one just is not starting a server.

V5.10.1 worked fine.

I have destroyed and recreated the terraform multiple no luck.
I deleted the termination-watcher.zip and added it back but no difference. I don't actually fully understand what this is doing.

However This is not starting any servers for me on 5.10.2 ... 5.10.1 was fine

[ryzr]

Suddenly started having issues, though was running 5.9. Not sure if related, but noticed this in our scale-up lambda logs, shortly after the "Received event" log.

"Ignoring error: response.json(...).catch is not a function"

EDIT: actually, I've only been getting the above error since upgrading to 5.10.x. Reverting to see what happens. My issue on 5.9 was likely something else.

EDIT2: response.json issue gone after reverting to 5.9.0. Curious if you find similar logs to above once you're all wiped/restored.

[lionslair]

I have now got myself into such a state trying to wipe everything from aws and recreate fresh.

I keep getting errors like The specified log group already exists

EntityAlreadyExists: Instance Profile *****-gh-ci-runner-profile already exists

EntityAlreadyExists: Instance Profile *****-gh-ci-runner-profile already exists

[lionslair]

Suddenly started having issues, though was running 5.9. Not sure if related, but noticed this in our scale-up lambda logs, shortly after the "Received event" log.

"Ignoring error: response.json(...).catch is not a function"

EDIT: actually, I've only been getting the above error since upgrading to 5.10.x. Reverting to see what happens. My issue on 5.9 was likely something else.

EDIT2: response.json issue gone after reverting to 5.9.0. Curious if you find similar logs to above once you're all wiped/restored.

Yes I have the same error in that log

    "level": "WARN",
    "message": "Ignoring error: response.json(...).catch is not a function",
    "service": "service_undefined",

[lionslair]

My uneducated guess is maybe this change
v5.10.1...v5.10.2#diff-4eccdb723f617ab228bd897ae3b78dba1fb4a779f818efa21e55ccb75b06113dL19

I have rolled back to 5.10.1 and servers are starting again. However no jobs are picked up.

[sykhro]

Same issue here.

[lionslair]

I have finally got things back up and working using v5.10.0 Last one I know worked fine.

I made a total mess of the system. Also the AMI user I was using within the instance images did not have all the permissions it needed and so instances were starting then shutting down again.

Once it had the permission to read from the s3 bucket download the github runner then things started to work again. When I cleared things out is where it really got messed up.

[espizo]

We were seeing the same thing, downgrading to 5.10.0 fixed the problem.

We have also been seeing this in lambdas across several versions, although things appears to be working:
"service": "service_undefined",

[joemiller]

Ran into this after upgrading v5.10.0 -> v5.10.2 as well. Instances would not spin up anymore. Rolled back to v5.10.0 and things seem to be working again.

[rsavage-nozominetworks]

I can confirm I am seeing the same issues. I upgraded both my runner code and lambdas from 5.9.0 to 5.10.2 and started seeing the problem with all of my scale-up lambdas (see below). If I downgrade just the lambdas from 5.10.2 to 5.9.0 they work. I have tried this twice (upgrade and downgrade) - the results are the same.

Scale Up Lambdas Errors

{
    "level": "WARN",
    "message": "Ignoring error: response.json(...).catch is not a function",
    "service": "service_undefined",
    "timestamp": "2024-05-01T14:03:59.036Z",
    "xray_trace_id": "1-66324bcd-b91f32230d4f1c5052a29617",
    "region": "us-east-1",
    "environment": "<REDACTED",
    "aws-request-id": "d9b95d22-1a54-59ca-9a20-038e545cb1f1",
    "function-name": "<REDACTED>-scale-up",
    "module": "lambda.ts"
}

This seems like a wide spread issue as I'm not the only one in the community experiencing this problem.

[npalm]

Thanks for reporting, we going to dig in further