Request larger runner support for GitHub Actions.

[austinlparker]

The Demo SIG has encountered significant challenges in regards to build time for releases with standard GitHub Actions runners. A full x86 build takes an hour currently; This is obviously challenging on its own, but we would like to perform multi-arch builds to resolve issues with the demo running on Apple Silicon. These multi-platform builds take over 4 hours. I'll provide more details below.

Per this page, Teams/Enterprise plans have access to larger runners if requested. Would we be able to request these runners in order to see if throwing more resources at the problem helps?

--

Our build times are significantly impacted by the overhead required for gRPC builds in several languages (C++, PHP, etc.). We aren't able to use local caching, because runners are ephemeral. We also are required to build each stage completely in the target architecture (can't build/share a x86 gRPC layer with an arm64 binary layer for example), so we can't do platform targeting in the Dockerfile itself to 'quickly' build underlying layers then only emulate arm for the final compilation step.

The only other thing we can really think to do would be to refactor the Dockerfiles to build + publish base images for our 'slow' builds, but since the base image would need OpenTelemetry libraries itself in the case of C++, I feel like we'd just be regenerating it every month anyway?

[austinlparker]

@open-telemetry/technical-committee Are y'all the GitHub admins? Would you be able to enable larger runners?

[mtwo]

@bogdandrutu @carlosalberto the TC owns GitHub permissions, correct? Who should Austin work with on the TC?

[dyladan]

I think @SergeyKanzhelev was in discussions with CNCF about the enterprise account. He may have some details

[trask]

The initial discussions (~2 years ago) were directly with GitHub, though I'm not sure if things have changed since then. I have the original GitHub contacts if needed.

I'd be interested to look over the admin settings for the GitHub org first, to see if there's any mention already of cost limits in place on the org. Other than the cost question, it looks doesn't look difficult to configure large runners based on link @austinlparker provided above, and I generally support the idea of throwing hardware at the problem at least for the time being.

If anyone from the TC wants to ping me on slack I can help with this.

[yurishkuro]

We also have an outstanding invitation to join CNCF Enterprise account, which is likely to have higher limits on the runners.

[bogdandrutu]

This is what I see in the "runners" I saw that I can enable larger once maybe, so you may be able to run on these:

[bogdandrutu]

Also we already joined CNCF Enterprise, so these are part of that CNCF enterprise. But they cannot be used apparently, I can manually enable some "repos" to use them looks like. If anyone interested in working on this, ping me on slack and we can hack around:

[bogdandrutu]

Access Log:

• Created a new runner with ubuntu 16 cores, price for this is 0.064 per minute of usage, limited to max 15 at a time.
• Created a runner group with only demo repo.

Looks like it works, but we need to pay attention to cost.

[bogdandrutu]

I disabled this experiment, we consumed only for the first build 634minutes * 0.064 => 40.576$

This is too much to justify the pros (build was still failing after 5h). Happy to re-enable if we get some info from CNCF about the budget we have and how much we can use.

[SergeyKanzhelev]

CC: @jeefy to comment on "allowance" for the project

[trask]

I've opened a CNCF Service Desk ticket to inquire about allowance / costs.

[dyladan]

@trask assigning you since you made the ticket

[RobertKielty]

@jeefy I've picked up the call on the Service Desk. Let's discuss this today.

@bogdandrutu thank you for your prudence. I'm reviewing the referenced GH Actions at the top of this issue will collate them so that I can go over them with Jeefy later today. I'll get back to you with any questions that I have.

[RobertKielty]

@trask I Just updated the Service Desk request.

Please check the new Github Action runners available to the projects and let me know when you first start making use of them.

[trask]

FYI we now have access to Arm GitHub runners, see #1821, and you can open a repo maintenance request to get access

[trask]

@austinlparker will that resolve this issue?

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[trask]

GitHub just recently announced official GitHub Arm runners, it would be great if someone wants to try these out (it may require @open-telemetry/technical-committee to

https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

GitHub is ecstatic to unveil ArmⓇ-based Linux and Windows runners for GitHub Actions are now in Public Beta.

OpenTelemetry is under the CNCF's GitHub Enterprise account, so we should have access to this feature

These runners are available to our customers on our GitHub Team and Enterprise Cloud plans.

[austinlparker]

If someone could check and turn these on for the demo repo, that’d be swell

…

On Fri, Jun 14 2024 at 11:08 AM, Trask Stalnaker < ***@***.*** > wrote: GitHub just recently announced official GitHub Arm runners, it would be great if someone wants to try these out (it may require @open-telemetry/technical-committee ( https://github.com/orgs/open-telemetry/teams/technical-committee ) to https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/ > > > GitHub is ecstatic to unveil ArmⓇ-based Linux and Windows runners for > GitHub Actions are now in Public Beta. > > OpenTelemetry is under the CNCF's GitHub Enterprise account, so we should have access to this feature > > > These runners are available to our customers on our GitHub Team and > Enterprise Cloud plans. > > — Reply to this email directly, view it on GitHub ( #1281 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AA7S5RGKQMRUNJLN7SPVGADZHMBODAVCNFSM6AAAAAARTDVAROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYGIZTQOBRHA ). You are receiving this because you were mentioned. Message ID: <open-telemetry/community/issues/1281/2168238818 @ github. com>

[trask]

If someone could check and turn these on for the demo repo, that’d be swell

done! see #2156 for more details