Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

progress: set --cache-max-size in osbuild #812

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

mvo5
Copy link
Collaborator

@mvo5 mvo5 commented Jan 27, 2025

[draft as this needs https://github.com//pull/811 so that we have a single way to run osbuild plus can actually control its options (which images:osbuild cannot currently]

This commit allows controlling the osbuild --cache-max-size
option. By default it will set the cache to 20GiB but allows
this to be controlled by the user.

Thanks to Simon for raising this.

mvo5 added 5 commits January 27, 2025 13:52
This commit adds catpure of os.Std{out,err} when running osbuild so
that we capture the error log and can display it as part of
an error from osbuild, e.g. when osbuild itself crashes.

This gives us more accurate error reporting if anything fails
during the osbuild building.
This commit tweaks an issue that potentially an incorrect status
from osbuild would fail the build with a bad error message and
without us getting the full buildlog.
This commit adds a `OSBuildOptions` struct that can be used to
pass (optional) options to the `progress.RunOSBuild()` helper.

This make it easier to expand with more options.
This commit adds a new `BuildLog` option to the `OSBuildOptions`
that can be used to generate a streamed buildlog (e.g. to a file
or a websocket).

This will be used in `ibcli` with a new `--with-buildlog` option.
This commit allows controlling the `osbuild --cache-max-size`
option. By default it will set the cache to 20GiB but allows
this to be controlled by the user.

Thanks to Simon for raising this.
@mvo5 mvo5 requested review from achilleas-k and supakeen January 27, 2025 17:03
Copy link
Member

@supakeen supakeen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy about this; a quick note on the size. A single Fedora Minimal build takes up about 10 GiB of space so 20 GiB is indeed a decent minimum.

Note also that due to how we setup pipelines right now basically no output gets cached (at least in in the image-builder-case, I didn't verify the bootc-image-builder case yet).

@achilleas-k
Copy link
Member

Without checkpoints, osbuild will only cache sources.

@achilleas-k
Copy link
Member

I didn't verify the bootc-image-builder case yet

_, err := osbuild.RunOSBuild(manifest, store, outputDirectory, exports, nil, extraEnv, false, os.Stderr)

No checkpoints there.

Actually, we never set any checkpoints in osbuild-composer or bootc-image-builder (so far).

@supakeen
Copy link
Member

supakeen commented Jan 29, 2025

I didn't verify the bootc-image-builder case yet

_, err := osbuild.RunOSBuild(manifest, store, outputDirectory, exports, nil, extraEnv, false, os.Stderr)

No checkpoints there.

Actually, we never set any checkpoints in osbuild-composer or bootc-image-builder (so far).

It would be 'OK' to pass --checkpoint="*", something we implemented a while back I think. However; as I've said previously as well with the current way we generate pipelines nothing will actually get cached post the buildroot even when passing checkpoints.

Perhaps it'd be fine to then only pass --checkpoint="buildroot" for things that are generated by images?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants