New Monitor class with medium verbosity level

[aesteve-rh]

So for builds, osbuild has two monitors that I could see, NullMonitor, and LogMonitor. While this is nice, it would be helpful to tune down a bit the verbosity of the LogMonitor so that we only print the name of the stage and some minimal information, ideally reduce it to one line per stage.

The reasoning behind this is to make it easier to spot fails/issues, as building an image with many stages turns to be too noisy to realize an error is printed (unless the build actually failed, of course).

Ideally, we could use rich or a similar library to beatify it and add ticks to indicate whether a stage succeeded or not. Maybe even a status bar that reaches 100% when no stages are left.

I am not really seeking for someone to do this, but rather checking on the general maintainers' opinion/reception before I start implementing.

[supakeen]

@aesteve-rh Generally we've (now) been doing this higher up in the stack in the things that use osbuild; see for example how things look in bootc-image-builder, or image-builder. We've tended to live in the idea that osbuild should really not be called directly by users.

I've previously taken a look at prettifying (some) output and logging with rich, however from what I recall that stalled on the availability of libraries on our target platforms. Your idea with a separate monitor (that might not be available on some platforms) might be an easier way to get that done.

Let me tag @mvo5 since he has done (most of) the work on the progress reporting things.

As an aside; are you often executing osbuild directly and if so in what usecase?

[aesteve-rh]

I work on automotive, so I rely on automotive-image-builder which in the end calls osbuild. So to answer your question, yes, I do use osbuild, just not directly. Still, the output comes from osbuild.

One could of course argue that we could beautify the output on automotive-image-builder. But there is no good way of knowing in advance the structured information on the stages a specific manifest is going to run. So for things like e.g., having a progress bar, we would end up having to rely on a count-the-number-of-lines system with an initial estimation to know when the build is going to finish. That's why we thought on moving this to osbuild.

Let me cc @alexlarsson to this conversation.

[supakeen]

Right so to address your middle concern directly, we already do progress bars and reporting in the projects that are using osbuild; for example building an image with image-builder (or bootc-image-builder):

€ sudo image-builder build --distro=fedora-41 minimal-raw
WARN[0000] Failed to load consumer certs: no consumer key found 
[/] Image building step
[3 / 7] Pipeline build [------------------------------------------>________________________________________________________] 42.86%
[1 / 3] Stage org.osbuild.rpm [------------------------------>_____________________________________________________________] 33.33%
Message: Starting module org.osbuild.rpm

For image-builder we have an issue to support direct building of manifests. Perhaps an option for automotive-image-builder is to call image-builder after using osbuild-mpp to generate a manifest? We could keep track of the requirements automotive-image-builder has while implementing the above (which probably mostly means allowing to pass arguments to the osbuild process?).

---

Personally I'm fine with reviewing a new monitor that does progress reporting inside osbuild however it feels like a pretty large duplication of effort and possibly more divergence.

[alexlarsson]

If you feel that the right place for this is the caller of osbuild, then we can add the code to a-i-b to fake some progress bars, but it seems like osbuild is the best place to do it correctly, as it knows exactly what work it is preparing to do.

[aesteve-rh]

Having osbuild handling the output sounds like removing duplication, not adding? Doing this higher in the stack leads to having a similar code in all software that calls osbuild.

I could go to image-builder or bootc-image-builder and do a similar strategy for the progress bar, but if osbuild would do that already, it'd remove some copying.

That said, I have no problem in implementing that on aib, as @alexlarsson said. Just wanted to raise the discussion and reach an agreement.

[mvo5]

Sorry for coming in a bit late. We build this json-seq based progress so that our consumers are flexible in how represent the progress/monitoring (one of our targets for this is also the webservice) and should have all the information that the monitor in python also has.

But if you really just want a tiny "PrettyMontior" with a bit of rich that seems fine to me, please make it robust against import failures (rich is afaik not available on all our target OSes so we cannot add it as a dependency everywhere) and please make sure that in case of failure the full stage output is shown for debug purposes and please add tests.

Having said this, just using something like https://github.com/osbuild/osbuild/blob/main/tools/osbuild-json-seq-progress-example-renderer will work today, something like:

sudo python3 -m osbuild --libdir . --monitor=JSONSeqMonitor --export image --output-directory /tmp/output  ./test/data/manifests/fedora-boot.json   |   ./tools/osbuild-json-seq-progress-example-renderer

or something (with nice rich sprinckled in) like:

import json
import sys


class SmallProgressRenderer:

    def __init__(self, inf):
        self._inf = inf
        
    def _read_json_seq_rec(self):
        while True:
            line = self._inf.readline()
            if not line:
                return None
            try:
                payload = json.loads(line.strip("\x1e"))
            except json.JSONDecodeError:
                print(f"WARN: invalid json: {line}")
                continue
            return payload

    def render(self):
        while True:
            js = self._read_json_seq_rec()
            if js is None:
                return

            ctx = js["context"]
            if ctx.get("origin") == "osbuild.monitor":
                print(js.get("message"))

            pipeline_result = js.get("result")
            if pipeline_result:
                print(js["message"])
                if not pipeline_result['success']:
                    print("pipeline output:")
                    print(pipeline_result["output"])


if __name__ == "__main__":
    prg = SmallProgressRenderer(sys.stdin)
    prg.render()

The "osbuild.monitor" origin is already kinda reporting this high level stuff it might be enough for your specific use-case. Progressbar is of course also possible, see the above example.