Use elemental's build-iso command instead of luet-makeiso #1245

davidcassany · 2022-04-19T15:03:42Z

Signed-off-by: David Cassany dcassany@suse.com

davidcassany · 2022-04-26T07:07:11Z

@bk201 this PR is adding in elemental binary a build-iso command which pretends to be luet-makeiso substitution, little by little we are moving most of the build and runtime operations over the elemental client, so we provide a more unified and consistent user experience. This PR is changing the manifest.yaml (which was used by luet-makeisp) and the iso related Makefile targets. Are you using any of these two in Harvester? Note that luet-makeiso is not dropped it is simply unused as in this PR Makefile targets are changed use elemental build-iso and the manifest.yaml adapted to the new syntax.

Iso syntax for elemental build-iso is something like:

iso:                                                                            
  rootfs:                                                                       
    - system/cos                                                                
  uefi:                                                                         
    - live/grub2-efi-image                                                      
  isoimage:                                                                     
    - live/grub2                                                                
    - live/grub2-efi-image                                                      
  label: "COS_LIVE"                                                             
                                                                                
name: "cOS-0"                                                                   
date: true

bk201 · 2022-04-26T07:17:46Z

@davidcassany Thanks for the pinging. We use luet-makeiso with a YAML file in our repo:

luet-makeiso iso.yaml --image "${HARVESTER_OS_IMAGE}" --output ${ARTIFACTS_DIR}/${ISO_PREFIX}

The YAML file: iso.yaml

If luet-makeiso still works as usual then it should work for us. Thanks again. cc @johnliu55tw

mudler · 2022-04-26T09:13:00Z

make/Makefile.iso

 ifneq ($(shell id -u), 0)
 	@echo "*** Please run 'make $@' as root"
 	@exit 1
 endif
 ifneq ("$(ISO)","")
 	@echo "'$(ISO) exists, run 'make clean_iso' folled by 'make $@' to recreate"
 else
-	$(LUET) makeiso -- $(MAKEISO_ARGS) $(MANIFEST) --local $(DESTINATION)


All looks good here, I'm just wondering how this works and how the local repo priority is processed behind the scenes. I could not catch this from the logs as looks like only the remote repo is added:

DEBU[2022-04-22T14:06:30Z] Luet config: &{Logging:{Path: EnableLogFile:false JSONFormat:false Level: EnableEmoji:false Color:false NoSpinner:false} General:{SameOwner:false Concurrency:2 Debug:false ShowBuildOutput:false FatalWarns:false HTTPTimeout:0 Quiet:false} System:{DatabaseEngine:boltdb DatabasePath:/var/luet/db Rootfs:/ PkgsCachePath:/tmp/cachepkgs3228759901 TmpDirBase:/var/tmp/luet} Solver:{SolverOptions:{Type:0 Concurrency:0} Type: LearnRate:0 Discount:0 MaxAttempts:0 Implementation:0} RepositoriesConfDir:[] ConfigProtectConfDir:[] ConfigProtectSkip:false ConfigFromHost:false SystemRepositories:[{Name:cos Description:cOS official Urls:[quay.io/costoolkit/releases-green] Type:docker Mode: Priority:1 Enable:true Cached:true Authentication:map[] TreePath: MetaPath: Verify:false Arch: ReferenceID: Revision:0 LastUpdate:}] FinalizerEnvs:[] ConfigProtectConfFiles:[]}

But I guess because there aren't any artifacts from the PR result?

I think logs are misleading here, I am not sure from where this log is coming from but this is likely to be the config loaded from the system. Then before installing any packages custom repositories are used if provided. Provided repositories are not appended to any existing repositories list, the provided repositories are the only ones being used. There is no mechanism to append repositories to the system wide ones, you either use whatever is in the system or the runtime provided list.

Provided repositories are not appended to any existing repositories list, the provided repositories are the only ones being used. There is no mechanism to append repositories to the system wide ones, you either use whatever is in the system or the runtime provided list.

What is the --repo flag here for then?

then I'm not sure if this might work. Before we were appending the local repository to the remote one, increasing the priority of the local. This is in order to let luet resolve packages from remote but take the newest from the local repo with the changes at hand

--repo flag (which can be repeated, the result is a string slice) will initiate a slice of v1.Repository which then in will be passed to pkg/luet/luet.go. The command build-iso will NEVER use the repositories defined in the system such as in /etc/luet/repos.d, if repos are not provided within the manifest (where you can define priority and other repo paramters) or within the command line using --repo flag, build-iso command will only use the default repository.

IMHO from an image builder perspective makes no sense reading the repositories form the system as this couples the build to the running host, which should never happen, the builds should be consistent regardless the host they are built on. So that repositories are explicitly provided (by the user or by elemental defaults).

then I'm not sure if this might work. Before we were appending the local repository to the remote one, increasing the priority of the local. This is in order to let luet resolve packages from remote but take the newest from the local repo with the changes at hand

There could be a case in which the local build and the remote build are having the same version then the criteria about which to pick cannot be considered deterministic.

@mudler What about this behavior:

default repo has a non 0 priority, something that is not the maximum, 10, 90... whatever.

--repo flag instead of replacing default repository it appends on top of default repository or whatever is defined within the manifest.yaml.

This way we can easily add a high priority repository over a defined list (by default or by manifest) and we still can completely use different repositories (no defaults) by setting a repository list in manifest.

Gonna make a PR for that.

mudler · 2022-04-26T09:17:05Z

Looking great! nice!

davidcassany · 2022-04-26T14:34:01Z

If luet-makeiso still works as usual then it should work for us. Thanks again. cc @johnliu55tw

Indeed this what I expected, you use your own yaml file for the ISO and directly call luet-makeiso, so the change should simply not affect you in any way. Once merged we can iterate on how to adapt to the new builder. No big changes, just the command call is slightly different and the manifest is simplified, but the essence is the same.

davidcassany · 2022-04-27T13:52:55Z

This PR should be rebased on top of #1263

davidcassany · 2022-04-28T07:41:46Z

Just rebased from master and updated elemental to include latest version with proper repository configuration.

Signed-off-by: David Cassany <dcassany@suse.com>

mudler · 2022-04-28T08:50:48Z

packages/toolchain/elemental-cli/collection.yaml

    fips: false
    labels:
      github.repo: "elemental"
      github.owner: "rancher-sandbox"
      autobump.revdeps: "true"
      autobump.strategy: "git_hash"
      autobump.git.branch: "main"
-      git.hash: "ed32b2e85140abdb2217e46740804bc237aca4be"


....sorry, I merged the other PR and I didn't noticed the bump was already here 🤦

no problem, this should go smooth and pass all tests, since few Monday they are passing 😉 But it is good we waited to merge until elemental got few small details fixed.

davidcassany marked this pull request as draft April 19, 2022 15:04

davidcassany force-pushed the use_build-iso_command branch 6 times, most recently from 9c88d1f to 6a209c8 Compare April 21, 2022 21:05

davidcassany marked this pull request as ready for review April 22, 2022 14:54

davidcassany requested review from Itxaka, mudler and kkaempf April 26, 2022 06:56

davidcassany mentioned this pull request Apr 26, 2022

elemental build-iso: port code from luet-makeiso #1217

Closed

8 tasks

mudler reviewed Apr 26, 2022

View reviewed changes

mudler approved these changes Apr 26, 2022

View reviewed changes

davidcassany added the status/blocked Issue depends on another one label Apr 27, 2022

davidcassany force-pushed the use_build-iso_command branch from 6a209c8 to 0644e12 Compare April 28, 2022 07:40

davidcassany removed the status/blocked Issue depends on another one label Apr 28, 2022

davidcassany enabled auto-merge (squash) April 28, 2022 07:42

Itxaka approved these changes Apr 28, 2022

View reviewed changes

davidcassany added 3 commits April 28, 2022 10:42

Use elemental's build-iso command instead of luet-makeiso

ce387df

Signed-off-by: David Cassany <dcassany@suse.com>

Update workflows

a20122d

Signed-off-by: David Cassany <dcassany@suse.com>

Bump elemental to latest

2dedff2

Signed-off-by: David Cassany <dcassany@suse.com>

davidcassany force-pushed the use_build-iso_command branch from 0644e12 to 2dedff2 Compare April 28, 2022 08:43

mudler reviewed Apr 28, 2022

View reviewed changes

davidcassany merged commit c24192a into master Apr 28, 2022

davidcassany deleted the use_build-iso_command branch April 28, 2022 10:53

j-landru mentioned this pull request Apr 29, 2022

elemental build-iso what about Syslinux bootloader alternative ? #1270

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use elemental's build-iso command instead of luet-makeiso #1245

Use elemental's build-iso command instead of luet-makeiso #1245

davidcassany commented Apr 19, 2022 •

edited

Loading

davidcassany commented Apr 26, 2022

bk201 commented Apr 26, 2022

mudler Apr 26, 2022 •

edited

Loading

davidcassany Apr 26, 2022

mudler Apr 26, 2022 •

edited

Loading

davidcassany Apr 27, 2022

davidcassany Apr 27, 2022

mudler commented Apr 26, 2022

davidcassany commented Apr 26, 2022

davidcassany commented Apr 27, 2022

davidcassany commented Apr 28, 2022

mudler Apr 28, 2022 •

edited

Loading

davidcassany Apr 28, 2022

Use elemental's build-iso command instead of luet-makeiso #1245

Use elemental's build-iso command instead of luet-makeiso #1245

Conversation

davidcassany commented Apr 19, 2022 • edited Loading

davidcassany commented Apr 26, 2022

bk201 commented Apr 26, 2022

mudler Apr 26, 2022 • edited Loading

Choose a reason for hiding this comment

davidcassany Apr 26, 2022

Choose a reason for hiding this comment

mudler Apr 26, 2022 • edited Loading

Choose a reason for hiding this comment

davidcassany Apr 27, 2022

Choose a reason for hiding this comment

davidcassany Apr 27, 2022

Choose a reason for hiding this comment

mudler commented Apr 26, 2022

davidcassany commented Apr 26, 2022

davidcassany commented Apr 27, 2022

davidcassany commented Apr 28, 2022

mudler Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

davidcassany Apr 28, 2022

Choose a reason for hiding this comment

davidcassany commented Apr 19, 2022 •

edited

Loading

mudler Apr 26, 2022 •

edited

Loading

mudler Apr 26, 2022 •

edited

Loading

mudler Apr 28, 2022 •

edited

Loading