Improved installer

[DevonianTeuchter]

Update to the installer primarily to handle install direct from file but includes:

• analyzes install packages to determine if any are local files
• implements capability to allow for custom repositories, currently complete GitHub mirror or a local directory; defaults to z/OS Open Tools main GH repo
• generates install graph in advance to allow for space calculation up-front
• implements pre/post phases to allow scripts to run at certain times. Default scripts:
  -- auto-cache-clean - clean package cache after installation transaction completes [enabled by default]
  -- man-db - run man-db automagically after install/alt/remove transactions [enabled if man-db package is installed]
  -- caveats - display any "caveats" that a package might have after installation [a caveat being something an end user has to either run, configure or be aware of when using a certain package]
• implements an a/all option for y/n questions to automatically answer yes
• generates a package database, updated during install/alt/removal
• performance improvements to queries, utilising package db and jq
• split install into "upgrade" and "install" scripts
• split query into "query" and "list" scripts
• multiple bug fixes and shellcheck updates
• removes [broken] customized filesystem type logic; always use default of usr/local/zopen for package installation

Should resolve/improve: #733 #717 #701 #694 #688 #675 #673 #661 #643 #633 #619 #617 #605 #567 #566 #558 #533 #472 #431 #381 #339 #240

[IgorTodorovskiIBM]

Impressive amount of changes! This will take me a few days to review!

[v1gnesh]

With a minor edit to the last line w.r.t issues closed, we can make GH auto-close them on the merge of this PR.
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword

[DevonianTeuchter]

With a minor edit to the last line w.r.t issues closed, we can make GH auto-close them on the merge of this PR.

My choice of words was deliberate to ensure this didn't happen ;-) As there is a lot of churn and some of the issues are subjective, explicitly checking those issues to see if the submitter is happy with the update is advised.

[DevonianTeuchter]

Investigating test harness failures...

[lbdyck]

Impressive

[IgorTodorovskiIBM]

I cloned this branch and ran zopen init on my existing file system and got this? Are we able to get it to work with existing file systems?

. ./.env
zopen init -y
Initializing zopen framework
- Enter fully-qualified path to your zopen root directory (default: /home/itodoro/zopen):

- Binaries will be symlinked under "/home/itodoro/zopen/usr/local/bin". Libraries will be symlinked under "/home/itodoro/zopen/usr/lib"
- Packages will be installed and maintained under the directory '/home/itodoro/zopen/'.
- Collecting usage statistics: true
- Re-initializing; zopen-config will be re-created
Generating File System
- Setting certificate environment variable
- Root FS available at /home/itodoro/zopen
Generating zopen-config
- Created config in /home/itodoro/zopen/etc/zopen-config
Installing zopen pre-requisites
/home/itodoro/projects/meta_new/packages/curl-8.4.0.20231011_155722.zos.pax.Z
- Sourcing environment
Processing z/OS Open Tools configuration...DONE
Updating CA Certificate
Getting latest CA Certificate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  224k  100  224k    0     0  1515k      0 --:--:-- --:--:-- --:--:-- 1515k
Re-creating /home/itodoro/zopen/etc/pki/tls/certs/cacert.pem
- zopen bootstrapping complete
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./bump/./bump'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./bzip2/./bzip2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./cli/./cli'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./expat/./expat'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./expect/./expect'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./findutils/./findutils'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gettext/./gettext'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gperf/./gperf'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gpg/./gpg'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./help2man/./help2man'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libgdbm/./libgdbm'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libmd/./libmd'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libpcre/./libpcre'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libpcre2/./libpcre2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libxml2/./libxml2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./llama/./llama'. Bad package install?
jq: invalid JSON text passed to --argjson
Use jq --help for help with command-line options,
or see the jq manpage, or online docs  at https://stedolan.github.io/jq
***ERROR: Could not add metadata for '' to install tracker. Run zopen --re-init to attempt regeneration.

[IgorTodorovskiIBM]

On a new zopen file system:

zopen init
Initializing zopen framework
- Enter fully-qualified path to your zopen root directory (default: /home/itodoro/zopen):
/home/itodoro/zopen-russell
- Binaries will be symlinked under "/home/itodoro/zopen-russell/usr/local/bin". Libraries will be symlinked under "/home/itodoro/zopen-russell/usr/lib"
- Packages will be installed and maintained under the directory '/home/itodoro/zopen-russell/'.
- Collecting usage statistics: true
Do you want to continue? [y/n]
y
Generating File System
- Setting certificate environment variable
- Root FS available at /home/itodoro/zopen-russell
Generating zopen-config
- Created config in /home/itodoro/zopen-russell/etc/zopen-config
Installing zopen pre-requisites
/home/itodoro/projects/meta_new/packages/curl-8.4.0.20231011_155722.zos.pax.Z
Setting up curl...
Setup completed.
Setting up jq...
Setup completed.
- Sourcing environment
Updating CA Certificate
Getting latest CA Certificate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  224k  100  224k    0     0  1458k      0 --:--:-- --:--:-- --:--:-- 1465k
Re-creating /home/itodoro/zopen-russell/etc/pki/tls/certs/cacert.pem
- zopen bootstrapping complete
- Configured zopen to use stable releaseline packages
find: FSUM6512 unable to access "./*/.": EDC5129I No such file or directory.
***ERROR: Unable to update the package db

I have z/OS Open Tools find in my PATH

[DevonianTeuchter]

I cloned this branch and ran zopen init on my existing file system and got this? Are we able to get it to work with existing file systems?

I had tested migrating from 0.8.0 and 0.8.2 - but that was using the --refresh rather than re-init options... let me check that migration path

find: FSUM6512 unable to access "./*/.": EDC5129I No such file or directory.
***ERROR: Unable to update the package db

Another thing to investigate...

[DevonianTeuchter]

@IgorTodorovskiIBM I've pushed a further commit that should resolve the issue when running in a new environment (ie. zopen init). slight issue is that when using the checked out version, it will automatically update meta to the latest in the repo so you then have to reapply the clone into the zopen filesystem. No easy way around that as copying the installing meta into the zopen filesystem could leave the system with an older meta than in the repo going forward, depengind on how quickly a packaged release is made available [if that makes sense].
Some of the changes in the commit are to fix the custom repo handling also.

[DevonianTeuchter]

If anyone has a chance [and is feeling up for a challenge!], the new installer for zopen is available for testing.
It has to be cloned from git since it cannot be pushed to the repos until tested/merged: git clone --single-branch -b new_installer https://github.com/ZOSOpenTools/meta.git
and there's a few hoops to jump through [since it's not available in the repos, zopen has to be tricked to use it]
If interested, please see below for more information

The updated zopen will generate metadata when first "upgraded" so this needs more than just copying the files across - there needs to be a refresh.
Once cloned:
= Installing into a clean test environment:

• cd into the cloned/meta/bin directory
• run ./zopen init -y ""
• Run the instruxtions below to integrate the new meta in place of the "latest" as installed during the initialization
  = An existing zopen environment can be upgraded by:
• move the cloned directory to <zopen-rootfs>/usr/local/zopen/meta/meta.git
• run zopen alt --select meta and pick the git version of zopen (which should be listed)
• run zopen init --refresh - this should generate a new package database.
  This is basically what happens under-the-covers during a meta upgrade, just manually run.
  There shouldn't be any errors when running the commands - at least, it works on my system anyway! :-)
  It should be possible to move back to an older meta by using zopen alt --select meta, the new files/directories generated in <zopenrootfs>/etc/zopen will however need to be manually deleted (as the older meta releases have no knowledge of these files)
  Note that for testing purposes, cloning directly into a directory meta.[xyz] under usr/local/zopen/meta means it is possible to test different versions using zopen alt to switch; using a symlink within the u/l/z/meta directory structure will not currently work due to how the "in-use" version is detected
  Any and all feedback would be appreciated - as this is a bit more than a bug fix or basic feature addition, having extra testing would be useful!

[IgorTodorovskiIBM]

[ITODORO@ZOSCAN2B ~/projects]$ zopen install ~/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
curl: (37) Couldn't open file /home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Could not download from file:///home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z. Correct any errors and potentially retry
[ITODORO@ZOSCAN2B ~/projects]$ zopen install $HOME/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
curl: (37) Couldn't open file /home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Could not download from file:///home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z. Correct any errors and potentially retry
[ITODORO@ZOSCAN2B ~/projects]$ zopen install zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: The following ports could not be installed:
***ERROR:     zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Check port name(s), remove any extra 'port' suffixes and retry command.
[ITODORO@ZOSCAN2B ~/projects]$

[DevonianTeuchter]

@IgorTodorovskiIBM One of the new commits should resolve your issue with relative local paxes.

[DevonianTeuchter]

To test the PR, replace with fq-path to zopen.test and all instances of zopen.test with test directory> :

1. git clone https://github.com/zopencommunity/meta.git -b new_installer zopen.test/usr/local/zopen/meta/meta.new_installer
2. zopen.test/usr/local/zopen/meta/meta.new_installer/bin/zopen init /<parents>/zopen.test
3. zopen alt --select meta -> option 1

[pjfarleyiii]

@DevonianTeuchter I would like to test your new installer, but my employer's z/OS system has NO access to github or any other outside network locations and I do not have any copy of git available even if this system did have access to any network locations. If I git clone the zopen.test tree to my work PC and upload the entire cloned directory to my z/OS Unix file system under my $HOME directory (maybe using a pax file of the whole tree?), is there any chance this new installer can work on this z/OS system (e.g., the cacert.pem retrieval using curl does not function on this z/OS, I have to go get it using a web browser and then upload it to the mainframe under my $HOME directory).