Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Establish Postgres database format upgrade #2780 #2785

Conversation

phillxnet
Copy link
Member

@phillxnet phillxnet commented Jan 23, 2024

Adds a shell script, to be run by rpm during update, to ensure any Postgres 10 (or later) formated DBs are updated to our current Postgres target version: established by rpm dependencies and 'alternatives'.

Fixes #2780

Testing

This script was developed on, and has had initial testing with, an install originally derived from the following installer:

Rockstor-Leap15.3-generic.x86_64-4.1.0-0.install.iso Jan 14 2022

  • The resulting install was then subscribed to the Stable updates channel.
  • An underlying OS update to a 15.4 base was then enacted via:
    Distribution update from 15.3 to 15.4: https://rockstor.com/docs/howtos/15-3_to_15-4.html
    Resulting also in an automatic update to our last stable release of 4.6.1-0.

The above system then represents a reproducer for a pending stable update issue found in the current testing channel; where the far newer Django and psycopg3 there-in are no longer compatible with a Postgres 10 formatted DB: such as a system with the above, or older heritage, will have.


Partnered with rockstor-rpmbuild PR: "Update DB format on install/upgrade if required ...": rockstor/rockstor-rpmbuild#59

Adds a shell script, to be run by rpm during update,
to ensure any Postgres 10 (or later) formated DBs are
updated to our current Postgres target version:
established by rpm dependencies and 'alternatives'.
@phillxnet
Copy link
Member Author

phillxnet commented Jan 24, 2024

Initial install test

Installing an rpm derived from this and the partner rockstor-rpmbuild referenced PR and we have the following partial success.

# 2024-01-24 11:13:48 rockstor-5.0.6-2785.x86_64.rpm installed ok
# Additional rpm output:
# No static/config-backups directory found.
# update-alternatives: using /usr/lib/postgresql13 to provide /usr/lib/postgresql (postgresql) in manual mode
# 
2024-01-24 11:13:48|install|rockstor|5.0.6-2785|x86_64|root@installer|Buildbot-Testing|2d58ec5d9f23863b66346f6dee1217a6b1a74e8d33b0e1bc62c09508e821c074|
# 2024-01-24 11:13:48 %posttrans(rockstor-5.0.6-2785.x86_64) script output:
# Updating DB format from 10 to 13 via pg_upgrade.
# /var/lib/pgsql /
# The files belonging to this database system will be owned by user "postgres".
# This user must also own the server process.
# 
# The database cluster will be initialized with locale "C".
# The default database encoding has accordingly been set to "SQL_ASCII".
# The default text search configuration will be set to "english".
# 
# Data page checksums are disabled.
# 
# creating directory /var/lib/pgsql/data13 ... ok
# creating subdirectories ... ok
# selecting dynamic shared memory implementation ... posix
# selecting default max_connections ... 100
# selecting default shared_buffers ... 128MB
# selecting default time zone ... Europe/Lisbon
# creating configuration files ... ok
# running bootstrap script ... ok
# performing post-bootstrap initialization ... ok
# syncing data to disk ... ok
# 
# initdb: warning: enabling "trust" authentication for local connections
# You can change this by editing pg_hba.conf or using the option -A, or
# --auth-local and --auth-host, the next time you run initdb.
# 
# Success. You can now start the database server using:
# 
#     /usr/lib/postgresql13/bin/pg_ctl -D /var/lib/pgsql/data13 -l logfile start
# 
# Performing Consistency Checks
# -----------------------------
# Checking cluster versions                                   ok
# Checking database user is the install user                  ok
# Checking database connection settings                       ok
# Checking for prepared transactions                          ok
# Checking for system-defined composite types in user tables  ok
# Checking for reg* data types in user tables                 ok
# Checking for contrib/isn with bigint-passing mismatch       ok
# Checking for removed "abstime" data type in user tables     ok
# Checking for removed "reltime" data type in user tables     ok
# Checking for removed "tinterval" data type in user tables   ok
# Checking for tables WITH OIDS                               ok
# Checking for invalid "sql_identifier" user columns          ok
# Creating dump of global objects                             ok
# Creating dump of database schemas
#   postgres
#   smartdb
#   storageadmin
#   template1
#                                                             ok
# 
# encodings for database "postgres" do not match:  old "UTF8", new "SQL_ASCII"
# Failure, exiting
# /root/.local/bin/poetry
# Poetry version 1.1.15 found - UNINSTALLING
# Removing Poetry (1.1.15)
# build.sh has PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Indicating that we have to explicitly indicate our encoding when initialising our new DB.:

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".

and

encodings for database "postgres" do not match: old "UTF8", new "SQL_ASCII"
Failure, exiting

When our new shell script was run at the terminal this was not observed. Follow-up commit in the works.

Includes:
- Enforce Django required UTF8 during initdb.
- Add more user feedback via `echo` additions.
- Drop use of --concurrently in `reindexdb`.
@phillxnet
Copy link
Member Author

It appears our current local 'mask' is not helping us here, but UTF8 encoding is now established:

# 2024-01-24 13:32:14 rockstor-5.0.6-2785.x86_64.rpm installed ok
# Additional rpm output:
# No static/config-backups directory found.
# update-alternatives: using /usr/lib/postgresql13 to provide /usr/lib/postgresql (postgresql) in manual mode
# 
2024-01-24 13:32:14|install|rockstor|5.0.6-2785|x86_64|root@installer|Buildbot-Testing|e485b7a616a6b58f78dc9038373ccf2d69072003668cb43e97b257bc2dbe5df9|
# 2024-01-24 13:32:14 %posttrans(rockstor-5.0.6-2785.x86_64) script output:
# Updating DB format from 10 to 13 via pg_upgrade.
# /var/lib/pgsql /
# The files belonging to this database system will be owned by user "postgres".
# This user must also own the server process.
# 
# The database cluster will be initialized with locale "C".
# The default text search configuration will be set to "english".
# 
# Data page checksums are disabled.
# 
# creating directory /var/lib/pgsql/data13 ... ok
# creating subdirectories ... ok
# selecting dynamic shared memory implementation ... posix
# selecting default max_connections ... 100
# selecting default shared_buffers ... 128MB
# selecting default time zone ... Europe/Lisbon
# creating configuration files ... ok
# running bootstrap script ... ok
# performing post-bootstrap initialization ... ok
# syncing data to disk ... ok
# 
# initdb: warning: enabling "trust" authentication for local connections
# You can change this by editing pg_hba.conf or using the option -A, or
# --auth-local and --auth-host, the next time you run initdb.
# 
# Success. You can now start the database server using:
# 
#     /usr/lib/postgresql13/bin/pg_ctl -D /var/lib/pgsql/data13 -l logfile start
# 
# Stopping postgresql
# mv /var/lib/pgsql/data /var/lib/pgsql/data10
# Performing Consistency Checks
# -----------------------------
# Checking cluster versions                                   ok
# Checking database user is the install user                  ok
# Checking database connection settings                       ok
# Checking for prepared transactions                          ok
# Checking for system-defined composite types in user tables  ok
# Checking for reg* data types in user tables                 ok
# Checking for contrib/isn with bigint-passing mismatch       ok
# Checking for removed "abstime" data type in user tables     ok
# Checking for removed "reltime" data type in user tables     ok
# Checking for removed "tinterval" data type in user tables   ok
# Checking for tables WITH OIDS                               ok
# Checking for invalid "sql_identifier" user columns          ok
# Creating dump of global objects                             ok
# Creating dump of database schemas
#   postgres
#   smartdb
#   storageadmin
#   template1
#                                                             ok
# 
# lc_collate values for database "postgres" do not match:  old "en_GB.UTF-8", new "C"
# Failure, exiting
# /root/.local/bin/poetry
# Poetry version 1.1.15 found - UNINSTALLING

Summary:

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

lc_collate values for database "postgres" do not match: old "en_GB.UTF-8", new "C"
Failure, exiting

- Establish LANG from install prior to invoking initdb.
- Improve output formatting.
@phillxnet
Copy link
Member Author

Adopting installs' LANG=en_GB.UTF-8
# The files belonging to this database system will be owned by user "postgres".
# This user must also own the server process.
# 
# The database cluster will be initialized with locale "C".
# The default text search configuration will be set to "english".

and consequently:

lc_collate values for database "postgres" do not match: old "en_GB.UTF-8", new "C"

@phillxnet
Copy link
Member Author

phillxnet commented Jan 24, 2024

We look to now have a successful, rpm %posttrans invoked, db_update.sh script run:

# 2024-01-24 18:10:16 rockstor-5.0.6-2785.x86_64.rpm installed ok
# Additional rpm output:
# No static/config-backups directory found.
# update-alternatives: using /usr/lib/postgresql13 to provide /usr/lib/postgresql (postgresql) in manual mode
# 
2024-01-24 18:10:16|install|rockstor|5.0.6-2785|x86_64|root@installer|Buildbot-Testing|8de6c96619531cf3e78cf734088c5f761acf1e4aee8bc187013d727293955a01|
# 2024-01-24 18:10:16 %posttrans(rockstor-5.0.6-2785.x86_64) script output:
# Updating DB format from 10 to 13 via pg_upgrade.
# /var/lib/pgsql /
# 
# Adopting installs' LANG=en_GB.UTF-8
# The files belonging to this database system will be owned by user "postgres".
# This user must also own the server process.
# 
# The database cluster will be initialized with locale "en_GB.UTF-8".
# The default text search configuration will be set to "english".
# 
# Data page checksums are disabled.
# 
# creating directory /var/lib/pgsql/data13 ... ok
# creating subdirectories ... ok
# selecting dynamic shared memory implementation ... posix
# selecting default max_connections ... 100
# selecting default shared_buffers ... 128MB
# selecting default time zone ... Europe/Lisbon
# creating configuration files ... ok
# running bootstrap script ... ok
# performing post-bootstrap initialization ... ok
# syncing data to disk ... ok
# 
# initdb: warning: enabling "trust" authentication for local connections
# You can change this by editing pg_hba.conf or using the option -A, or
# --auth-local and --auth-host, the next time you run initdb.
# 
# Success. You can now start the database server using:
# 
#     /usr/lib/postgresql13/bin/pg_ctl -D /var/lib/pgsql/data13 -l logfile start
# 
# Stopping postgresql
# 
# mv /var/lib/pgsql/data /var/lib/pgsql/data10
# 
# Performing Consistency Checks
# -----------------------------
# Checking cluster versions                                   ok
# Checking database user is the install user                  ok
# Checking database connection settings                       ok
# Checking for prepared transactions                          ok
# Checking for system-defined composite types in user tables  ok
# Checking for reg* data types in user tables                 ok
# Checking for contrib/isn with bigint-passing mismatch       ok
# Checking for removed "abstime" data type in user tables     ok
# Checking for removed "reltime" data type in user tables     ok
# Checking for removed "tinterval" data type in user tables   ok
# Checking for tables WITH OIDS                               ok
# Checking for invalid "sql_identifier" user columns          ok
# Creating dump of global objects                             ok
# Creating dump of database schemas
#   postgres
#   smartdb
#   storageadmin
#   template1
#                                                             ok
# Checking for presence of required libraries                 ok
# Checking database user is the install user                  ok
# Checking for prepared transactions                          ok
# Checking for new cluster tablespace directories             ok
# 
# If pg_upgrade fails after this point, you must re-initdb the
# new cluster before continuing.
# 
# Performing Upgrade
# ------------------
# Analyzing all rows in the new cluster                       ok
# Freezing all rows in the new cluster                        ok
# Deleting files from new pg_xact                             ok
# Copying old pg_xact to new server                           ok
# Setting oldest XID for new cluster                          ok
# Setting next transaction ID and epoch for new cluster       ok
# Deleting files from new pg_multixact/offsets                ok
# Copying old pg_multixact/offsets to new server              ok
# Deleting files from new pg_multixact/members                ok
# Copying old pg_multixact/members to new server              ok
# Setting next multixact ID and offset for new cluster        ok
# Resetting WAL archives                                      ok
# Setting frozenxid and minmxid counters in new cluster       ok
# Restoring global objects in the new cluster                 ok
# Restoring database schemas in the new cluster
#   template1
#   postgres
#   smartdb
#   storageadmin
#                                                             ok
# Copying user relation files
#   /var/lib/pgsql/data10/base/13655/2613
#   /var/lib/pgsql/data10/base/13655/2683
#   /var/lib/pgsql/data10/base/16384/2613
#   /var/lib/pgsql/data10/base/16384/2683
#   /var/lib/pgsql/data10/base/16384/17559
#   /var/lib/pgsql/data10/base/16384/17564
#   /var/lib/pgsql/data10/base/16384/17569
#   /var/lib/pgsql/data10/base/16384/17574
#   /var/lib/pgsql/data10/base/16384/17579
#   /var/lib/pgsql/data10/base/16384/17584
#   /var/lib/pgsql/data10/base/16384/17589
#   /var/lib/pgsql/data10/base/16384/17594
#   /var/lib/pgsql/data10/base/16384/17599
#   /var/lib/pgsql/data10/base/16384/17604
#   /var/lib/pgsql/data10/base/16384/17609
#   /var/lib/pgsql/data10/base/16384/17612
#   /var/lib/pgsql/data10/base/16384/17614
#   /var/lib/pgsql/data10/base/16384/17617
#   /var/lib/pgsql/data10/base/16384/17620
#   /var/lib/pgsql/data10/base/16384/17622
#   /var/lib/pgsql/data10/base/16384/17625
#   /var/lib/pgsql/data10/base/16384/17628
#   /var/lib/pgsql/data10/base/16384/17630
#   /var/lib/pgsql/data10/base/16384/17633
#   /var/lib/pgsql/data10/base/16384/17636
#   /var/lib/pgsql/data10/base/16384/17638
#   /var/lib/pgsql/data10/base/16384/17641
#   /var/lib/pgsql/data10/base/16384/17644
#   /var/lib/pgsql/data10/base/16384/17646
#   /var/lib/pgsql/data10/base/16384/17649
#   /var/lib/pgsql/data10/base/16384/17649_fsm
#   /var/lib/pgsql/data10/base/16384/17649_vm
#   /var/lib/pgsql/data10/base/16384/17652
#   /var/lib/pgsql/data10/base/16384/17654
#   /var/lib/pgsql/data10/base/16384/17657
#   /var/lib/pgsql/data10/base/16384/17662
#   /var/lib/pgsql/data10/base/16384/17662_fsm
#   /var/lib/pgsql/data10/base/16384/17662_vm
#   /var/lib/pgsql/data10/base/16384/17665
#   /var/lib/pgsql/data10/base/16384/17667
#   /var/lib/pgsql/data10/base/16384/17670
#   /var/lib/pgsql/data10/base/16384/17673
#   /var/lib/pgsql/data10/base/16384/17675
#   /var/lib/pgsql/data10/base/16384/17678
#   /var/lib/pgsql/data10/base/16384/17683
#   /var/lib/pgsql/data10/base/16384/17686
#   /var/lib/pgsql/data10/base/16384/17688
#   /var/lib/pgsql/data10/base/16384/17691
#   /var/lib/pgsql/data10/base/16384/17696
#   /var/lib/pgsql/data10/base/16384/17699
#   /var/lib/pgsql/data10/base/16384/17701
#   /var/lib/pgsql/data10/base/16384/17727
#   /var/lib/pgsql/data10/base/16384/17729
#   /var/lib/pgsql/data10/base/16384/17731
#   /var/lib/pgsql/data10/base/16384/17733
#   /var/lib/pgsql/data10/base/16384/17735
#   /var/lib/pgsql/data10/base/16384/17737
#   /var/lib/pgsql/data10/base/16384/17739
#   /var/lib/pgsql/data10/base/16384/17741
#   /var/lib/pgsql/data10/base/16384/17743
#   /var/lib/pgsql/data10/base/16384/17745
#   /var/lib/pgsql/data10/base/16384/17747
#   /var/lib/pgsql/data10/base/16384/17749
#   /var/lib/pgsql/data10/base/16384/17751
#   /var/lib/pgsql/data10/base/16384/17753
#   /var/lib/pgsql/data10/base/16384/17755
#   /var/lib/pgsql/data10/base/16384/17757
#   /var/lib/pgsql/data10/base/16384/17759
#   /var/lib/pgsql/data10/base/16384/17761
#   /var/lib/pgsql/data10/base/16384/17763
#   /var/lib/pgsql/data10/base/16384/17765
#   /var/lib/pgsql/data10/base/16384/17767
#   /var/lib/pgsql/data10/base/16384/17769
#   /var/lib/pgsql/data10/base/16384/17771
#   /var/lib/pgsql/data10/base/16384/17773
#   /var/lib/pgsql/data10/base/16384/17775
#   /var/lib/pgsql/data10/base/16384/17777
#   /var/lib/pgsql/data10/base/16384/17779
#   /var/lib/pgsql/data10/base/16384/17781
#   /var/lib/pgsql/data10/base/16384/17782
#   /var/lib/pgsql/data10/base/16384/17783
#   /var/lib/pgsql/data10/base/16384/17784
#   /var/lib/pgsql/data10/base/16384/17785
#   /var/lib/pgsql/data10/base/16384/17786
#   /var/lib/pgsql/data10/base/16384/17787
#   /var/lib/pgsql/data10/base/16384/17788
#   /var/lib/pgsql/data10/base/16384/17789
#   /var/lib/pgsql/data10/base/16384/17790
#   /var/lib/pgsql/data10/base/16384/17791
#   /var/lib/pgsql/data10/base/16384/17792
#   /var/lib/pgsql/data10/base/16384/17793
#   /var/lib/pgsql/data10/base/16384/17794
#   /var/lib/pgsql/data10/base/16384/17795
#   /var/lib/pgsql/data10/base/16384/17796
#   /var/lib/pgsql/data10/base/16384/17797
#   /var/lib/pgsql/data10/base/16384/17798
#   /var/lib/pgsql/data10/base/16384/17799
#   /var/lib/pgsql/data10/base/16384/17800
#   /var/lib/pgsql/data10/base/16384/17801
#   /var/lib/pgsql/data10/base/16384/17802
#   /var/lib/pgsql/data10/base/16384/17803
#   /var/lib/pgsql/data10/base/16384/17804
#   /var/lib/pgsql/data10/base/16384/17805
#   /var/lib/pgsql/data10/base/16384/17806
#   /var/lib/pgsql/data10/base/16384/17807
#   /var/lib/pgsql/data10/base/16384/17808
#   /var/lib/pgsql/data10/base/16384/17809
#   /var/lib/pgsql/data10/base/16384/17810
#   /var/lib/pgsql/data10/base/16384/17811
#   /var/lib/pgsql/data10/base/16384/17812
#   /var/lib/pgsql/data10/base/16384/17813
#   /var/lib/pgsql/data10/base/16384/17814
#   /var/lib/pgsql/data10/base/16384/17873
#   /var/lib/pgsql/data10/base/16384/17877
#   /var/lib/pgsql/data10/base/16384/17879
#   /var/lib/pgsql/data10/base/16384/17880
#   /var/lib/pgsql/data10/base/16385/2613
#   /var/lib/pgsql/data10/base/16385/2683
#   /var/lib/pgsql/data10/base/16385/16387
#   /var/lib/pgsql/data10/base/16385/16392
#   /var/lib/pgsql/data10/base/16385/16397
#   /var/lib/pgsql/data10/base/16385/16397_fsm
#   /var/lib/pgsql/data10/base/16385/16402
#   /var/lib/pgsql/data10/base/16385/16405
#   /var/lib/pgsql/data10/base/16385/16412
#   /var/lib/pgsql/data10/base/16385/16417
#   /var/lib/pgsql/data10/base/16385/16421
#   /var/lib/pgsql/data10/base/16385/16423
#   /var/lib/pgsql/data10/base/16385/16426
#   /var/lib/pgsql/data10/base/16385/16431
#   /var/lib/pgsql/data10/base/16385/16434
#   /var/lib/pgsql/data10/base/16385/16436
#   /var/lib/pgsql/data10/base/16385/16437
#   /var/lib/pgsql/data10/base/16385/16442
#   /var/lib/pgsql/data10/base/16385/16445
#   /var/lib/pgsql/data10/base/16385/16447
#   /var/lib/pgsql/data10/base/16385/18289
#   /var/lib/pgsql/data10/base/16385/18312
#   /var/lib/pgsql/data10/base/16385/18335
#   /var/lib/pgsql/data10/base/16385/18357
#   /var/lib/pgsql/data10/base/16385/16477
#   /var/lib/pgsql/data10/base/16385/16480
#   /var/lib/pgsql/data10/base/16385/16482
#   /var/lib/pgsql/data10/base/16385/16485
#   /var/lib/pgsql/data10/base/16385/16488
#   /var/lib/pgsql/data10/base/16385/16490
#   /var/lib/pgsql/data10/base/16385/16493
#   /var/lib/pgsql/data10/base/16385/16498
#   /var/lib/pgsql/data10/base/16385/16501
#   /var/lib/pgsql/data10/base/16385/16503
#   /var/lib/pgsql/data10/base/16385/16506
#   /var/lib/pgsql/data10/base/16385/16509
#   /var/lib/pgsql/data10/base/16385/16511
#   /var/lib/pgsql/data10/base/16385/16514
#   /var/lib/pgsql/data10/base/16385/16519
#   /var/lib/pgsql/data10/base/16385/16522
#   /var/lib/pgsql/data10/base/16385/16524
#   /var/lib/pgsql/data10/base/16385/16527
#   /var/lib/pgsql/data10/base/16385/16530
#   /var/lib/pgsql/data10/base/16385/16532
#   /var/lib/pgsql/data10/base/16385/16535
#   /var/lib/pgsql/data10/base/16385/16538
#   /var/lib/pgsql/data10/base/16385/16540
#   /var/lib/pgsql/data10/base/16385/16543
#   /var/lib/pgsql/data10/base/16385/16546
#   /var/lib/pgsql/data10/base/16385/16548
#   /var/lib/pgsql/data10/base/16385/16551
#   /var/lib/pgsql/data10/base/16385/16556
#   /var/lib/pgsql/data10/base/16385/16559
#   /var/lib/pgsql/data10/base/16385/16561
#   /var/lib/pgsql/data10/base/16385/16564
#   /var/lib/pgsql/data10/base/16385/16567
#   /var/lib/pgsql/data10/base/16385/16569
#   /var/lib/pgsql/data10/base/16385/17987
#   /var/lib/pgsql/data10/base/16385/17987_fsm
#   /var/lib/pgsql/data10/base/16385/17987_vm
#   /var/lib/pgsql/data10/base/16385/18052
#   /var/lib/pgsql/data10/base/16385/16588
#   /var/lib/pgsql/data10/base/16385/16591
#   /var/lib/pgsql/data10/base/16385/16593
#   /var/lib/pgsql/data10/base/16385/16596
#   /var/lib/pgsql/data10/base/16385/16599
#   /var/lib/pgsql/data10/base/16385/16601
#   /var/lib/pgsql/data10/base/16385/16604
#   /var/lib/pgsql/data10/base/16385/16609
#   /var/lib/pgsql/data10/base/16385/16612
#   /var/lib/pgsql/data10/base/16385/16614
#   /var/lib/pgsql/data10/base/16385/16617
#   /var/lib/pgsql/data10/base/16385/16622
#   /var/lib/pgsql/data10/base/16385/16635
#   /var/lib/pgsql/data10/base/16385/16638
#   /var/lib/pgsql/data10/base/16385/16640
#   /var/lib/pgsql/data10/base/16385/16643
#   /var/lib/pgsql/data10/base/16385/16646
#   /var/lib/pgsql/data10/base/16385/16648
#   /var/lib/pgsql/data10/base/16385/16651
#   /var/lib/pgsql/data10/base/16385/16654
#   /var/lib/pgsql/data10/base/16385/16656
#   /var/lib/pgsql/data10/base/16385/16659
#   /var/lib/pgsql/data10/base/16385/16662
#   /var/lib/pgsql/data10/base/16385/16664
#   /var/lib/pgsql/data10/base/16385/18206
#   /var/lib/pgsql/data10/base/16385/16672
#   /var/lib/pgsql/data10/base/16385/16677
#   /var/lib/pgsql/data10/base/16385/16680
#   /var/lib/pgsql/data10/base/16385/16682
#   /var/lib/pgsql/data10/base/16385/16685
#   /var/lib/pgsql/data10/base/16385/16685_fsm
#   /var/lib/pgsql/data10/base/16385/16685_vm
#   /var/lib/pgsql/data10/base/16385/16688
#   /var/lib/pgsql/data10/base/16385/16690
#   /var/lib/pgsql/data10/base/16385/17998
#   /var/lib/pgsql/data10/base/16385/18092
#   /var/lib/pgsql/data10/base/16385/16706
#   /var/lib/pgsql/data10/base/16385/16711
#   /var/lib/pgsql/data10/base/16385/16714
#   /var/lib/pgsql/data10/base/16385/16716
#   /var/lib/pgsql/data10/base/16385/16719
#   /var/lib/pgsql/data10/base/16385/16722
#   /var/lib/pgsql/data10/base/16385/16724
#   /var/lib/pgsql/data10/base/16385/18010
#   /var/lib/pgsql/data10/base/16385/16735
#   /var/lib/pgsql/data10/base/16385/16740
#   /var/lib/pgsql/data10/base/16385/17909
#   /var/lib/pgsql/data10/base/16385/17909_fsm
#   /var/lib/pgsql/data10/base/16385/17909_vm
#   /var/lib/pgsql/data10/base/16385/16753
#   /var/lib/pgsql/data10/base/16385/16756
#   /var/lib/pgsql/data10/base/16385/16758
#   /var/lib/pgsql/data10/base/16385/16761
#   /var/lib/pgsql/data10/base/16385/16764
#   /var/lib/pgsql/data10/base/16385/16766
#   /var/lib/pgsql/data10/base/16385/16769
#   /var/lib/pgsql/data10/base/16385/16774
#   /var/lib/pgsql/data10/base/16385/16777
#   /var/lib/pgsql/data10/base/16385/16779
#   /var/lib/pgsql/data10/base/16385/16782
#   /var/lib/pgsql/data10/base/16385/16785
#   /var/lib/pgsql/data10/base/16385/16787
#   /var/lib/pgsql/data10/base/16385/16790
#   /var/lib/pgsql/data10/base/16385/16795
#   /var/lib/pgsql/data10/base/16385/16798
#   /var/lib/pgsql/data10/base/16385/16800
#   /var/lib/pgsql/data10/base/16385/16803
#   /var/lib/pgsql/data10/base/16385/16808
#   /var/lib/pgsql/data10/base/16385/16811
#   /var/lib/pgsql/data10/base/16385/16813
#   /var/lib/pgsql/data10/base/16385/16816
#   /var/lib/pgsql/data10/base/16385/16819
#   /var/lib/pgsql/data10/base/16385/16821
#   /var/lib/pgsql/data10/base/16385/16824
#   /var/lib/pgsql/data10/base/16385/16827
#   /var/lib/pgsql/data10/base/16385/16829
#   /var/lib/pgsql/data10/base/16385/16832
#   /var/lib/pgsql/data10/base/16385/16835
#   /var/lib/pgsql/data10/base/16385/16837
#   /var/lib/pgsql/data10/base/16385/16840
#   /var/lib/pgsql/data10/base/16385/16843
#   /var/lib/pgsql/data10/base/16385/16845
#   /var/lib/pgsql/data10/base/16385/16848
#   /var/lib/pgsql/data10/base/16385/16851
#   /var/lib/pgsql/data10/base/16385/16853
#   /var/lib/pgsql/data10/base/16385/16856
#   /var/lib/pgsql/data10/base/16385/16928
#   /var/lib/pgsql/data10/base/16385/16930
#   /var/lib/pgsql/data10/base/16385/16932
#   /var/lib/pgsql/data10/base/16385/16934
#   /var/lib/pgsql/data10/base/16385/16936
#   /var/lib/pgsql/data10/base/16385/16938
#   /var/lib/pgsql/data10/base/16385/16940
#   /var/lib/pgsql/data10/base/16385/16942
#   /var/lib/pgsql/data10/base/16385/16944
#   /var/lib/pgsql/data10/base/16385/16946
#   /var/lib/pgsql/data10/base/16385/16948
#   /var/lib/pgsql/data10/base/16385/16952
#   /var/lib/pgsql/data10/base/16385/16954
#   /var/lib/pgsql/data10/base/16385/16956
#   /var/lib/pgsql/data10/base/16385/16958
#   /var/lib/pgsql/data10/base/16385/16960
#   /var/lib/pgsql/data10/base/16385/16962
#   /var/lib/pgsql/data10/base/16385/18318
#   /var/lib/pgsql/data10/base/16385/16976
#   /var/lib/pgsql/data10/base/16385/16978
#   /var/lib/pgsql/data10/base/16385/16980
#   /var/lib/pgsql/data10/base/16385/16982
#   /var/lib/pgsql/data10/base/16385/16984
#   /var/lib/pgsql/data10/base/16385/16986
#   /var/lib/pgsql/data10/base/16385/16988
#   /var/lib/pgsql/data10/base/16385/16990
#   /var/lib/pgsql/data10/base/16385/16992
#   /var/lib/pgsql/data10/base/16385/16994
#   /var/lib/pgsql/data10/base/16385/16996
#   /var/lib/pgsql/data10/base/16385/16998
#   /var/lib/pgsql/data10/base/16385/17000
#   /var/lib/pgsql/data10/base/16385/17002
#   /var/lib/pgsql/data10/base/16385/17004
#   /var/lib/pgsql/data10/base/16385/17008
#   /var/lib/pgsql/data10/base/16385/17012
#   /var/lib/pgsql/data10/base/16385/17014
#   /var/lib/pgsql/data10/base/16385/17016
#   /var/lib/pgsql/data10/base/16385/17018
#   /var/lib/pgsql/data10/base/16385/17020
#   /var/lib/pgsql/data10/base/16385/17994
#   /var/lib/pgsql/data10/base/16385/18058
#   /var/lib/pgsql/data10/base/16385/18059
#   /var/lib/pgsql/data10/base/16385/18060
#   /var/lib/pgsql/data10/base/16385/17032
#   /var/lib/pgsql/data10/base/16385/17034
#   /var/lib/pgsql/data10/base/16385/17036
#   /var/lib/pgsql/data10/base/16385/17038
#   /var/lib/pgsql/data10/base/16385/17040
#   /var/lib/pgsql/data10/base/16385/17042
#   /var/lib/pgsql/data10/base/16385/17044
#   /var/lib/pgsql/data10/base/16385/17046
#   /var/lib/pgsql/data10/base/16385/17048
#   /var/lib/pgsql/data10/base/16385/17050
#   /var/lib/pgsql/data10/base/16385/17052
#   /var/lib/pgsql/data10/base/16385/17054
#   /var/lib/pgsql/data10/base/16385/17062
#   /var/lib/pgsql/data10/base/16385/17064
#   /var/lib/pgsql/data10/base/16385/17066
#   /var/lib/pgsql/data10/base/16385/17068
#   /var/lib/pgsql/data10/base/16385/17070
#   /var/lib/pgsql/data10/base/16385/17072
#   /var/lib/pgsql/data10/base/16385/18209
#   /var/lib/pgsql/data10/base/16385/18210
#   /var/lib/pgsql/data10/base/16385/17080
#   /var/lib/pgsql/data10/base/16385/17082
#   /var/lib/pgsql/data10/base/16385/17084
#   /var/lib/pgsql/data10/base/16385/17086
#   /var/lib/pgsql/data10/base/16385/17088
#   /var/lib/pgsql/data10/base/16385/17090
#   /var/lib/pgsql/data10/base/16385/17092
#   /var/lib/pgsql/data10/base/16385/17094
#   /var/lib/pgsql/data10/base/16385/18004
#   /var/lib/pgsql/data10/base/16385/18095
#   /var/lib/pgsql/data10/base/16385/17100
#   /var/lib/pgsql/data10/base/16385/17102
#   /var/lib/pgsql/data10/base/16385/17104
#   /var/lib/pgsql/data10/base/16385/18016
#   /var/lib/pgsql/data10/base/16385/18017
#   /var/lib/pgsql/data10/base/16385/18018
#   /var/lib/pgsql/data10/base/16385/17112
#   /var/lib/pgsql/data10/base/16385/17114
#   /var/lib/pgsql/data10/base/16385/17116
#   /var/lib/pgsql/data10/base/16385/17915
#   /var/lib/pgsql/data10/base/16385/17916
#   /var/lib/pgsql/data10/base/16385/17122
#   /var/lib/pgsql/data10/base/16385/17124
#   /var/lib/pgsql/data10/base/16385/17126
#   /var/lib/pgsql/data10/base/16385/17128
#   /var/lib/pgsql/data10/base/16385/17130
#   /var/lib/pgsql/data10/base/16385/17132
#   /var/lib/pgsql/data10/base/16385/17134
#   /var/lib/pgsql/data10/base/16385/17136
#   /var/lib/pgsql/data10/base/16385/17138
#   /var/lib/pgsql/data10/base/16385/17140
#   /var/lib/pgsql/data10/base/16385/17142
#   /var/lib/pgsql/data10/base/16385/17144
#   /var/lib/pgsql/data10/base/16385/17146
#   /var/lib/pgsql/data10/base/16385/17148
#   /var/lib/pgsql/data10/base/16385/17150
#   /var/lib/pgsql/data10/base/16385/17152
#   /var/lib/pgsql/data10/base/16385/17154
#   /var/lib/pgsql/data10/base/16385/17156
#   /var/lib/pgsql/data10/base/16385/17158
#   /var/lib/pgsql/data10/base/16385/17160
#   /var/lib/pgsql/data10/base/16385/17162
#   /var/lib/pgsql/data10/base/16385/17164
#   /var/lib/pgsql/data10/base/16385/17165
#   /var/lib/pgsql/data10/base/16385/17166
#   /var/lib/pgsql/data10/base/16385/17167
#   /var/lib/pgsql/data10/base/16385/17168
#   /var/lib/pgsql/data10/base/16385/17169
#   /var/lib/pgsql/data10/base/16385/17170
#   /var/lib/pgsql/data10/base/16385/17171
#   /var/lib/pgsql/data10/base/16385/17173
#   /var/lib/pgsql/data10/base/16385/17174
#   /var/lib/pgsql/data10/base/16385/17175
#   /var/lib/pgsql/data10/base/16385/17176
#   /var/lib/pgsql/data10/base/16385/17177
#   /var/lib/pgsql/data10/base/16385/18295
#   /var/lib/pgsql/data10/base/16385/18319
#   /var/lib/pgsql/data10/base/16385/18320
#   /var/lib/pgsql/data10/base/16385/18341
#   /var/lib/pgsql/data10/base/16385/18360
#   /var/lib/pgsql/data10/base/16385/17189
#   /var/lib/pgsql/data10/base/16385/17190
#   /var/lib/pgsql/data10/base/16385/17191
#   /var/lib/pgsql/data10/base/16385/17192
#   /var/lib/pgsql/data10/base/16385/17193
#   /var/lib/pgsql/data10/base/16385/17194
#   /var/lib/pgsql/data10/base/16385/17195
#   /var/lib/pgsql/data10/base/16385/17196
#   /var/lib/pgsql/data10/base/16385/17197
#   /var/lib/pgsql/data10/base/16385/17198
#   /var/lib/pgsql/data10/base/16385/17199
#   /var/lib/pgsql/data10/base/16385/17200
#   /var/lib/pgsql/data10/base/16385/17996
#   /var/lib/pgsql/data10/base/16385/18061
#   /var/lib/pgsql/data10/base/16385/17204
#   /var/lib/pgsql/data10/base/16385/17205
#   /var/lib/pgsql/data10/base/16385/17206
#   /var/lib/pgsql/data10/base/16385/17207
#   /var/lib/pgsql/data10/base/16385/17208
#   /var/lib/pgsql/data10/base/16385/17209
#   /var/lib/pgsql/data10/base/16385/17210
#   /var/lib/pgsql/data10/base/16385/17211
#   /var/lib/pgsql/data10/base/16385/17213
#   /var/lib/pgsql/data10/base/16385/17214
#   /var/lib/pgsql/data10/base/16385/17215
#   /var/lib/pgsql/data10/base/16385/17216
#   /var/lib/pgsql/data10/base/16385/17217
#   /var/lib/pgsql/data10/base/16385/17218
#   /var/lib/pgsql/data10/base/16385/18211
#   /var/lib/pgsql/data10/base/16385/18212
#   /var/lib/pgsql/data10/base/16385/17221
#   /var/lib/pgsql/data10/base/16385/17222
#   /var/lib/pgsql/data10/base/16385/17223
#   /var/lib/pgsql/data10/base/16385/17224
#   /var/lib/pgsql/data10/base/16385/18005
#   /var/lib/pgsql/data10/base/16385/18096
#   /var/lib/pgsql/data10/base/16385/17227
#   /var/lib/pgsql/data10/base/16385/17228
#   /var/lib/pgsql/data10/base/16385/18019
#   /var/lib/pgsql/data10/base/16385/17917
#   /var/lib/pgsql/data10/base/16385/17918
#   /var/lib/pgsql/data10/base/16385/17232
#   /var/lib/pgsql/data10/base/16385/17233
#   /var/lib/pgsql/data10/base/16385/17234
#   /var/lib/pgsql/data10/base/16385/17235
#   /var/lib/pgsql/data10/base/16385/17236
#   /var/lib/pgsql/data10/base/16385/17237
#   /var/lib/pgsql/data10/base/16385/17238
#   /var/lib/pgsql/data10/base/16385/17239
#   /var/lib/pgsql/data10/base/16385/17240
#   /var/lib/pgsql/data10/base/16385/17241
#   /var/lib/pgsql/data10/base/16385/17242
#   /var/lib/pgsql/data10/base/16385/17243
#   /var/lib/pgsql/data10/base/16385/17244
#   /var/lib/pgsql/data10/base/16385/17245
#   /var/lib/pgsql/data10/base/16385/17246
#   /var/lib/pgsql/data10/base/16385/17247
#   /var/lib/pgsql/data10/base/16385/17248
#   /var/lib/pgsql/data10/base/16385/17862
#   /var/lib/pgsql/data10/base/16385/17866
#   /var/lib/pgsql/data10/base/16385/17868
#   /var/lib/pgsql/data10/base/16385/17869
#   /var/lib/pgsql/data10/base/16385/17912
#   /var/lib/pgsql/data10/base/16385/17914
#   /var/lib/pgsql/data10/base/16385/17921
#   /var/lib/pgsql/data10/base/16385/17925
#   /var/lib/pgsql/data10/base/16385/17927
#   /var/lib/pgsql/data10/base/16385/17928
#   /var/lib/pgsql/data10/base/16385/17930
#   /var/lib/pgsql/data10/base/16385/17937
#   /var/lib/pgsql/data10/base/16385/17940
#   /var/lib/pgsql/data10/base/16385/17944
#   /var/lib/pgsql/data10/base/16385/17946
#   /var/lib/pgsql/data10/base/16385/17947
#   /var/lib/pgsql/data10/base/16385/17954
#   /var/lib/pgsql/data10/base/16385/17957
#   /var/lib/pgsql/data10/base/16385/17961
#   /var/lib/pgsql/data10/base/16385/17963
#   /var/lib/pgsql/data10/base/16385/17964
#   /var/lib/pgsql/data10/base/16385/17966
#   /var/lib/pgsql/data10/base/16385/17973
#   /var/lib/pgsql/data10/base/16385/17990
#   /var/lib/pgsql/data10/base/16385/17992
#   /var/lib/pgsql/data10/base/16385/18001
#   /var/lib/pgsql/data10/base/16385/18003
#   /var/lib/pgsql/data10/base/16385/17993
#   /var/lib/pgsql/data10/base/16385/17995
#   /var/lib/pgsql/data10/base/16385/18013
#   /var/lib/pgsql/data10/base/16385/18015
#   /var/lib/pgsql/data10/base/16385/18034
#   /var/lib/pgsql/data10/base/16385/18038
#   /var/lib/pgsql/data10/base/16385/18040
#   /var/lib/pgsql/data10/base/16385/18041
#   /var/lib/pgsql/data10/base/16385/18045
#   /var/lib/pgsql/data10/base/16385/18049
#   /var/lib/pgsql/data10/base/16385/18055
#   /var/lib/pgsql/data10/base/16385/18057
#   /var/lib/pgsql/data10/base/16385/18062
#   /var/lib/pgsql/data10/base/16385/18064
#   /var/lib/pgsql/data10/base/16385/18071
#   /var/lib/pgsql/data10/base/16385/18082
#   /var/lib/pgsql/data10/base/16385/18083
#   /var/lib/pgsql/data10/base/16385/16950
#   /var/lib/pgsql/data10/base/16385/17172
#   /var/lib/pgsql/data10/base/16385/18089
#   /var/lib/pgsql/data10/base/16385/18091
#   /var/lib/pgsql/data10/base/16385/18296
#   /var/lib/pgsql/data10/base/16385/18297
#   /var/lib/pgsql/data10/base/16385/18342
#   /var/lib/pgsql/data10/base/16385/18343
#   /var/lib/pgsql/data10/base/16385/18298
#   /var/lib/pgsql/data10/base/16385/18363
#   /var/lib/pgsql/data10/base/16385/18321
#   /var/lib/pgsql/data10/base/16385/18344
#   /var/lib/pgsql/data10/base/16385/18299
#   /var/lib/pgsql/data10/base/16385/18364
#   /var/lib/pgsql/data10/base/16385/18213
#   /var/lib/pgsql/data10/base/16385/18345
#   /var/lib/pgsql/data10/base/16385/18365
#   /var/lib/pgsql/data10/base/16385/18292
#   /var/lib/pgsql/data10/base/16385/18294
#   /var/lib/pgsql/data10/base/16385/18315
#   /var/lib/pgsql/data10/base/16385/18317
#   /var/lib/pgsql/data10/base/16385/18338
#   /var/lib/pgsql/data10/base/16385/18340
#   /var/lib/pgsql/data10/base/16385/18366
#   /var/lib/pgsql/data10/base/16385/18373
#   /var/lib/pgsql/data10/base/1/2613
#   /var/lib/pgsql/data10/base/1/2683
#                                                             ok
# Setting next OID for new cluster                            ok
# Sync data directory to disk                                 ok
# Creating script to analyze new cluster                      ok
# Creating script to delete old cluster                       ok
# Checking for extension updates                              ok
# 
# Upgrade Complete
# ----------------
# Optimizer statistics are not transferred by pg_upgrade so,
# once you start the new server, consider running:
#     ./analyze_new_cluster.sh
# 
# Running this script will delete the old cluster's data files:
#     ./delete_old_cluster.sh
# 
# Linking data -> data13
# 
# Starting postgresql
# 
# vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
# vacuumdb: processing database "smartdb": Generating minimal optimizer statistics (1 target)
# vacuumdb: processing database "storageadmin": Generating minimal optimizer statistics (1 target)
# vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
# vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
# vacuumdb: processing database "smartdb": Generating medium optimizer statistics (10 targets)
# vacuumdb: processing database "storageadmin": Generating medium optimizer statistics (10 targets)
# vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
# vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
# vacuumdb: processing database "smartdb": Generating default (full) optimizer statistics
# vacuumdb: processing database "storageadmin": Generating default (full) optimizer statistics
# vacuumdb: processing database "template1": Generating default (full) optimizer statistics
# 
# reindexdb: reindexing database "postgres"
# reindexdb: reindexing database "smartdb"
# reindexdb: reindexing database "storageadmin"
# reindexdb: reindexing database "template1"
# 
# Restoring pior pwd
# /
# 
# All Rockstor services, if running, will have been stopped.
# Restart via 'systemctl start rockstor-bootstrap.service'
# /root/.local/bin/poetry
# Poetry version 1.1.15 found - UNINSTALLING
# Removing Poetry (1.1.15)
# build.sh has PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# creating virtual environment...
# creating shared libraries...
# upgrading shared libraries...
# installing poetry from spec 'poetry==1.7.1'...
# done! ✨ 🌟 ✨#   installed package poetry 1.7.1, installed using Python 3.11.5
#   These apps are now globally available
#     - poetry
# No config-backups-rpmsave directory found.
2024-01-24 18:11:26|patch  |openSUSE-SLE-15.4-2022-1656|1|noarch|repo-sle-update|moderate|recommended|not-needed|applied|

Curiously I've not seen all the `/var/lib/pgsql/data10/base/#/# line outputs before, when running from NOT within %posttrans but from a regular root shell.

And there-after our postgresql systemd service selected the V13 binary to run:

installer:~ # ps aufx | grep '^postgres.* -D'
postgres 14978  0.0  1.2 201368 25900 ?        Ss   18:10   0:00 /usr/lib/postgresql13/bin/postgres -D /var/lib/pgsql/data

@phillxnet
Copy link
Member Author

phillxnet commented Jan 24, 2024

As we have the remaining failure, in testing branch, to successfully install the new .venv, we can test rockstor's function with this updated/rather than freshly created V13 DB format via:

cd /opt/rockstor/
./build.sh
systemctl start rockstor-bootstrap

With what looks to be promising DB tune and migration log results:

[24/Jan/2024 18:31:41] INFO [scripts.initrock:473] API CLIENT_SECRET set/reset successfully.
[24/Jan/2024 18:31:41] INFO [scripts.initrock:569] --DB-- Tune Postgres --DB--
[24/Jan/2024 18:31:41] INFO [scripts.initrock:571] --DB-- Running - Setup_host_based_auth
[24/Jan/2024 18:31:41] INFO [scripts.initrock:576] --DB-- Done with Setup_host_based_auth.
[24/Jan/2024 18:31:41] INFO [scripts.initrock:571] --DB-- Running - Reload_config
[24/Jan/2024 18:31:41] INFO [scripts.initrock:576] --DB-- Done with Reload_config.
[24/Jan/2024 18:31:41] INFO [scripts.initrock:571] --DB-- Running - PG_tune
[24/Jan/2024 18:31:41] INFO [scripts.initrock:576] --DB-- Done with PG_tune.
[24/Jan/2024 18:31:41] INFO [scripts.initrock:577] --DB-- Tune Postgres Done --DB--.
[24/Jan/2024 18:31:41] INFO [scripts.initrock:581] Running app database migrations...
[24/Jan/2024 18:31:58] INFO [scripts.initrock:617] Prior migrations for oauth2_provider are: ['oauth2_provider', ' [X] 0001_initial', ' [ ] 0002_auto_20190406_1805', ' [ ] 0003_auto_20201211_1314', ' [ ] 0004_auto_20200902_2022', ' [ ] 0005_auto_20211222_2352', ' [ ] 0006_alter_application_client_secret', ' [ ] 0007_application_post_logout_redirect_uris', '']
[24/Jan/2024 18:32:03] INFO [scripts.initrock:625] Post migrations for oauth2_provider are: ['oauth2_provider', ' [X] 0001_initial', ' [X] 0002_auto_20190406_1805', ' [X] 0003_auto_20201211_1314', ' [X] 0004_auto_20200902_2022', ' [X] 0005_auto_20211222_2352', ' [X] 0006_alter_application_client_secret', ' [X] 0007_application_post_logout_redirect_uris', '']
[24/Jan/2024 18:32:03] INFO [scripts.initrock:627] DB Migrations Done
[24/Jan/2024 18:32:03] INFO [scripts.initrock:629] Running Django prep_db.
[24/Jan/2024 18:32:05] INFO [scripts.initrock:631] Done

The resulting Web-UI required a full cache refresh vai: Ctrl + Shift then (press-release) R
And bar the failing venv creation within rpm, resolved via above, we have what looks to be a fully working 5.0.6.2785 (with given caveat).

@phillxnet phillxnet closed this Jan 24, 2024
@phillxnet phillxnet deleted the 2780-Establish-Postgres-database-format-upgrade branch January 24, 2024 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant