Upgrade Postgres 9.2 to 9.6 #1310

srenatus · 2017-06-16T13:44:38Z

I looks like the old machinery made for going from 9.1 to 9.2 still works just fine! 🎉

Pending:

automatic upgrade tests (qa-chef-server-cluster)
check tunables one-by-one to see if any other ones are gone, clarify release notes
figure out an edge case or bug with the sentinel file creation
cleanup 9.2 stuff

marcparadise · 2017-06-16T14:58:26Z

omnibus/files/private-chef-cookbooks/private-chef/attributes/default.rb

+# Note: the checkpoint_segments setting was removed.
+# https://www.postgresql.org/docs/9.6/static/release-9-5.html says
+#   max_wal_size = (3 * checkpoint_segments) * 16MB
+# would be a usable conversion rule, but it also says the new setting's default


👍 the defaults are sane according to several related docs I read (official, mailng list threads).

srenatus · 2017-06-21T10:48:01Z

I thought there was a problem when, for whatever reason, the upgrade process would fail; I think I saw a situation where we'd still get the sentinel file. However, I've tried reproducing this and couldn't (recorded a video to convince myself).

srenatus · 2017-06-21T11:46:49Z

want to give this another round of qa-chef-server-cluster tests, then I'll remove the [hold]. Reviews would be great, though 😀

srenatus · 2017-06-21T14:53:31Z

🏓 @chef/chef-server-maintainers

marcparadise · 2017-06-21T15:07:47Z

omnibus/config/software/postgresql.rb

@@ -0,0 +1,62 @@
+#


At this point it looks like we might bre able to use the omnibus-software definition of postgres? Or are we still stuck because of the version-specific prefix?
I'm wonder if we should just port that change over to the official definition? Or make an alternative official definition (postgresql-versioned.rb?) that mirrors the pg one except for install path?

🎉 yes good idea. I'll do that.

⏩ chef/omnibus-software#845 Any hints on how to make this less copy-pasty are very welcome!

marcparadise · 2017-06-21T15:08:49Z

omnibus/config/software/postgresql92-bin.rb

@@ -14,7 +14,12 @@
 # limitations under the License.
 #

-name "postgresql92"


Similarly, a shared postgres-versioned-bin-only in omnibus-software?

marcparadise · 2017-06-21T15:17:36Z

omnibus/files/private-chef-cookbooks/private-chef/recipes/post_11_upgrade_cleanup.rb

-#
-# pg_upgrade will helpfully generate a cleanup script which removes
-# the previous database cluster.  This is nice, because we don't have
-# to worry about keeping track of the location of the old cluster.


Removing this gets messy because of enterprise/private/old-oss chef server installations that could be getting upgraded from.
I'm concerned that the cleanup of the old data will no longer happen - because the postgres data was not always in a versioned directory IIRC - there may be older install that have the path as /var/opt/opscode/postgresql/data instead of postgresql/9.2/data.

Perhaps updating the new postgres_upgrade_cleanup to handle both would work?

This might also merit a modification of the upgrade pipeline to verify that old directories are removed.

It's merely moved to a generic location -- because it's now for the cleanup of any postgres upgrade. The comments there made me think that this script will contain the old data dir, irregardless of whether it is /var/opt/opscode/postgresql/data or versioned. (So it wouldn't care about going from 9.1 (unversioned) to 9.2, or 9.1 (unversioned) to 9.2, or 9.2 to 9.6.)

But I can look the left overs in the upgrade pipelines 👍

marcparadise · 2017-06-21T15:20:17Z

RELEASE_NOTES.md

+* [Upgrade to PostgreSQL 9.6](https://github.com/chef/chef-server/pull/1310),
+  Chef Server now uses the latest stable version of the 9.6 series (9.6.3). Upgrades of existing
+  installations are done automatically, but creating backups is advised.
+  Note that the `checkpoint_segments` configuration setting is gone, so if you before have set


A preface like "The information below only applies if you have set a custom value for checkpoint_segments in your /etc/opscode/chef-server.rb. If you have not set a custom value, there is nothing to change" might make it clearer that most folks don't have to worry about this.

Separately:Maybe a preflight check against checkpoint_segments being present?

Yeah you're right. 🙄 (Just kidding. You simple are right there, and I'll add this 😉)

srenatus · 2017-06-22T14:18:42Z

@marcparadise I've checked off all the things from my list now ✔️

Well, a missing bit is checking the deleted old data dir in upgrade tests. But if I find the right place for that test, it'll be in qa-chef-server-cluster anyways. So, could you throw some 👀 at those changes, please? I've intentionally not messed with the history yet, I'll squash and drop before merge.

stevendanna · 2017-06-26T08:08:02Z

omnibus_overrides.rb

@@ -10,3 +10,6 @@
 override :rubygems, version: "2.6.8"
 # This SHA is the last commit before the 6.0 release
 override :'berkshelf-no-depselector', version: '6016ca10b2f46508b1b107264228668776f505d9'
+
+override :'postgresql-bin-versioned', version: "9.2.21" # needed for upgrading


[minor] I wonder if we want to put the major version in the package name. That is, this package would be called postgresql92-bin.

That's actually what was there before -- but then commenters convinced me that there's a value in having postgres-bin-versioned in omnibus-software, which can be used for any version we'd like to upgrade from. However, that bit hasn't passed review yet either

("commenters" = @marcparadise here, and @sdelano in slack)

stevendanna

Overall this looks good to me.

ryancragun · 2017-06-26T16:57:29Z

This looks good to me with the assumption that the omnibus/omnibus-software changes on your branches are merged upstream and we go back to following the master branch.

srenatus · 2017-06-26T17:13:14Z

@ryancragun yeah I'll take care of cross-testing those things before merging both tomorrow 😃

srenatus · 2017-06-27T16:18:27Z

Re: [hold]: This is probably ok, but needs one final verification with the packages from here.

Signed-off-by: Stephan Renatus <srenatus@chef.io>

* warn if removed pg setting is configured * check requirements >= 9.2, not 9.6 Rationale: let's burn that incompatibility bridge when we get there. Currently, we ship 9.6 but we don't require any of its features. Fixes #441. Signed-off-by: Stephan Renatus <srenatus@chef.io>

remove checkpoint_segments, expose max_wal_size, min_wal_size, and checkpoint_flush_after. This follows the logic we've had before: all options regarding "Checkpoints" and "Archiving" (19.5.2 and 19.5.3 in https://www.postgresql.org/docs/9.6/static/runtime-config-wal.html) are exposed, and given their defaults as default attribute values. I have not filled in the values from 19.5.1, because we haven't done that before either. Signed-off-by: Stephan Renatus <srenatus@chef.io>

Signed-off-by: Stephan Renatus <srenatus@chef.io>

I've moved that resource from post_11_upgrade_cleanup. While it would in just the same way from _there_, this should make the intention clearer. Also, we'll delete the old log files of 9.2 now. The addition the the release notes would theoretically be unnecessary since this is what the postinst[1] instructions say anyways. However, I believe it doesn't hurt to point it out again. [1] https://github.com/chef/chef-server/blob/master/omnibus/package-scripts/chef-server/postinst#L30-L52 Signed-off-by: Stephan Renatus <srenatus@chef.io>

This brings in changes from chef-backend (at 55caa8742b6). While that new old_data_dir handling doesn't depend on chef-server-running.json being written to disk successfully, I'm not convinced that the filter logic in that code actually works if there's more than one data dir on disk. Given that reviewers agree, I'll "backport" this change to chef-backend. (Maybe we should look into a shared cookbook again, though?) Signed-off-by: Stephan Renatus <srenatus@chef.io>

Signed-off-by: Stephan Renatus <srenatus@chef.io>

marcparadise reviewed Jun 16, 2017

View reviewed changes

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 2 times, most recently from 291a0e8 to 6e57566 Compare June 21, 2017 10:45

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 2 times, most recently from e62713e to b41db1a Compare June 21, 2017 10:58

srenatus changed the title ~~[hold] Upgrade Postgres 9.2 to 9.6~~ Upgrade Postgres 9.2 to 9.6 Jun 21, 2017

srenatus changed the title ~~Upgrade Postgres 9.2 to 9.6~~ [hold] Upgrade Postgres 9.2 to 9.6 Jun 21, 2017

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch from bdf676c to c15289e Compare June 21, 2017 11:36

srenatus changed the title ~~[hold] Upgrade Postgres 9.2 to 9.6~~ Upgrade Postgres 9.2 to 9.6 Jun 21, 2017

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch from 2b4634a to 2d7578f Compare June 21, 2017 14:36

marcparadise reviewed Jun 21, 2017

View reviewed changes

srenatus changed the title ~~Upgrade Postgres 9.2 to 9.6~~ [hold] Upgrade Postgres 9.2 to 9.6 Jun 22, 2017

srenatus changed the title ~~[hold] Upgrade Postgres 9.2 to 9.6~~ [wip] Upgrade Postgres 9.2 to 9.6 Jun 22, 2017

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 6 times, most recently from 1328977 to db8fef2 Compare June 22, 2017 12:36

srenatus changed the title ~~[wip] Upgrade Postgres 9.2 to 9.6~~ Upgrade Postgres 9.2 to 9.6 Jun 22, 2017

stevendanna reviewed Jun 26, 2017

View reviewed changes

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 2 times, most recently from 30c44fb to 667118e Compare June 27, 2017 08:45

srenatus mentioned this pull request Jun 27, 2017

[hold] postgresql: add versioned and bin-only versioned variants chef/omnibus-software#845

Closed

1 task

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch from 667118e to e27882f Compare June 27, 2017 13:19

srenatus changed the title ~~Upgrade Postgres 9.2 to 9.6~~ [hold] Upgrade Postgres 9.2 to 9.6 Jun 27, 2017

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 5 times, most recently from add1d29 to c651468 Compare June 28, 2017 09:35

omnibus: add postgres92-bin and postgres96

a74d51e

Signed-off-by: Stephan Renatus <srenatus@chef.io>

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch 2 times, most recently from 7f226c8 to 6055bce Compare June 28, 2017 10:11

srenatus changed the title ~~[hold] Upgrade Postgres 9.2 to 9.6~~ Upgrade Postgres 9.2 to 9.6 Jun 28, 2017

srenatus added 6 commits June 28, 2017 14:23

RELEASE_NOTES: PostgreSQL 9.6.3

c743bc4

Signed-off-by: Stephan Renatus <srenatus@chef.io>

travis: bump postgres 9.3 -> 9.6

664cc29

Signed-off-by: Stephan Renatus <srenatus@chef.io>

srenatus force-pushed the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch from 6055bce to 664cc29 Compare June 28, 2017 12:24

stevendanna approved these changes Jun 28, 2017

View reviewed changes

stevendanna merged commit fc7abfa into master Jun 28, 2017

srenatus deleted the sr/pool-634/upgrade-postgres-9.2-to-9.6 branch June 28, 2017 12:44

btm mentioned this pull request Oct 24, 2019

Update postgresql from 9.2.9 to latest #181

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Postgres 9.2 to 9.6 #1310

Upgrade Postgres 9.2 to 9.6 #1310

srenatus commented Jun 16, 2017 •

edited

Loading

marcparadise Jun 16, 2017

srenatus commented Jun 21, 2017

srenatus commented Jun 21, 2017

srenatus commented Jun 21, 2017

marcparadise Jun 21, 2017 •

edited

Loading

srenatus Jun 21, 2017

srenatus Jun 21, 2017

marcparadise Jun 21, 2017

marcparadise Jun 21, 2017

srenatus Jun 21, 2017

marcparadise Jun 21, 2017 •

edited

Loading

srenatus Jun 21, 2017

srenatus commented Jun 22, 2017

stevendanna Jun 26, 2017

srenatus Jun 26, 2017 •

edited

Loading

srenatus Jun 26, 2017

stevendanna left a comment

ryancragun commented Jun 26, 2017

srenatus commented Jun 26, 2017

srenatus commented Jun 27, 2017

Upgrade Postgres 9.2 to 9.6 #1310

Upgrade Postgres 9.2 to 9.6 #1310

Conversation

srenatus commented Jun 16, 2017 • edited Loading

Choose a reason for hiding this comment

srenatus commented Jun 21, 2017

srenatus commented Jun 21, 2017

srenatus commented Jun 21, 2017

marcparadise Jun 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcparadise Jun 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srenatus commented Jun 22, 2017

Choose a reason for hiding this comment

srenatus Jun 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevendanna left a comment

Choose a reason for hiding this comment

ryancragun commented Jun 26, 2017

srenatus commented Jun 26, 2017

srenatus commented Jun 27, 2017

srenatus commented Jun 16, 2017 •

edited

Loading

marcparadise Jun 21, 2017 •

edited

Loading

marcparadise Jun 21, 2017 •

edited

Loading

srenatus Jun 26, 2017 •

edited

Loading