Postgresql upgrade checklist

Checklist for major postgresql database upgrade:

  • shutdown standby servers
  • on each standby perform pg_upgrade
  • shutdown master
  • perform pg_upgrade
  • if pg_upgrade fails on postgres database with encoding mismatch, re-init database with initdb
  • start master
  • verify master is ok
  • check if recovery.conf present in slave datadir
  • start slaves one-by-one
  • check if slaves are live and recovering
  • if not recovering, check if entry in pg_replication_slots is present
  • if not there create it with pg_create_physical_replication_slot
  • once everything is live, check pg_replication_slots

This procedure will bring DB system down for a while, because we shutdown slaves and the master at once. It should be possible to start slave, while master is not yet upgraded and hope that upgraded slave will deal correctly with non-upgraded data feed from master. Or one can perform a master switch to an already upgraded slave. Both this options, are considered potentially unsafe and should be tested first on non-production database.

For re-creation of standby:

  • perform pg_basebackup on master (rep1 is our replication user)
    master: pg_basebackup -U rep1 -Ft -D - | bzip2 | nc SLAVE-HOST 9999
    slave: nc -l -p 9999 -v > backup.tar.bz2
    
  • unpack backup.tar.gz (replace existing datadir)
  • add recovery.conf to datadir
  • check replication slots as by upgrade