pgBackRest User Guide - Debian & Ubuntu / PostgreSQL 9.4

Introduction

This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Backup section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.

Although the examples are targeted at Debian/Ubuntu and PostgreSQL 9.4, it should be fairly easy to apply this guide to any Unix distribution and PostgreSQL version. Note that only 64-bit distributions are currently supported due to 64-bit operations in the Perl code. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the locations to install Perl libraries and executables may vary.

Configuration information and documentation for PostgreSQL can be found in the PostgreSQL Manual.

A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.

All commands are intended to be run as an unprivileged user that has sudo privileges for both the root and postgres users. It's also possible to run the commands directly as their respective users without modification and in that case the sudo commands can be stripped off.

Concepts

The following concepts are defined as they are relevant to pgBackRest, PostgreSQL, and this user guide.

Backup

A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new standby.

Full Backup: pgBackRest copies the entire contents of the database cluster to the backup server. The first backup of the database cluster is always a Full Backup. pgBackRest is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.

Differential Backup: pgBackRest copies only those database cluster files that have changed since the last full backup. pgBackRest restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.

Incremental Backup: pgBackRest copies only those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.

Restore

A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.

Write Ahead Log (WAL)

WAL is the mechanism that PostgreSQL uses to ensure that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.

WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention 0000000100000A1E000000FE where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the logical sequence number (LSN).

Installation

pgBackRest is written in Perl which is included with Debian/Ubuntu by default. The DBD::Pg module must also be installed.

db-master ⇒ Install the DBD::Pg module

sudo apt-get install libdbd-pg-perl

No Debian/Ubuntu packages are currently available for pgBackRest but it is easy to download the source and install manually.

db-master ⇒ Download version 1.03 of pgBackRest

wget -q -O - \
       https://github.com/pgbackrest/pgbackrest/archive/release/1.03.tar.gz | \
       tar zx -C ~

If pgBackRest has been installed before it's best to be sure that no prior copies of it are still installed. Depending on how old the version of pgBackRest is it may have been installed in a few different locations. The following commands will remove all prior versions of pgBackRest.

db-master ⇒ Remove prior pgBackRest installations

sudo rm -f /usr/bin/pgbackrest

sudo rm -f /usr/bin/pg_backrest

sudo rm -rf /usr/lib/perl5/BackRest

sudo rm -rf /usr/share/perl5/BackRest

sudo rm -rf /usr/lib/perl5/pgBackRest

sudo rm -rf /usr/share/perl5/pgBackRest

The new version can now be installed.

db-master ⇒ Install pgBackRest

sudo cp -r ~/pgbackrest-release-1.03/lib/pgBackRest \
       /usr/share/perl5

sudo find /usr/share/perl5/pgBackRest -type f -exec chmod 644 {} +

sudo find /usr/share/perl5/pgBackRest -type d -exec chmod 755 {} +

sudo cp ~/pgbackrest-release-1.03/bin/pgbackrest /usr/bin/pgbackrest

sudo chmod 755 /usr/bin/pgbackrest

sudo mkdir -m 770 /var/log/pgbackrest

sudo chown postgres:postgres /var/log/pgbackrest

pgBackRest should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running pgBackRest from the command line.

db-master ⇒ Make sure the installation worked

sudo -u postgres pgbackrest

pgBackRest 1.03 - General help

Usage:
    pgbackrest [options] [command]

Commands:
    archive-get   Get a WAL segment from the archive.
    archive-push  Push a WAL segment to the archive.
    backup        Backup a database cluster.
    check         Check the configuration.
    expire        Expire backups that exceed retention.
    help          Get help.
    info          Retrieve information about backups.
    restore       Restore a database cluster.
    start         Allow pgBackRest processes to run.
    stop          Stop pgBackRest processes from running.
    version       Get version.

Use 'pgbackrest help [command]' for more information.

Quick Start

The Quick Start section will cover basic configuration of pgBackRest and PostgreSQL and introduce the backup, restore, and info commands.

Setup Demo Cluster

Creating the demo cluster is optional but is strongly recommended, especially for new users, since the example commands in the user guide reference the demo cluster; the examples assume the demo cluster is running on the default port (i.e. 5432). The cluster will not be started until a later section because there is still some configuration to do.

db-master ⇒ Create the demo cluster

sudo pg_createcluster 9.4 demo

Creating new cluster 9.4/demo ...
  config /etc/postgresql/9.4/demo
  data   /var/lib/postgresql/9.4/demo
  locale C
  socket /var/run/postgresql
  port   5432

By default PostgreSQL will only accept local connections. The examples in this guide will require connections from other servers so listen_addresses is configured to listen on all interfaces. This may not be appropriate for secure installations.

db-master:/etc/postgresql/9.4/demo/postgresql.conf ⇒ Set listen_addresses

listen_addresses = '*'

For demonstration purposes the log_line_prefix setting will be minimally configured. This keeps the log output as brief as possible to better illustrate important information.

db-master:/etc/postgresql/9.4/demo/postgresql.conf ⇒ Set log_line_prefix

listen_addresses = '*'
log_line_prefix = ''

Configure Cluster Stanza

A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one Postgres database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.

It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod.

The name 'demo' describes the purpose of this cluster accurately so that will also make a good stanza name.

pgBackRest needs to know where the base data directory for the PostgreSQL cluster is located. The path can be requested from PostgreSQL directly but in a recovery scenario the PostgreSQL process will not be available. During backups the value supplied to pgBackRest will be compared against the path that PostgreSQL is running on and they must be equal or the backup will return an error. Make sure that db-path is exactly equal to data_directory in postgresql.conf.

By default Debian/Ubuntu stores clusters in /var/lib/postgresql/[version]/[cluster] so it is easy to determine the correct path for the data directory.

When creating the /etc/pgbackrest.conf file, the database owner (usually postgres) must be granted read privileges.

db-master:/etc/pgbackrest.conf ⇒ Configure the PostgreSQL cluster data directory

[demo]
db-path=/var/lib/postgresql/9.4/demo

pgBackRest configuration files follow the Windows INI convention. Sections are denoted by text in brackets and key/value pairs are contained in each section. Lines beginning with # are ignored and can be used as comments.

Create the Repository

The repository is where pgBackRest stores backup and archives WAL segments.

If you are new to backup then it will be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves.

For this demonstration the repository will be stored on the same host as the PostgreSQL server. This is the simplest configuration and is useful in cases where traditional backup software is employed to backup the database host.

db-master ⇒ Create the pgBackRest repository

sudo mkdir /var/lib/pgbackrest

sudo chmod 750 /var/lib/pgbackrest

sudo chown postgres:postgres /var/lib/pgbackrest

The repository path must be configured so pgBackRest knows where to find it.

db-master:/etc/pgbackrest.conf ⇒ Configure the pgBackRest repository path

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest

Configure Archiving

Backing up a running PostgreSQL cluster requires WAL archiving to be enabled. Note that at least one WAL segment will be created during the backup process even if no explicit writes are made to the cluster.

db-master:/etc/postgresql/9.4/demo/postgresql.conf ⇒ Configure archive settings

archive_command = 'pgbackrest --stanza=demo archive-push %p'
archive_mode = on
listen_addresses = '*'
log_line_prefix = ''
max_wal_senders = 3
wal_level = hot_standby

The wal_level setting must be set to archive at a minimum but hot_standby and logical also work fine for backups. Setting wal_level to hot_standy and increasing max_wal_senders is a good idea even if you do not currently run a hot standby as this will allow them to be added later without restarting the master cluster.

The PostgreSQL cluster must be restarted after making these changes and before performing a backup.

db-master ⇒ Restart the demo cluster

sudo pg_ctlcluster 9.4 demo restart

Check the Configuration

The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the database or the backup host.

Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog() are called to force PostgreSQL to archive a WAL segment. Restore points are only supported in PostgreSQL >= 9.1 so for older versions the check command may fail if there has been no write activity since the last log rotation.

db-master ⇒ Check the configuration

sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check

  INFO: check start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo
  INFO: switch xlog 000000010000000000000001

  INFO: WAL segment 000000010000000000000001 successfully stored in the archive at '/var/lib/pgbackrest/archive/demo/9.4-1/0000000100000000/000000010000000000000001-45c1331c9cd2fae667f2a85e9dddd5673ef8e5b0.gz'

  INFO: check stop

Perform a Backup

To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command.

db-master ⇒ Backup the demo cluster

sudo -u postgres pgbackrest --stanza=demo \
       --log-level-console=info backup

  INFO: backup start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo

  WARN: no prior backup exists, incr backup has been changed to full

  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:08:40": backup begins after the next regular checkpoint completes
  INFO: archive start: 0/2000028
       [filtered 759 lines of output]
  INFO: backup file /var/lib/postgresql/9.4/demo/base/1/11885 (0B, 100%)
  INFO: backup file /var/lib/postgresql/9.4/demo/global/pg_control (8KB) checksum 79ea8211b81a7d657767f3ed12fb384c915462b0

  INFO: full backup size = 19.3MB

  INFO: execute exclusive pg_stop_backup() and wait for all WAL segments to archive
  INFO: archive stop: 0/20000F0
       [filtered 4 lines of output]

By default pgBackRest will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed pgBackRest ran a full backup instead.

The type option can be used to specify a full or differential backup.

db-master ⇒ Differential backup of the demo cluster

sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup

       [filtered 4 lines of output]
  INFO: backup file /var/lib/postgresql/9.4/demo/backup_label (236B, 100%) checksum 5a3c99e525cd0ecd3ce99f028cd7ea84d78ff613
  INFO: backup file /var/lib/postgresql/9.4/demo/global/pg_control (8KB) checksum 5413b0dab8460b2ba833de8b6cde194416d41901

  INFO: diff backup size = 8.2KB

  INFO: execute exclusive pg_stop_backup() and wait for all WAL segments to archive
  INFO: archive stop: 0/30000F0
       [filtered 4 lines of output]

This time there was no warning because a full backup already existed. While incremental backups can be based on a full or differential backup, differential backups must be based on a full backup. A full backup can be performed by running the backup command with --type=full.

More information about the backup command can be found in the Backup section.

Backup Information

Use the info command to get information about backups.

db-master ⇒ Get info for the demo cluster

sudo -u postgres pgbackrest info

stanza: demo
    status: ok

    full backup: 20160702-140845F

        start / stop timestamp: 2016-07-02 14:08:40 / 2016-07-02 14:08:45
        database size: 19.3MB, backup size: 19.3MB
        repository size: 2.2MB, repository backup size: 2.2MB

    diff backup: 20160702-140845F_20160702-140848D

        start / stop timestamp: 2016-07-02 14:08:45 / 2016-07-02 14:08:48
        database size: 19.3MB, backup size: 8.2KB
        repository size: 2.2MB, repository backup size: 345B
        backup reference list: 20160702-140845F

The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).

The 'start / stop timestamp' defines the time period when the backup ran. The 'stop timestamp' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.

The 'database size' is the full uncompressed size of the database while 'backup size' is the amount of data actually backed up (these will be the same for full backups). The 'repository size' includes all the files from this backup and any referenced backups that are required to restore the database while 'repository backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest or the filesystem.

The 'backup reference list' contains the additional backups that are required to restore this backup.

Restore a Backup

Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important PostgreSQL cluster file.

db-master ⇒ Stop the demo cluster and delete the pg_control file

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres rm /var/lib/postgresql/9.4/demo/global/pg_control

Starting the cluster without this important file will result in an error.

db-master ⇒ Attempt to start the corrupted demo cluster

sudo pg_ctlcluster 9.4 demo start

The PostgreSQL server failed to start. Please check the log output:

postgres: could not find the database system

Expected to find it in the directory "/var/lib/postgresql/9.4/demo",
but could not open file "/var/lib/postgresql/9.4/demo/global/pg_control": No such file or directory

To restore a backup of the PostgreSQL cluster run pgBackRest with the restore command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the PostgreSQL data directory.

db-master ⇒ Remove old files from demo cluster

sudo -u postgres find /var/lib/postgresql/9.4/demo -mindepth 1 -delete

db-master ⇒ Restore the demo cluster and start PostgreSQL

sudo -u postgres pgbackrest --stanza=demo restore

sudo pg_ctlcluster 9.4 demo start

This time the cluster started successfully since the restore replaced the missing pg_control file.

More information about the restore command can be found in the Restore section.

Backup

The Backup section introduces additional backup command features.

Fast Start Option

By default pgBackRest will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the checkpoint_timeout and checkpoint_segments settings in PostgreSQL it may be quite some time before a checkpoint completes and the backup can begin.

db-master ⇒ Incremental backup of the demo cluster with the regularly scheduled checkpoint

sudo -u postgres pgbackrest --stanza=demo --type=incr \
       --log-level-console=info backup

  INFO: backup start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo --type=incr
  INFO: last backup label = 20160702-140845F_20160702-140848D, version = 1.03

  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:08:58": backup begins after the next regular checkpoint completes

  INFO: archive start: 0/5000028
  INFO: backup file /var/lib/postgresql/9.4/demo/base/12139/pg_internal.init (115.2KB, 71%) checksum b997e54b258af88b244f2dca258a02e7d668b72a
       [filtered 15 lines of output]

When --start-fast is passed on the command-line or start-fast=y is set in /etc/pgbackrest.conf an immediate checkpoint is requested and the backup will start more quickly. This is convenient for testing and for ad-hoc backups. For instance, if a backup is being taken at the beginning of a release window it makes no sense to wait for a checkpoint. Since regularly scheduled backups generally only happen once per day it is unlikely that enabling the start-fast in /etc/pgbackrest.conf will negatively affect performance, however for high-volume transactional systems you may want to pass --start-fast on the command-line instead. Alternately, it is possible to override the setting in the configuration file by passing --no-start-fast on the command-line.

db-master:/etc/pgbackrest.conf ⇒ Enable the start-fast option

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest
start-fast=y

db-master ⇒ Incremental backup of the demo cluster with an immediate checkpoint

sudo -u postgres pgbackrest --stanza=demo --type=incr \
       --log-level-console=info backup

  INFO: backup start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo --start-fast --type=incr
  INFO: last backup label = 20160702-140845F_20160702-140901I, version = 1.03

  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:09:02": backup begins after the requested immediate checkpoint completes

  INFO: archive start: 0/6000028
  INFO: backup file /var/lib/postgresql/9.4/demo/backup_label (236B, 100%) checksum a8e281488bedaf3321a2d0e527110e5896c19a24
       [filtered 8 lines of output]

Automatic Stop Option

Sometimes pgBackRest will exit unexpectedly and the backup in progress on the PostgreSQL cluster will not be properly stopped. pgBackRest exits as quickly as possible when an error occurs so that the cause can be reported accurately and is not masked by another problem that might happen during a more extensive cleanup.

Here an error is intentionally caused by removing repository permissions.

db-master ⇒ Revoke write privileges in the pgBackRest repository and attempt a backup

sudo chmod 550 /var/lib/pgbackrest/temp

sudo -u postgres pgbackrest --stanza=demo --type=incr \
       --log-level-console=info backup

       [filtered 2 lines of output]
  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:09:05": backup begins after the requested immediate checkpoint completes
  INFO: archive start: 0/7000028

 ERROR: [122]: unable to create /var/lib/pgbackrest/temp/demo.tmp: Permission denied

  INFO: backup stop

Even when the permissions are fixed pgBackRest will still be unable to perform a backup because the PostgreSQL cluster is stuck in backup mode.

db-master ⇒ Restore write privileges in the pgBackRest repository and attempt a backup

sudo chmod 750 /var/lib/pgbackrest/temp

sudo -u postgres pgbackrest --stanza=demo --type=incr \
       --log-level-console=info backup

  INFO: backup start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo --start-fast --type=incr
  INFO: last backup label = 20160702-140845F_20160702-140905I, version = 1.03
  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:09:07": backup begins after the requested immediate checkpoint completes

 ERROR: [132]: ERROR:  a backup is already in progress

              HINT:  Run pg_stop_backup() and try again.:
              select to_char(current_timestamp, 'YYYY-MM-DD HH24:MI:SS.US TZ'), lsn::text from pg_start_backup('pgBackRest backup started at 2016-07-02 14:09:07', true) as lsn

Enabling the stop-auto option allows pgBackRest to stop the current backup if it detects that no other pgBackRest backup process is running.

db-master:/etc/pgbackrest.conf ⇒ Enable the stop-auto option

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest
start-fast=y
stop-auto=y

Now pgBackRest will stop the old backup and start a new one so the process completes successfully.

db-master ⇒ Perform an incremental backup

sudo -u postgres pgbackrest --stanza=demo --type=incr \
       --log-level-console=info backup

  INFO: backup start: --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo --start-fast --stop-auto --type=incr
  INFO: last backup label = 20160702-140845F_20160702-140905I, version = 1.03

  WARN: the cluster is already in backup mode but no backup process is running. pg_stop_backup() will be called so a new backup can be started.

  INFO: execute exclusive pg_stop_backup() and wait for all WAL segments to archive

  INFO: execute exclusive pg_start_backup() with label "pgBackRest backup started at 2016-07-02 14:09:08": backup begins after the requested immediate checkpoint completes

  INFO: archive start: 0/9000028
  INFO: backup file /var/lib/postgresql/9.4/demo/backup_label (236B, 100%) checksum 528109da415f9fc9109bc7ac6c51f1364b3188e6
       [filtered 8 lines of output]

Although useful this feature may not be appropriate when another third-party backup solution is being used to take online backups as pgBackRest will not recognize that the other software is running and may terminate a backup started by that software. However, it would be unusual to run more than one third-party backup solution at the same time so this is not likely to be a problem.

Note that pg_dump and pg_base_backup do not take online backups so are not affected. It is safe to run them in conjunction with pgBackRest.

Retention

Generally it is best to retain as many backups as possible to provide a greater window for Point-in-Time Recovery, but practical concerns such as disk space must also be considered. Retention options remove older backups once they are no longer needed.

Full Backup Retention

Set full-retention to the number of full backups required. New backups must be completed before expiration will occur — that means if retention-full=2 then there will be three full backups stored before the oldest one is expired.

db-master:/etc/pgbackrest.conf ⇒ Configure retention-full

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest
retention-full=2
start-fast=y
stop-auto=y

Backup retention-full=2 but currently there is only one full backup so the next full backup to run will not expire any full backups.

db-master ⇒ Perform a full backup

sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=detail backup

       [filtered 768 lines of output]
  INFO: backup stop
  INFO: expire start: --log-level-console=detail --repo-path=/var/lib/pgbackrest --retention-full=2 --stanza=demo

DETAIL: archive retention on backup 20160702-140845F, start = 000000010000000000000002
DETAIL: remove archive: start = 000000010000000000000001, stop = 000000010000000000000001

  INFO: expire stop

Archive is expired because WAL segments were generated before the oldest backup. These are not useful for recovery — only WAL segments generated after a backup can be used to recover that backup.

db-master ⇒ Perform a full backup

sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=info backup

       [filtered 768 lines of output]
  INFO: backup stop
  INFO: expire start: --log-level-console=info --repo-path=/var/lib/pgbackrest --retention-full=2 --stanza=demo

  INFO: expire full backup set: 20160702-140845F, 20160702-140845F_20160702-140848D, 20160702-140845F_20160702-140901I, 20160702-140845F_20160702-140905I, 20160702-140845F_20160702-140912I

  INFO: remove expired backup 20160702-140845F_20160702-140912I
  INFO: remove expired backup 20160702-140845F_20160702-140905I
       [filtered 3 lines of output]

The 20160702-140845F full backup is expired and archive retention is based on the 20160702-140918F which is now the oldest full backup.

Differential Backup Retention

Set retention-diff to the number of differential backups required. Differentials only rely on the prior full backup so it is possible to create a rolling set of differentials for the last day or more. This allows quick restores to recent points-in-time but reduces overall space consumption.

db-master:/etc/pgbackrest.conf ⇒ Configure retention-diff

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest
retention-diff=1
retention-full=2
start-fast=y
stop-auto=y

Backup retention-diff=1 so two differentials will need to be performed before one is expired. An incremental backup is added to demonstrate incremental expiration. Incremental backups cannot be expired independently — they are always expired with their related full or differential backup.

db-master ⇒ Perform differential and incremental backups

sudo -u postgres pgbackrest --stanza=demo --type=diff backup

sudo -u postgres pgbackrest --stanza=demo --type=incr backup

Now performing a differential backup will expire the previous differential and incremental backups leaving only one differential backup.

db-master ⇒ Perform a differential backup

sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup

       [filtered 10 lines of output]
  INFO: backup stop
  INFO: expire start: --log-level-console=info --repo-path=/var/lib/pgbackrest --retention-diff=1 --retention-full=2 --stanza=demo

  INFO: expire diff backup set: 20160702-140923F_20160702-140927D, 20160702-140923F_20160702-140930I

  INFO: remove expired backup 20160702-140923F_20160702-140930I
  INFO: remove expired backup 20160702-140923F_20160702-140927D

Restore

The Restore section introduces additional restore command features.

Delta Option

Restore a Backup in Quick Start required the database cluster directory to be cleaned before the restore could be performed. The delta option allows pgBackRest to automatically determine which files in the database cluster directory can be preserved and which ones need to be restored from the backup — it also removes files not present in the backup manifest so it will dispose of divergent changes. This is accomplished by calculating a SHA-1 cryptographic hash for each file in the database cluster directory. If the SHA-1 hash does not match the hash stored in the backup then that file will be restored. This operation is very efficient when combined with the thread-max option. Since the PostgreSQL process is shut down during the restore, a larger number of threads can be used than might be desirable during a backup when the PostgreSQL process is running.

db-master ⇒ Stop the demo cluster, perform delta restore

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta \
       --log-level-console=detail restore

       [filtered 691 lines of output]
DETAIL: restore file /var/lib/postgresql/9.4/demo/base/12134/PG_VERSION - exists and matches backup (4B, 99%) checksum 8dbabb96e032b8d9f1993c0e4b9141e71ade01a1
DETAIL: restore file /var/lib/postgresql/9.4/demo/base/1/PG_VERSION - exists and matches backup (4B, 99%) checksum 8dbabb96e032b8d9f1993c0e4b9141e71ade01a1

DETAIL: restore file /var/lib/postgresql/9.4/demo/PG_VERSION - exists and matches backup (4B, 100%) checksum 8dbabb96e032b8d9f1993c0e4b9141e71ade01a1

DETAIL: restore file /var/lib/postgresql/9.4/demo/global/12086 - exists and is zero size (0B, 100%)
DETAIL: restore file /var/lib/postgresql/9.4/demo/global/12038 - exists and is zero size (0B, 100%)
       [filtered 83 lines of output]
DETAIL: restore file /var/lib/postgresql/9.4/demo/base/1/11885 - exists and is zero size (0B, 100%)
  INFO: wrote /var/lib/postgresql/9.4/demo/recovery.conf

  INFO: restore global/pg_control (copied last to ensure aborted restores cannot be started)

  INFO: restore file /var/lib/postgresql/9.4/demo/global/pg_control (8KB) checksum 504a37019cf088b5b12c7b13ca862601f49ef168
  INFO: restore stop

db-master ⇒ Restart PostgreSQL

sudo pg_ctlcluster 9.4 demo start

Restore Selected Databases

There may be cases where it is desirable to selectively restore specific databases from a cluster backup. This could be done for performance reasons or to move selected databases to a machine that does not have enough space to restore the entire cluster backup.

To demonstrate this feature two databases are created: test1 and test2. A fresh backup is run so pgBackRest is aware of the new databases.

db-master ⇒ Create two test databases and perform a backup

sudo -u postgres psql -c "create database test1;"

CREATE DATABASE

sudo -u postgres psql -c "create database test2;"

CREATE DATABASE

sudo -u postgres pgbackrest --stanza=demo --type=incr backup

Each test database will be seeded with tables and data to demonstrate that recovery works with selective restore.

db-master ⇒ Create a test table in each database

sudo -u postgres psql -c "create table test1_table (id int); \
       insert into test1_table (id) values (1);" test1

INSERT 0 1

sudo -u postgres psql -c "create table test2_table (id int); \
       insert into test2_table (id) values (2);" test2

INSERT 0 1

One of the main reasons to use selective restore is to save space. The size of the test1 database is shown here so it can be compared with the disk utilization after a selective restore.

db-master ⇒ Show space used by test1 database

sudo -u postgres du -sh /var/lib/postgresql/9.4/demo/base/16384

6.4M	/var/lib/postgresql/9.4/demo/base/16384

Stop the cluster and restore only the test2 database. Built-in databases (template0, template1, and postgres) are always restored.

db-master ⇒ Restore from last backup including only the test2 database

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta \
       --db-include=test2 restore

sudo pg_ctlcluster 9.4 demo start

Once recovery is complete the test2 database will contain all previously created tables and data.

db-master ⇒ Demonstrate that the test2 database was recovered

sudo -u postgres psql -c "select * from test2_table;" test2

 id 
----
  2
(1 row)

The test1 database, despite successful recovery, is not accessible. This is because the entire database was restored as sparse, zeroed files. PostgreSQL can successfully apply WAL on the zeroed files but the database as a whole will not be valid because key files contain no data. This is purposeful to prevent the database from being accidentally used when it might contain partial data that was applied during WAL replay.

db-master ⇒ Attempting to connect to the test1 database will produce an error

sudo -u postgres psql -c "select * from test1_table;" test1

psql: FATAL:  relation mapping file "base/16384/pg_filenode.map" contains invalid data

Since the test1 database is restored with sparse, zeroed files it will only require as much space as the amount of WAL that is written during recovery. While the amount of WAL generated during a backup and applied during recovery can be significant it will generally be a small fraction of the total database size, especially for large databases where this feature is most likely to be useful.

It is clear that the test1 database uses far less disk space during the selective restore than it would have if the entire database had been restored.

db-master ⇒ Show space used by test1 database after recovery

sudo -u postgres du -sh /var/lib/postgresql/9.4/demo/base/16384

152K	/var/lib/postgresql/9.4/demo/base/16384

At this point the only action that can be taken on the invalid test1 database is drop database. pgBackRest does not automatically drop the database since this cannot be done until recovery is complete and the cluster is accessible.

db-master ⇒ Drop the test1 database

sudo -u postgres psql -c "drop database test1;"

DROP DATABASE

Now that the invalid test1 database has been dropped only the test2 and built-in databases remain.

db-master ⇒ List remaining databases

sudo -u postgres psql -c "select oid, datname from pg_database order by oid;"

  oid  |  datname  
-------+-----------
     1 | template1
 12134 | template0
 12139 | postgres

 16385 | test2

(4 rows)

Point-in-Time Recovery

Restore a Backup in Quick Start performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is usually the best choice but for data corruption scenarios (whether machine or human in origin) Point-in-Time Recovery (PITR) is often more appropriate.

Point-in-Time Recovery (PITR) allows the WAL to be played from the last backup to a specified time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A typical recovery scenario is to restore a table that was accidentally dropped or data that was accidentally deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.

db-master ⇒ Backup the demo cluster and create a table with very important data

sudo -u postgres pgbackrest --stanza=demo --type=diff backup

sudo -u postgres psql -c "begin; \
       create table important_table (message text); \
       insert into important_table values ('Important Data'); \
       commit; \
       select * from important_table;"

    message     
----------------

 Important Data

(1 row)

It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.

db-master ⇒ Get the time from PostgreSQL

sudo -u postgres psql -Atc "select current_timestamp"

2016-07-02 14:09:58.282805+00

Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.

db-master ⇒ Drop the important table

sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"

ERROR:  relation "important_table" does not exist

LINE 1: ...le important_table;     commit;     select * from important_...
                                                             ^

Now the restore can be performed with time-based recovery to bring back the missing table.

db-master ⇒ Stop PostgreSQL, restore the demo cluster to 2016-07-02 14:09:58.282805+00, and display recovery.conf

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=2016-07-02 14:09:58.282805+00" restore

sudo -u postgres cat /var/lib/postgresql/9.4/demo/recovery.conf

restore_command = '/usr/bin/pgbackrest --stanza=demo archive-get %f "%p"'

recovery_target_time = '2016-07-02 14:09:58.282805+00'

The recovery.conf file has been automatically generated by pgBackRest so PostgreSQL can be started immediately. Once PostgreSQL has finished recovery the table will exist again and can be queried.

db-master ⇒ Start PostgreSQL and check that the important table exists

sudo pg_ctlcluster 9.4 demo start

sudo -u postgres psql -c "select * from important_table"

    message     
----------------

 Important Data

(1 row)

The PostgreSQL log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.

db-master ⇒ Examine the PostgreSQL log output

sudo -u postgres cat /var/log/postgresql/postgresql-9.4-demo.log

LOG:  database system was interrupted; last known up at 2016-07-02 14:09:54 UTC
LOG:  creating missing WAL directory "pg_xlog/archive_status"

LOG:  starting point-in-time recovery to 2016-07-02 14:09:58.282805+00

LOG:  restored log file "00000004.history" from archive
LOG:  restored log file "000000040000000000000014" from archive
       [filtered 2 lines of output]
LOG:  incomplete startup packet
LOG:  restored log file "000000040000000000000015" from archive

LOG:  recovery stopping before commit of transaction 689, time 2016-07-02 14:09:58.443347+00

LOG:  redo done at 0/150157F0

LOG:  last completed transaction was at log time 2016-07-02 14:09:58.144318+00

LOG:  selected new timeline ID: 5
LOG:  restored log file "00000004.history" from archive
       [filtered 4 lines of output]

This example was rigged to give the correct result. If a backup after the required time is chosen then PostgreSQL will not be able to recover the lost table. PostgreSQL can only play forward, not backward. To demonstrate this the important table must be dropped (again).

db-master ⇒ Drop the important table (again)

sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"

ERROR:  relation "important_table" does not exist

LINE 1: ...le important_table;     commit;     select * from important_...
                                                             ^

Now take a new backup and attempt recovery from the new backup.

db-master ⇒ Perform a backup then attempt recovery from that backup

sudo -u postgres pgbackrest --stanza=demo --type=incr backup

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=2016-07-02 14:09:58.282805+00" restore

sudo pg_ctlcluster 9.4 demo start

sudo -u postgres psql -c "select * from important_table"

ERROR:  relation "important_table" does not exist

LINE 1: select * from important_table
                      ^

Looking at the log output it's not obvious that recovery failed to restore the table. The key is to look for the presence of the recovery stopping before... and last completed transaction... log messages. If they are not present then the recovery to the specified point-in-time was not successful.

db-master ⇒ Examine the PostgreSQL log output to discover the recovery was not successful

sudo -u postgres cat /var/log/postgresql/postgresql-9.4-demo.log

LOG:  database system was interrupted; last known up at 2016-07-02 14:10:06 UTC
LOG:  creating missing WAL directory "pg_xlog/archive_status"

LOG:  starting point-in-time recovery to 2016-07-02 14:09:58.282805+00

FATAL:  the database system is starting up
FATAL:  the database system is starting up
LOG:  restored log file "00000005.history" from archive
LOG:  restored log file "000000050000000000000016" from archive
LOG:  redo starts at 0/16000028

LOG:  consistent recovery state reached at 0/160000F0

LOG:  incomplete startup packet
LOG:  redo done at 0/160000F0
       [filtered 9 lines of output]

Using an earlier backup will allow PostgreSQL to play forward to the correct time. The info command can be used to find the next to last backup.

db-master ⇒ Get backup info for the demo cluster

sudo -u postgres pgbackrest info

stanza: demo
    status: ok

    full backup: 20160702-140918F
        start / stop timestamp: 2016-07-02 14:09:13 / 2016-07-02 14:09:18
        database size: 19.3MB, backup size: 19.3MB
        repository size: 2.2MB, repository backup size: 2.2MB

    full backup: 20160702-140923F
        start / stop timestamp: 2016-07-02 14:09:18 / 2016-07-02 14:09:23
        database size: 19.3MB, backup size: 19.3MB
        repository size: 2.2MB, repository backup size: 2.2MB

    diff backup: 20160702-140923F_20160702-140957D

        start / stop timestamp: 2016-07-02 14:09:54 / 2016-07-02 14:09:57
        database size: 25.7MB, backup size: 6.5MB
        repository size: 3MB, repository backup size: 788.8KB
        backup reference list: 20160702-140923F

    incr backup: 20160702-140923F_20160702-141008I
        start / stop timestamp: 2016-07-02 14:10:05 / 2016-07-02 14:10:08
        database size: 25.6MB, backup size: 1.9MB
        repository size: 2.9MB, repository backup size: 214.4KB

        backup reference list: 20160702-140923F, 20160702-140923F_20160702-140957D

The default behavior for restore is to use the last backup but an earlier backup can be specified with the --set option.

db-master ⇒ Stop PostgreSQL, restore from the selected backup, and start PostgreSQL

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=2016-07-02 14:09:58.282805+00" \
       --set=20160702-140923F_20160702-140957D restore

sudo pg_ctlcluster 9.4 demo start

sudo -u postgres psql -c "select * from important_table"

    message     
----------------

 Important Data

(1 row)

Now the the log output will contain the expected recovery stopping before... and last completed transaction... messages showing that the recovery was successful.

db-master ⇒ Examine the PostgreSQL log output for log messages indicating success

sudo -u postgres cat /var/log/postgresql/postgresql-9.4-demo.log

LOG:  database system was interrupted; last known up at 2016-07-02 14:09:54 UTC
LOG:  creating missing WAL directory "pg_xlog/archive_status"

LOG:  starting point-in-time recovery to 2016-07-02 14:09:58.282805+00

LOG:  restored log file "00000004.history" from archive
LOG:  restored log file "000000040000000000000014" from archive
       [filtered 2 lines of output]
LOG:  incomplete startup packet
LOG:  restored log file "000000040000000000000015" from archive

LOG:  recovery stopping before commit of transaction 689, time 2016-07-02 14:09:58.443347+00

LOG:  redo done at 0/150157F0

LOG:  last completed transaction was at log time 2016-07-02 14:09:58.144318+00

LOG:  restored log file "00000005.history" from archive
LOG:  restored log file "00000006.history" from archive
       [filtered 8 lines of output]

Dedicated Backup Host

The configuration described in Quickstart is suitable for simple installations but for enterprise configurations it is more typical to have a dedicated backup host. This separates the backups and WAL archive from the database server so database host failures have less impact. It is still a good idea to employ traditional backup software to backup the backup host.

Installation and Configuration

For this example a new host named backup has been created to store the cluster backups. Follow the instructions in Installation to install pgBackRest and Create the Repository to create the pgBackRest repository. The backup host must also be configured with the database host/user and database path.

backup:/etc/pgbackrest.conf ⇒ Configure db-host/db-user and db-path

[demo]
db-host=db-master
db-path=/var/lib/postgresql/9.4/demo

[global]
repo-path=/var/lib/pgbackrest
start-fast=y

The database host must be configured with the backup host/user.

db-master:/etc/pgbackrest.conf ⇒ Configure backup-host/backup-user

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
backup-host=backup
repo-path=/var/lib/pgbackrest

The repository directory will also be removed from the database host. It will not be used anymore so leaving it around may be confusing later on.

db-master ⇒ Remove repository now that it will be located on the database server

sudo find /var/lib/pgbackrest -delete

Commands are run the same as on a single host configuration except that the backup and expire command are run from the backup host and all other commands are run from the database host.

Check that the configuration is correct on both the database and backup hosts. More information about the check command can be found in Check the Configuration.

db-master ⇒ Check the configuration

sudo -u postgres pgbackrest --stanza=demo check

backup ⇒ Check the configuration

sudo -u backrest pgbackrest --stanza=demo check

Perform a Backup

To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command on the backup host.

backup ⇒ Backup the demo cluster

sudo -u backrest pgbackrest --stanza=demo backup

  WARN: no prior backup exists, incr backup has been changed to full

Since a new repository was created on the backup host the warning about the incremental backup changing to a full backup was emitted.

Restore a Backup

To perform a restore of the PostgreSQL cluster run pgBackRest with the restore command on the database host.

db-master ⇒ Stop the demo cluster, restore, and restart PostgreSQL

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta restore

sudo pg_ctlcluster 9.4 demo start

A new backup must be performed due to the timeline switch.

backup ⇒ Backup the demo cluster

sudo -u backrest pgbackrest --stanza=demo backup

Asynchronous Archiving

The archive-async option offloads WAL archiving to a separate process to improve throughput. WAL segments are temporarily stored in a local queue on the database server, specified by the spool-path option, before being transferred to the repository on the backup server.

The spool directory is created to hold the WAL segments while they are waiting to be (optionally) compressed and transferred.

db-master ⇒ Create the spool directory

sudo mkdir -m 750 /var/spool/pgbackrest

sudo chown postgres:postgres /var/spool/pgbackrest

The spool path must be configured and asynchronous archiving enabled.

db-master:/etc/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving

[demo]
db-path=/var/lib/postgresql/9.4/demo

[global]
archive-async=y
backup-host=backup
repo-path=/var/lib/pgbackrest
spool-path=/var/spool/pgbackrest

The check command ensures that asynchronous archiving is working.

db-master ⇒ Check asynchronous archiving

sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check

  INFO: check start: --backup-host=backup --db-path=/var/lib/postgresql/9.4/demo --log-level-console=info --repo-path=/var/lib/pgbackrest --stanza=demo
  INFO: switch xlog 00000008000000000000001B

  INFO: WAL segment 00000008000000000000001B successfully stored in the archive at '/var/lib/pgbackrest/archive/demo/9.4-1/0000000800000000/00000008000000000000001B-614847b479ea313bedefded949a02cb4fea4b473.gz'

  INFO: check stop

Starting and Stopping

Sometimes it is useful to prevent pgBackRest from running on a system. For example, when failing over from a master to a standby it's best to prevent pgBackRest from running on the old master in case PostgreSQL gets restarted or can't be completely killed. This will also prevent pgBackRest from running on cron.

db-master ⇒ Stop the pgBackRest services

sudo -u postgres pgbackrest stop

New pgBackRest processes will no longer run.

db-master ⇒ Attempt a backup

sudo -u postgres pgbackrest --stanza=demo backup

 ERROR: [137]: stop file exists for all stanzas

Specify the --force option to terminate any pgBackRest process that are currently running. If pgBackRest is already stopped then stopping again will generate a warning.

db-master ⇒ Stop the pgBackRest services again

sudo -u postgres pgbackrest stop

  WARN: stop file already exists for all stanzas

Start pgBackRest processes again with the start command.

db-master ⇒ Start the pgBackRest services

sudo -u postgres pgbackrest start

It is also possible to stop pgBackRest for a single stanza.

db-master ⇒ Stop pgBackRest services for the demo stanza

sudo -u postgres pgbackrest --stanza=demo stop

sudo -u postgres pgbackrest --stanza=demo backup

 ERROR: [137]: stop file exists for stanza demo

The stanza must also be specified when starting the pgBackRest processes for a single stanza.

db-master ⇒ Start the pgBackRest services for the demo stanza

sudo -u postgres pgbackrest --stanza=demo start

Replication

Replication allows multiple copies of a PostgreSQL cluster (called standbys) to be created from a single master. The standbys are useful for balancing reads and to provide redundancy in case the master host fails.

Hot Standby

A hot standby performs replication using the WAL archive and allows read-only queries.

A new host named db-standby will be created to run the standby. Follow the instructions in Installation to install pgBackRest, Setup Demo Cluster to setup the demo cluster, and Create the Repository to create the pgBackRest repository on the db-standby host.

pgBackRest configuration is very similar to db-master except that the standby_mode setting will be enabled to keep the cluster in recovery mode when the end of the WAL stream has been reached.

db-standby:/etc/pgbackrest.conf ⇒ Configure pgBackRest on the standby

[demo]
db-path=/var/lib/postgresql/9.4/demo
recovery-option=standby_mode=on

[global]
backup-host=backup
repo-path=/var/lib/pgbackrest

Now the standby can be created with the restore command.

db-standby ⇒ Restore the demo standby cluster

sudo -u postgres pgbackrest --stanza=demo --delta restore

sudo -u postgres cat /var/lib/postgresql/9.4/demo/recovery.conf

standby_mode = 'on'
restore_command = '/usr/bin/pgbackrest --stanza=demo archive-get %f "%p"'

Note that the standby_mode setting has been written into the recovery.conf file. Configuring recovery settings in pgBackRest means that the recovery.conf file does not need to be stored elsewhere since it will be properly recreated with each restore. The --type=preserve option can be used with the restore to leave the existing recovery.conf file in place if that behavior is preferred.

The hot_standby setting must be enabled before starting PostgreSQL to allow read-only connections on db-standby. Otherwise, connection attempts will be refused.

db-standby:/etc/postgresql/9.4/demo/postgresql.conf ⇒ Enable hot_standby

hot_standby = on
log_filename = 'postgresql.log'
log_line_prefix = ''

db-standby ⇒ Start PostgreSQL

sudo pg_ctlcluster 9.4 demo start

The PostgreSQL log gives valuable information about the recovery. Note especially that the cluster has entered standby mode and is ready to accept read-only connections.

db-standby ⇒ Examine the PostgreSQL log output for log messages indicating success

sudo -u postgres cat /var/log/postgresql/postgresql-9.4-demo.log

LOG:  database system was interrupted; last known up at 2016-07-02 14:10:42 UTC
LOG:  creating missing WAL directory "pg_xlog/archive_status"

LOG:  entering standby mode

LOG:  restored log file "00000008.history" from archive
LOG:  incomplete startup packet
LOG:  restored log file "00000008000000000000001A" from archive
LOG:  redo starts at 0/1A000028
LOG:  consistent recovery state reached at 0/1A0000F0

LOG:  database system is ready to accept read only connections

LOG:  restored log file "00000008000000000000001B" from archive

An easy way to test that replication is properly configured is to create a table on db-master.

db-master ⇒ Create a new table on the master

sudo -u postgres psql -c " \
       begin; \
       create table replicated_table (message text); \
       insert into replicated_table values ('Important Data'); \
       commit; \
       select * from replicated_table";

    message     
----------------

 Important Data

(1 row)

And then query the same table on db-standby.

db-standby ⇒ Query new table on the standby

sudo -u postgres psql -c "select * from replicated_table;"

ERROR:  relation "replicated_table" does not exist

LINE 1: select * from replicated_table;
                      ^

So, what went wrong? Since PostgreSQL is pulling WAL segments from the archive to perform replication, changes won't be seen on the standby until the WAL segment that contains those changes is pushed from db-master.

This can be done manually by calling pg_switch_xlog() which pushes the current WAL segment to the archive (a new WAL segment is created to contain further changes).

db-master ⇒ Call pg_switch_xlog()

sudo -u postgres psql -c "select *, current_timestamp from pg_switch_xlog()";

 pg_switch_xlog |              now              
----------------+-------------------------------
 0/1C017918     | 2016-07-02 14:11:03.220459+00
(1 row)

Now after a short delay the table will appear on db-standby.

db-standby ⇒ Now the new table exists on the standby (may require a few retries)

sudo -u postgres psql -c " \
       select *, current_timestamp from replicated_table"

    message     |              now              
----------------+-------------------------------

 Important Data | 2016-07-02 14:11:06.363685+00

(1 row)

Streaming Replication

Instead of relying solely on the WAL archive, streaming replication makes a direct connection to the master and applies changes as soon as they are made on the master. This results in much less lag between the master and standby.

Streaming replication requires a user with the replication privilege.

db-master ⇒ Create replication user

sudo -u postgres psql -c " \
       create user replicator password 'jw8s0F4' replication";

CREATE ROLE

The pg_hba.conf file must be updated to allow the standby to connect as the replication user. Be sure to replace the IP address below with the actual IP address of your db-master. A reload will be required after modifying the pg_hba.conf file.

db-master ⇒ Create pg_hba.conf entry for replication user

sudo -u postgres sh -c 'echo \
       "host    replication     replicator      172.17.0.4/32           md5" \
       >> /etc/postgresql/9.4/demo/pg_hba.conf'

sudo pg_ctlcluster 9.4 demo reload

The standby needs to know how to contact the master so the primary_conninfo setting will be configured in pgBackRest.

db-standby:/etc/pgbackrest.conf ⇒ Set primary_conninfo

[demo]
db-path=/var/lib/postgresql/9.4/demo
recovery-option=standby_mode=on
recovery-option=primary_conninfo=host=172.17.0.2 port=5432 user=replicator

[global]
backup-host=backup
repo-path=/var/lib/pgbackrest

It is possible to configure a password in the primary_conninfo setting but using a .pgpass file is more flexible and secure.

db-standby ⇒ Configure the replication password in the .pgpass file.

sudo -u postgres sh -c 'echo \
       "172.17.0.2:*:replication:replicator:jw8s0F4" \
       >> /home/postgres/.pgpass'

sudo -u postgres chmod 600 /home/postgres/.pgpass

Now the standby can be created with the restore command.

db-standby ⇒ Stop PostgreSQL and restore the demo standby cluster

sudo pg_ctlcluster 9.4 demo stop

sudo -u postgres pgbackrest --stanza=demo --delta restore

sudo -u postgres cat /var/lib/postgresql/9.4/demo/recovery.conf

primary_conninfo = 'host=172.17.0.2 port=5432 user=replicator'
standby_mode = 'on'
restore_command = '/usr/bin/pgbackrest --stanza=demo archive-get %f "%p"'

db-standby ⇒ Start PostgreSQL

sudo pg_ctlcluster 9.4 demo start

The PostgreSQL log will confirm that streaming replication has started.

db-standby ⇒ Examine the PostgreSQL log output for log messages indicating success

sudo -u postgres cat /var/log/postgresql/postgresql-9.4-demo.log

       [filtered 9 lines of output]
LOG:  restored log file "00000008000000000000001B" from archive
LOG:  restored log file "00000008000000000000001C" from archive

LOG:  started streaming WAL from primary at 0/1D000000 on timeline 8

Now when a table is created on db-master it will appear on db-standby quickly and without the need to call pg_switch_xlog().

db-master ⇒ Create a new table on the master

sudo -u postgres psql -c " \
       begin; \
       create table stream_table (message text); \
       insert into stream_table values ('Important Data'); \
       commit; \
       select *, current_timestamp from stream_table";

    message     |              now              
----------------+-------------------------------

 Important Data | 2016-07-02 14:11:13.523097+00

(1 row)

db-standby ⇒ Query table on the standby

sudo -u postgres psql -c " \
       select *, current_timestamp from stream_table"

    message     |              now              
----------------+-------------------------------

 Important Data | 2016-07-02 14:11:13.769359+00

(1 row)