AGILITY backup and restore

AGILITY uses Databases to persist data. This guide explains how to configure backup and how to restore based on agility-services database solution.

agility-services uses PGO to manage Database instances. PGO uses pgBackRest for backup management. Detailed information about Database backup options can be found on:

S3 Backup configuration for AGILITY

AGILITY provides local backup per default and it is suggested not to change this setting. However, it is possible to optionally enable remote S3 backup.

Create a kubernetes secret named agility-db-backup with the remote S3 credentials.

Declare the S3 credentials as environment variables and the namespace where AGILITY is installed

export NS=agility
export S3_BACKUP_CREDENTIALS_FILE=/tmp/s3-backup.ini
export S3_ACCESS_KEY=<Access key>
export S3_ACCESS_KEY_SECRET=<Secret access key>

Store in a temporary file the S3 credentials with the following format

cat <<EOF > ${S3_BACKUP_CREDENTIALS_FILE}
[global]
repo2-s3-key=${S3_ACCESS_KEY}
repo2-s3-key-secret=${S3_ACCESS_KEY_SECRET}
EOF

Create the kubernetes secret

kubectl --namespace ${NS} create secret generic agility-db-backup --from-file=s3.conf=${S3_BACKUP_CREDENTIALS_FILE}

Define backup override values file

Define the required information

export S3_BUCKET_NAME=<bucket_name>
export S3_ENDPOINT=<endpoint>
export S3_REGION=<region>
export S3_URI_STYLE=<host|path>

As per pgBackRest documentation, S3_URI_STYLE should be either host or path. Usually path is the preferred option:

S3 URI Style. The following URI styles are supported: host - Connect to bucket.endpoint host. path - Connect to endpoint host and prepend bucket to URIs.

Declare the file

cd agility-charts
cat <<EOF> agility-services-values-backup.yaml
postgrescluster:
  pgbackrest:
    backup:
      bucket_s3:
        enabled: true
        bucket_name: ${S3_BUCKET_NAME}
        endpoint: ${S3_ENDPOINT}
        region: ${S3_REGION}
        path: /backups/pgbackrest/${NS}/agility-db/repo2
        uri_style: ${S3_URI_STYLE}
    otherSpecs:
      manual:
        repoName: repo2
        options:
        - --type=full
EOF

Run the Helm command to deploy agility-services:

helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml

Once deployed, check backup configuration is healthy

export DB_POD=$(kubectl --namespace ${NS} get pod -l "postgres-operator.crunchydata.com/role=master" --no-headers -o custom-columns=":metadata.name")
kubectl --namespace ${NS} exec ${DB_POD} -c pgbackrest -- pgbackrest info --output=json | jq '.[].repo[] | select(.key == 2)'

Output should be the following:

{
  "cipher": "none",
  "key": 2,
  "status": {
    "code": 2,
    "message": "no valid backups"
  }
}

Enforce first full backup execution on the S3 bucket

kubectl --namespace ${NS} annotate postgrescluster agility-db postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"

A new pod will be created, wait until it gets Completed status

kubectl --namespace ${NS} get pod -l postgres-operator.crunchydata.com/pgbackrest-repo=repo2 -w

S3 Backup verification

Once S3 backup is configured, it is possible to verify the current status:

DB_POD=$(kubectl --namespace ${NS} get pod -l "postgres-operator.crunchydata.com/role=master" --no-headers -o custom-columns=":metadata.name")
kubectl --namespace ${NS} exec ${DB_POD} -c pgbackrest -- pgbackrest info --output=json | jq '.[].repo[] | select(.key == 2)'

Expected output should be similar to the following message:

{
  "cipher": "none",
  "key": 2,
  "status": {
    "code": 0,
    "message": "ok"
  }
}

Restore AGILITY

agility-services database can be restored in place.

Define the time agility-services database will be restored
```
TIME=<2021-06-09 14:15:11-04>
```
From PGO documentation:
"To perform a Pont In Time Recovery (PITR), you must have a backup that finished before your PITR time. In other words, you can’t perform a PITR back to a time where you do not have a backup!"

Update the same values file it was created to backup configuration.

Open with your preferred edit the values file

cd agility-charts
vi agility-services-values-backup.yaml

Add the restore section under postgrescluster.pgbackrest.otherSpec:

postgrescluster:
  pgbackrest:
    otherSpecs:
      manual:
        repoName: repo2
        options:
        - --type=full
      restore:
        enabled: true
        repoName: repo2
        options:
        - --type=time
        - --target=<TIME>

Note --target option must be changed with your defined date

Run the Helm command to deploy agility-services:

helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml

Annotate agility-db postgrescluster to start restore process

kubectl --namespace ${NS} annotate postgrescluster agility-db --overwrite postgres-operator.crunchydata.com/pgbackrest-restore=id1

Wait until restoration job finish and gets terminated

kubectl --namespace ${NS} get pod -l job-name=agility-db-pgbackrest-restore -w

Output should be similar to the following code block

NAME                                  READY   STATUS      RESTARTS   AGE
agility-db-pgbackrest-restore-htsqw   0/1     Completed   0          31s
agility-db-pgbackrest-restore-htsqw   0/1     Terminating   0          35s
agility-db-pgbackrest-restore-htsqw   0/1     Terminating   0          35s

Once restoration jobs is Terminated, agility-db pod will be created and restored with data from the selected date

Recreate affected services

kubectl --namespace ${NS} scale sts/agility-keycloak deploy/agility-backend deploy/agility-autoloader-server --replicas 0 && kubectl --namespace ${NS} scale sts/agility-keycloak deploy/agility-backend deploy/agility-autoloader-server --replicas 1

Once AGILITY is available, disable restoration to avoid undesired restoration

Set postgrescluster.pgbackrest.otherSpecs.restore.enabled: false on agility-services-values-backup.yaml file

postgrescluster:
  pgbackrest:
    otherSpecs:
      manual:
        repoName: repo2
        options:
        - --type=full
      restore:
        enabled: false
        repoName: repo2
        options:
        - --type=time
        - --target=<TIME>

Run the Helm command to deploy agility-services:

helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml

AGILITY backup and restore

S3 Backup configuration for AGILITY

S3 Backup verification

Restore AGILITY

0 Comments