AGILITY backup and restore

AGILITY uses Databases to persist data. This guide explains how to configure backup and how to restore based on agility-services database solution.

agility-services uses PGO to manage Database instances. PGO uses pgBackRest for backup management. Detailed information about Database backup options can be found on:

• PGO backup configuration reference

• pgBackRest configuration reference

S3 Backup configuration for AGILITY

AGILITY provides local backup per default and it is suggested not to change this setting. However, it is possible to optionally enable remote S3 backup.

1. Create a kubernetes secret named agility-db-backup with the remote S3 credentials.

   • Declare the S3 credentials as environment variables and the namespace where AGILITY is installed

   export NS=agility
   export S3_BACKUP_CREDENTIALS_FILE=/tmp/s3-backup.ini
   export S3_ACCESS_KEY=<Access key>
   export S3_ACCESS_KEY_SECRET=<Secret access key>

   • Store in a temporary file the S3 credentials with the following format

   cat <<EOF > ${S3_BACKUP_CREDENTIALS_FILE}
   [global]
   repo2-s3-key=${S3_ACCESS_KEY}
   repo2-s3-key-secret=${S3_ACCESS_KEY_SECRET}
   EOF

   • Create the kubernetes secret

   kubectl --namespace ${NS} create secret generic agility-db-backup --from-file=s3.conf=${S3_BACKUP_CREDENTIALS_FILE}

2. Define backup override values file

   • Define the required information

   export S3_BUCKET_NAME=<bucket_name>
   export S3_ENDPOINT=<endpoint>
   export S3_REGION=<region>
   export S3_URI_STYLE=<host|path>

   As per pgBackRest documentation, S3_URI_STYLE should be either host or path. Usually path is the preferred option:

   S3 URI Style. The following URI styles are supported: host - Connect to bucket.endpoint host. path - Connect to endpoint host and prepend bucket to URIs.

   • Declare the file

   cd agility-charts
   cat <<EOF> agility-services-values-backup.yaml
   postgrescluster:
     pgbackrest:
       backup:
         bucket_s3:
           enabled: true
           bucket_name: ${S3_BUCKET_NAME}
           endpoint: ${S3_ENDPOINT}
           region: ${S3_REGION}
           path: /backups/pgbackrest/${NS}/agility-db/repo2
           uri_style: ${S3_URI_STYLE}
       otherSpecs:
         manual:
           repoName: repo2
           options:
           - --type=full
   EOF

3. Run the Helm command to deploy agility-services:

   helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml

4. Once deployed, check backup configuration is healthy

   export DB_POD=$(kubectl --namespace ${NS} get pod -l "postgres-operator.crunchydata.com/role=master" --no-headers -o custom-columns=":metadata.name")
   kubectl --namespace ${NS} exec ${DB_POD} -c pgbackrest -- pgbackrest info --output=json | jq '.[].repo[] | select(.key == 2)'

   Output should be the following:

   {
     "cipher": "none",
     "key": 2,
     "status": {
       "code": 2,
       "message": "no valid backups"
     }
   }

5. Enforce first full backup execution on the S3 bucket

   kubectl --namespace ${NS} annotate postgrescluster agility-db postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"

   A new pod will be created, wait until it gets Completed status

   kubectl --namespace ${NS} get pod -l postgres-operator.crunchydata.com/pgbackrest-repo=repo2 -w

S3 Backup verification

Once S3 backup is configured, it is possible to verify the current status:

DB_POD=$(kubectl --namespace ${NS} get pod -l "postgres-operator.crunchydata.com/role=master" --no-headers -o custom-columns=":metadata.name")
kubectl --namespace ${NS} exec ${DB_POD} -c pgbackrest -- pgbackrest info --output=json | jq '.[].repo[] | select(.key == 2)'

Expected output should be similar to the following message:

{
  "cipher": "none",
  "key": 2,
  "status": {
    "code": 0,
    "message": "ok"
  }
}

Restore AGILITY

agility-services database can be restored in place.

1. Define the time agility-services database will be restored

   TIME=<2021-06-09 14:15:11-04>
   

   From PGO documentation:

   "To perform a Pont In Time Recovery (PITR), you must have a backup that finished before your PITR time. In other words, you can’t perform a PITR back to a time where you do not have a backup!"

2. Update the same values file it was created to backup configuration.

   • Open with your preferred edit the values file

   cd agility-charts
   vi agility-services-values-backup.yaml

   • Add the restore section under postgrescluster.pgbackrest.otherSpec:

   postgrescluster:
     pgbackrest:
       otherSpecs:
         manual:
           repoName: repo2
           options:
           - --type=full
         restore:
           enabled: true
           repoName: repo2
           options:
           - --type=time
           - --target=<TIME>

   Note --target option must be changed with your defined date

3. Run the Helm command to deploy agility-services:

   helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml

4. Annotate agility-db postgrescluster to start restore process

   kubectl --namespace ${NS} annotate postgrescluster agility-db --overwrite postgres-operator.crunchydata.com/pgbackrest-restore=id1

5. Wait until restoration job finish and gets terminated

   kubectl --namespace ${NS} get pod -l job-name=agility-db-pgbackrest-restore -w

   Output should be similar to the following code block

   NAME                                  READY   STATUS      RESTARTS   AGE
   agility-db-pgbackrest-restore-htsqw   0/1     Completed   0          31s
   agility-db-pgbackrest-restore-htsqw   0/1     Terminating   0          35s
   agility-db-pgbackrest-restore-htsqw   0/1     Terminating   0          35s

   Once restoration jobs is Terminated, agility-db pod will be created and restored with data from the selected date

6. Recreate affected services

   kubectl --namespace ${NS} scale sts/agility-keycloak deploy/agility-backend deploy/agility-autoloader-server --replicas 0 && kubectl --namespace ${NS} scale sts/agility-keycloak deploy/agility-backend deploy/agility-autoloader-server --replicas 1

7. Once AGILITY is available, disable restoration to avoid undesired restoration

   • Set postgrescluster.pgbackrest.otherSpecs.restore.enabled: false on agility-services-values-backup.yaml file

   postgrescluster:
     pgbackrest:
       otherSpecs:
         manual:
           repoName: repo2
           options:
           - --type=full
         restore:
           enabled: false
           repoName: repo2
           options:
           - --type=time
           - --target=<TIME>

   • Run the Helm command to deploy agility-services:

   helm --namespace ${NS} upgrade --install --create-namespace agility-services ./agility-services --values agility-services-values-backup.yaml