Configure Automated Backup and Restore

Backup Pods need read-only access to Kubernetes resources such as Pods, CronJobs, and Jobs. They also need write access to Events and the CouchbaseBackup/CouchbaseBackupRestore custom resources. Without these resources, backup jobs will still run as scheduled, but they will ultimately fail as the pods won’t have the required permissions.

You can use the cao tool to create the resources that grant the required permissions. The following command creates the necessary resources in the default namespace:

To create the resources in a custom namespace, use the -n flag:

To make your own edits to these resources, you can use cao generate backup to generate the YAML output instead of creating the resources in Kubernetes immediately.

Backups allow data to be filtered so that you only backup what you need, minimizing storage space and improving performance. Backup options can only be modified on creation of a new backup repository, so when using a full/incremental backup strategy, modifications will be deferred until the next full backup.

Consider the following specification:

Further details can be found on the CouchbaseBackup resource reference.

Additional options from cbbackupmgr restore may also be specified.

Only the preexisting schedules and volume size of a CouchbaseBackup resource can be edited. Attempts to edit things like the name or strategy will fail.

A Backup PVC that is referenced by an existing CouchbaseBackup resource can be resized manually by the user, or automatically by the Autonomous Operator.

When a CouchbaseBackup resource is deleted, any associated Cronjob(s) are deleted. Jobs and their respective Pods from those Cronjobs are orphaned; the number of these resources that are left over is determined by the limits spec.successfulJobsHistoryLimit and spec.failedJobsHistoryLimit.

If a backup job is running whilst the parent CouchbaseBackup is deleted then the job will continue until completion or eventual failure.

If anything goes wrong during a backup job, and backup pods return the Error status, detailed logging is stored on the Persistent Volume Claim for the backup. You can access these logs by creating a Kubernetes job that creates a pod that mounts this PVC and then running kubectl exec to shell into this pod. From there you can access the logs and backup data directly.

The following is an example file that creates such a Kubernetes job. The job creates a pod and mounts the PVC on the path /data as the backup and restore pods themselves would.

Backups are available to view at /data/backups and their respective logs at /data/scriptlogs. Inside /data/scriptlogs will be three folders, full_only, incremental, and restore. The first two folders correspond to any logs run under the relevant CouchbaseBackup strategy and the last folder is for CouchbaseBackupRestore operations exclusively.

As backups are performed on separate pods you will need to consider careful node scheduling when it comes to these pods in order to avoid performance issues and noisy neighbor problems. The following YAML example builds upon the initial YAML in Enable Automated Backup.

Further reference on all of these fields can be found in the couchbaseclusters.spec.backup resource configuration. For more overall information please see Couchbase Scheduling and Isolation.

When deciding on the Cron schedules for the Full/Incremental strategy, you should take care that the schedules are not defined in a way for a potential clash between Full and Incremental backups. For the example given in this documentation and the cbbackupmgr documentation, this is obviously very unlikely but in a scenario where a backup is not given enough of a time window to complete, this could cause problems. This particularly common in situations where backups have been scheduled too frequently.

Scheduling of backup and restore jobs is exactly the same as the mechanism used for Couchbase Server pods. The affinity and anti-affinity mechanisms are described in Couchbase Scheduling and Isolation.

Backup and restore job affinity can be set, per CouchbaseCluster, with the couchbaseclusters.spec.backup.nodeSelector attribute, and toleration of anti-affinity rules can be set with the couchbaseclusters.spec.backup.tolerations attribute.

When manually providing an access key and secret id, a separate Secret must be created that holds the AWS region name, access key ID, and secret access key.

In your CouchbaseCluster object, you will need to reference this secret so the Operator knows where to extract the credentials from.

To allow backup to automatically use an attached IAM Role, enable the useIAMRole parameter. A secret with the region key set is still required. If you have attached the IAM Role to an EKS node directly then this is sufficient configuration.

If you have setup IAM roles for service accounts, the role ARN annotation must be applied to the backup service account either manually or when running cao using bin/cao create backup --iam-role-arn arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>

The S3 bucket that we wish to hold backups needs to be specified in the desired CouchbaseBackup and CouchbaseBackupRestore CRDs. Note that the s3:// prefix is required, otherwise the Admission Controller will not allow the creation of the CRD.

Please note that operations involving S3 take more time to complete in comparison to regular backup to PVCs so please bear this in mind when configuring your automated backup schedules.