Backup with VMware Velero

How-to backup your Operator and Couchbase clusters against deletion.

Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

Velero is a full system backup and restore tool. Unlike a Couchbase backup — that is only concerned with data and requires duplication — Velero allows an almost instantaneous backup of your data and Kubernetes resources. It achieves this by combining volume snapshots with resource archival to an external bucket.

This tutorial will step through a procedure to backup a Couchbase cluster, simulate a disaster and how to successfully recover.

Use of Velero backup will only work with Operator version 2.0.0 or greater.

Installing Velero

Velero is a stand-alone binary that can be downloaded from their website. This tutorial was tested using Velero version 1.2.0, using the Google Kubernetes Engine (GKE).

It is important to correctly configure Velero so that it has the ability to perform volume snapshots. A list of providers are provided in the Velero support matrix.

Installing Couchbase Autonomous Operator

For this tutorial we will install the dynamic admission controller (DAC) in the default namespace:

$ cao create admission -n default
serviceaccount/couchbase-operator-admission created
clusterrole.rbac.authorization.k8s.io/couchbase-operator-admission created
clusterrolebinding.rbac.authorization.k8s.io/couchbase-operator-admission created
secret/couchbase-operator-admission created
deployment.apps/couchbase-operator-admission created
service/couchbase-operator-admission created
mutatingwebhookconfiguration.admissionregistration.k8s.io/couchbase-operator-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/couchbase-operator-admission created

And the Operator in a separate, per-cluster namespace (test) as per our recommended best-practices:

$ kubectl create namespace test
namespace/test created

$ cao create operator -n test
serviceaccount/couchbase-operator created
role.rbac.authorization.k8s.io/couchbase-operator created
rolebinding.rbac.authorization.k8s.io/couchbase-operator created
deployment.apps/couchbase-operator created
service/couchbase-operator created

Installing Couchbase Server

Next up, create a Couchbase Server cluster:

$ kubectl create -n test -f secret.yaml
secret/cb-example-auth created

$ kubectl create -n test -f couchbase-cluster.yaml
couchbasebucket.couchbase.com/default created
couchbasecluster.couchbase.com/cb-example created

The Operator will begin to provision the cluster. In the mean time let’s have a look at the configuration:

apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
  name: default
---
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: cb-example
spec:
  image: couchbase/server:7.0.3
  security:
    adminSecret: cb-example-auth
  buckets:
    managed: true (1)
  servers:
  - size: 3 (2)
    name: all_services
    services:
    - data
    - index
    - query
    volumeMounts: (3)
      default: couchbase
  volumeClaimTemplates:
  - metadata:
       name: couchbase
    spec:
      storageClassName: dynamic-bound
      resources:
        requests:
          storage: 1Gi

The configuration we have selected is fairly minimal:

1	Buckets are managed, and we have created a `default` one for use.
2	The cluster has 3 nodes, totally arbitrary, but is a recommended starting point.
3	Persistent volume mounts are in use, therefore data is persisted on a volume. Velero is able to take snapshots of the backing storage.

Once the cluster has provisioned you can populate it with some data to prove that backups have worked. List the pod names and forward the console port:

$ kubectl -n test get pods
NAME                                  READY   STATUS    RESTARTS   AGE
cb-example-0000                       1/1     Running   0          2m8s
cb-example-0001                       1/1     Running   0          98s
cb-example-0002                       1/1     Running   0          65s

$ kubectl -n test port-forward cb-example-0000 8091

From here connect to http://localhost:8091 and log into the console. Try manually adding some documents to the default bucket. From the dashboard home page you can see disk IO as these documents are flushed to disk and persisted.

Creating a Velero Backup

The following is a typical backup command:

$ velero backup create \
  test \ (1)
  --include-namespaces test \ (2)
  --include-resources couchbaseclusters.couchbase.com,couchbasebuckets.couchbase.com,persistentvolumes,persistentvolumeclaims,secrets,deployments,roles.rbac.authorization.k8s.io,rolebindings.rbac.authorization.k8s.io,serviceaccounts,configmaps (3)

1	`test` is the name of the backup we wish to create. This is globally scoped so needs to be unique.
2	We specify that this backup should only look at the `test` namespace. This is another good reason for deploying Couchbase clusters in their own namespaces — you can limit the scope of and control third party applications.
3	We specify exactly what resources to backup. We want to backup all Couchbase resources — for brevity I’ve used only what is in use. Backing up persistent volumes will create snapshots. Backing up persistent volume claims and `ConfigMap` resources will save all the persistent metadata required by the Operator for recovery. Backing up secrets will protect the admin username and password. Finally deployments, roles, role bindings and service accounts will allow the Operator to restart.

You may have noticed that neither pods nor services were backed up. This is because the Operator will be able to recreate them from the cluster ConfigMap, metadata attached to the persistent volume claims, and the CouchbaseCluster resource itself. Likewise the deployment will be able to recreate the Operator pod.

Once the backup is complete it will look like the following:

$ velero backup describe test
Name:         test
Namespace:    velero
Labels:       velero.io/backup=test
              velero.io/pv=pvc-837e29fa-518b-11ea-8b64-42010a8e021e
              velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed (1)

Namespaces:
  Included:  test
  Excluded:  <none>

Resources:
  Included:        couchbaseclusters.couchbase.com, couchbasebuckets.couchbase.com, persistentvolumes, persistentvolumeclaims, secrets, deployments, roles.rbac.authorization.k8s.io, rolebindings.rbac.authorization.k8s.io, serviceaccounts, configmaps
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-02-17 13:46:36 +0000 GMT
Completed:  2020-02-17 13:46:40 +0000 GMT

Expiration:  2020-03-18 13:46:36 +0000 GMT

Persistent Volumes:  3 of 3 snapshots completed successfully (specify --details for more information) (2)

1	The backup has completed successfully
2	The backup has registered and has take a snapshot of our persistent volumes.

Disaster Strikes!

The Operator can easily handle pods and persistent volumes being deleted. What it can’t cater for is someone deleting everything, which is where Velero comes in. So let’s simulate total devastation to see Velero in action:

$ kubectl delete namespace test
namespace "test" deleted

You can double check that the namespace and everything in it are gone before proceeding.

Disaster Recovery

This is where all the hard work correctly configuring the backup — and testing it — pays dividends. In a real-world disaster recovery situation you are probably panicking, and thinking irrationally. Velero makes restoration very easy, and therefore virtually fool-proof while under pressure:

$ velero restore create --from-backup test
Restore request "test-20200217134839" submitted successfully.
Run `velero restore describe test-20200217134839` or `velero restore logs test-20200217134839` for more details.

After a few short moments you will see the namespace and resources being recreated. Eventually the Operator will restart as its deployment recognizes it has not pods running:

$ kubectl -n test logs -f deployment/couchbase-operator
{"level":"info","ts":1581948125.9243767,"logger":"main","msg":"couchbase-operator","version":"2.0.0","revision":"master 948b7d6d5d6a8870c36209e982466f85cb759016"}
{"level":"info","ts":1581948125.9268959,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1581948126.068716,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1581948127.1986287,"logger":"leader","msg":"Not the leader. Waiting."}

However, as you will see, the Operator’s leader election hangs. This is because we backed the ConfigMap that is used to determine leadership. But this is actually a good thing.

It is worthwhile taking a moment and checking that things are as you expect before continuing. The most important resources to double check are the persistent volume claims as they contain your data and recovery metadata. They should be present and Bound, ready to be used.

$ kubectl -n test get all
NAME                                      READY   STATUS    RESTARTS   AGE
pod/couchbase-operator-7b6588c7c6-pkdm4   1/1     Running   0          54s

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/couchbase-operator   1/1     1            1           55s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/couchbase-operator-7b6588c7c6   1         1         1       55s

NAME                                    MEMORY QUOTA   REPLICAS   IO PRIORITY   EVICTION POLICY   CONFLICT RESOLUTION   AGE
couchbasebucket.couchbase.com/default   100Mi          1          low           valueOnly         seqno                 56s

NAME                                        VERSION   SIZE   STATUS   UUID   AGE
couchbasecluster.couchbase.com/cb-example

$ kubectl -n test get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
cb-example-0000-default-01   Bound    pvc-8a84b9ff-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s
cb-example-0001-default-01   Bound    pvc-9ea97330-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s
cb-example-0002-default-01   Bound    pvc-adb55be6-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s

Once satisfied things look correct, allow the Operator to start up:

$ kubectl -n test delete configmap/couchbase-operator
configmap "couchbase-operator" deleted

After a few moments the Operator will recreate a Couchbase pod, then restore the rest of the cluster. You can now reconnect to the Couchbase web console and check that the documents you inserted earlier are fully restored.