Backup with VMware Velero

      How-to backup your Operator and Couchbase clusters against deletion.

      Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

      Velero is a full system backup and restore tool. Unlike a Couchbase backup — that is only concerned with data and requires duplication — Velero allows an almost instantaneous backup of your data and Kubernetes resources. It achieves this by combining volume snapshots with resource archival to an external bucket.

      This tutorial will step through a procedure to backup a Couchbase cluster, simulate a disaster and how to successfully recover.

      Use of Velero backup will only work with Operator version 2.0.0 or greater.

      Installing Velero

      Velero is a stand-alone binary that can be downloaded from their website. This tutorial was tested using Velero version 1.2.0, using the Google Kubernetes Engine (GKE).

      It is important to correctly configure Velero so that it has the ability to perform volume snapshots. A list of providers are provided in the Velero support matrix.

      Installing Couchbase Autonomous Operator

      For this tutorial we will install the dynamic admission controller (DAC) in the default namespace:

      $ cao create admission -n default
      serviceaccount/couchbase-operator-admission created created created
      secret/couchbase-operator-admission created
      deployment.apps/couchbase-operator-admission created
      service/couchbase-operator-admission created created created

      And the Operator in a separate, per-cluster namespace (test) as per our recommended best-practices:

      $ kubectl create namespace test
      namespace/test created
      $ cao create operator -n test
      serviceaccount/couchbase-operator created created created
      deployment.apps/couchbase-operator created
      service/couchbase-operator created

      Installing Couchbase Server

      Next up, create a Couchbase Server cluster:

      $ kubectl create -n test -f secret.yaml
      secret/cb-example-auth created
      $ kubectl create -n test -f couchbase-cluster.yaml created created

      The Operator will begin to provision the cluster. In the mean time let’s have a look at the configuration:

      kind: CouchbaseBucket
        name: default
      kind: CouchbaseCluster
        name: cb-example
        image: couchbase/server:7.0.3
          adminSecret: cb-example-auth
          managed: true (1)
        - size: 3 (2)
          name: all_services
          - data
          - index
          - query
          volumeMounts: (3)
            default: couchbase
        - metadata:
             name: couchbase
            storageClassName: dynamic-bound
                storage: 1Gi

      The configuration we have selected is fairly minimal:

      1 Buckets are managed, and we have created a default one for use.
      2 The cluster has 3 nodes, totally arbitrary, but is a recommended starting point.
      3 Persistent volume mounts are in use, therefore data is persisted on a volume. Velero is able to take snapshots of the backing storage.

      Once the cluster has provisioned you can populate it with some data to prove that backups have worked. List the pod names and forward the console port:

      $ kubectl -n test get pods
      NAME                                  READY   STATUS    RESTARTS   AGE
      cb-example-0000                       1/1     Running   0          2m8s
      cb-example-0001                       1/1     Running   0          98s
      cb-example-0002                       1/1     Running   0          65s
      $ kubectl -n test port-forward cb-example-0000 8091

      From here connect to http://localhost:8091 and log into the console. Try manually adding some documents to the default bucket. From the dashboard home page you can see disk IO as these documents are flushed to disk and persisted.

      Creating a Velero Backup

      The following is a typical backup command:

      $ velero backup create \
        test \ (1)
        --include-namespaces test \ (2)
        --include-resources,,persistentvolumes,persistentvolumeclaims,secrets,deployments,,,serviceaccounts,configmaps (3)
      1 test is the name of the backup we wish to create. This is globally scoped so needs to be unique.
      2 We specify that this backup should only look at the test namespace. This is another good reason for deploying Couchbase clusters in their own namespaces — you can limit the scope of and control third party applications.
      3 We specify exactly what resources to backup. We want to backup all Couchbase resources — for brevity I’ve used only what is in use. Backing up persistent volumes will create snapshots. Backing up persistent volume claims and ConfigMap resources will save all the persistent metadata required by the Operator for recovery. Backing up secrets will protect the admin username and password. Finally deployments, roles, role bindings and service accounts will allow the Operator to restart.

      You may have noticed that neither pods nor services were backed up. This is because the Operator will be able to recreate them from the cluster ConfigMap, metadata attached to the persistent volume claims, and the CouchbaseCluster resource itself. Likewise the deployment will be able to recreate the Operator pod.

      Once the backup is complete it will look like the following:

      $ velero backup describe test
      Name:         test
      Namespace:    velero
      Annotations:  <none>
      Phase:  Completed (1)
        Included:  test
        Excluded:  <none>
        Included:,, persistentvolumes, persistentvolumeclaims, secrets, deployments,,, serviceaccounts, configmaps
        Excluded:        <none>
        Cluster-scoped:  auto
      Label selector:  <none>
      Storage Location:  default
      Snapshot PVs:  auto
      TTL:  720h0m0s
      Hooks:  <none>
      Backup Format Version:  1
      Started:    2020-02-17 13:46:36 +0000 GMT
      Completed:  2020-02-17 13:46:40 +0000 GMT
      Expiration:  2020-03-18 13:46:36 +0000 GMT
      Persistent Volumes:  3 of 3 snapshots completed successfully (specify --details for more information) (2)
      1 The backup has completed successfully
      2 The backup has registered and has take a snapshot of our persistent volumes.

      Disaster Strikes!

      The Operator can easily handle pods and persistent volumes being deleted. What it can’t cater for is someone deleting everything, which is where Velero comes in. So let’s simulate total devastation to see Velero in action:

      $ kubectl delete namespace test
      namespace "test" deleted

      You can double check that the namespace and everything in it are gone before proceeding.

      Disaster Recovery

      This is where all the hard work correctly configuring the backup — and testing it — pays dividends. In a real-world disaster recovery situation you are probably panicking, and thinking irrationally. Velero makes restoration very easy, and therefore virtually fool-proof while under pressure:

      $ velero restore create --from-backup test
      Restore request "test-20200217134839" submitted successfully.
      Run `velero restore describe test-20200217134839` or `velero restore logs test-20200217134839` for more details.

      After a few short moments you will see the namespace and resources being recreated. Eventually the Operator will restart as its deployment recognizes it has not pods running:

      $ kubectl -n test logs -f deployment/couchbase-operator
      {"level":"info","ts":1581948125.9243767,"logger":"main","msg":"couchbase-operator","version":"2.0.0","revision":"master 948b7d6d5d6a8870c36209e982466f85cb759016"}
      {"level":"info","ts":1581948125.9268959,"logger":"leader","msg":"Trying to become the leader."}
      {"level":"info","ts":1581948126.068716,"logger":"leader","msg":"Not the leader. Waiting."}
      {"level":"info","ts":1581948127.1986287,"logger":"leader","msg":"Not the leader. Waiting."}

      However, as you will see, the Operator’s leader election hangs. This is because we backed the ConfigMap that is used to determine leadership. But this is actually a good thing.

      It is worthwhile taking a moment and checking that things are as you expect before continuing. The most important resources to double check are the persistent volume claims as they contain your data and recovery metadata. They should be present and Bound, ready to be used.

      $ kubectl -n test get all
      NAME                                      READY   STATUS    RESTARTS   AGE
      pod/couchbase-operator-7b6588c7c6-pkdm4   1/1     Running   0          54s
      NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.apps/couchbase-operator   1/1     1            1           55s
      NAME                                            DESIRED   CURRENT   READY   AGE
      replicaset.apps/couchbase-operator-7b6588c7c6   1         1         1       55s
      NAME                                    MEMORY QUOTA   REPLICAS   IO PRIORITY   EVICTION POLICY   CONFLICT RESOLUTION   AGE   100Mi          1          low           valueOnly         seqno                 56s
      NAME                                        VERSION   SIZE   STATUS   UUID   AGE
      $ kubectl -n test get pvc
      NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
      cb-example-0000-default-01   Bound    pvc-8a84b9ff-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s
      cb-example-0001-default-01   Bound    pvc-9ea97330-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s
      cb-example-0002-default-01   Bound    pvc-adb55be6-518d-11ea-8b64-42010a8e021e   1Gi        RWO            dynamic-bound   68s

      Once satisfied things look correct, allow the Operator to start up:

      $ kubectl -n test delete configmap/couchbase-operator
      configmap "couchbase-operator" deleted

      After a few moments the Operator will recreate a Couchbase pod, then restore the rest of the cluster. You can now reconnect to the Couchbase web console and check that the documents you inserted earlier are fully restored.

      Further Reading