Online Persistent Volume Expansion

Learn how to use the Autonomous Operator to perform online persistent volume expansion for Couchbase Server deployments in Kubernetes.

Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

Introduction

In this tutorial you’ll learn how to use the Autonomous Operator to expand persistent volumes that are already in use by Couchbase clusters without needing to perform an upgrade on the underlying storage subsystem. The Autonomous Operator performs storage upgrades by working in conjunction with Kubernetes Persistent Volume Expansion to claim additional storage for running pods without any downtime.

Before You Begin

This tutorial uses the context of Azure Kubernetes Service (AKS), but the steps generally apply to any Kubernetes environment.

Before you begin, you’ll need to set up a few things first:

You’ll need a Kubernetes cluster with at least 3 available worker nodes that have at least 20GiB of storage capacity.
- If you need to set up a cluster on AKS, refer to Quickstart: Deploy an AKS cluster using the Azure portal.
Your environment needs to have a StorageClass capable of performing volume expansions. Refer to Online Volume Expansion for more information about supported storage classes.
- This tutorial references the azurefile storage class which is provided by the Azure Kubernetes Service (AKS). You’ll need to use the name of your particular storage class if installing in a non-AKS Kubernetes environment.
You’ll need Helm version 3.1 or higher for installing the necessary dependencies (e.g. the Autonomous Operator, the Couchbase cluster, etc.)
- Once you have Helm installed, you’ll need to add the Couchbase chart repository:
  $ helm repo add couchbase https://couchbase-partners.github.io/helm-charts/
  Then make sure to update the repository index:
  $ helm repo update
- If needed, the following guide can be helpful to familiarize yourself with Helm on AKS: Install existing applications with Helm in AKS.

Create the Couchbase Cluster Deployment

Let’s start by setting up our Couchbase deployment. To speed up the process, we’ll be using the Couchbase Helm chart to conveniently install a Couchbase cluster with persistent volumes.

Run the following command to create a file with the necessary override values for the Couchbase chart:

$ cat << EOF > pvc_resize_values.yaml
---
cluster:
  cluster:
    dataServiceMemoryQuota: 4Gi
    indexServiceMemoryQuota: 6Gi
  name:
  enableOnlineVolumeExpansion: true (1)
  servers:
    default:
      size: 2
      services:
        - data
      volumeMounts:
        default: default
        data: data-expanding  (2)
    search:
      size: 1
      services:
        - query
        - index
  volumeClaimTemplates: (3)
  - metadata:
      name: data-expanding
    spec:
      storageClassName: azurefile
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: default
    spec:
      storageClassName: default
      resources:
        requests:
          storage: 2Gi
EOF

1	`couchbaseclusters.spec.enableOnlineVolumeExpansion`: Setting this field to `true` enables online expansion of persistent volumes.
2	`couchbaseclusters.spec.servers.volumeMounts`: With this configuration we’re telling the Autonomous Operator to provision persistent volume claims for the `data` mount path according to the `data-expanding` claim template.
3	`couchbaseclusters.spec.volumeClaimTemplates`: This configuration defines the `data-expanding` claim template. `volumeClaimTemplates.spec.storageClassName`: This field refers to the name of the storage class that services volume claims. The storage class must support volume expansion in order to add additional storage in-place. Here we’re using the `azurefile` storage class. `volumeClaimTemplates.spec.resources`: This field defines the size of the volume that will be allocated. Here we’re requesting `5Gi` volumes. Volume expansion is triggered when this value is increased.

Now, install the Couchbase chart, making sure to specify the values override file we just created:

$ helm install -f pvc_resize_values.yaml expand couchbase/couchbase-operator

The Couchbase chart deploys the Autonomous Operator by default. If you already have the Autonomous Operator deployed in the current namespace, then you’ll need to specify additional overrides during chart installation so that only the Couchbase cluster is deployed:

$ helm install -f pvc_resize_values.yaml --set install.couchbaseOperator=false,install.admissionController=false expand couchbase/couchbase-operator

Verify the Installation

The configuration we’re using calls for a three-node Couchbase cluster (two data nodes and one search node), which will take a few minutes to be created. You can run the following command to verify the deployment status:

$ kubectl describe couchbasecluster expand-couchbase-cluster

In the console output, you should check for the events that signal the creation of the three nodes in the Couchbase cluster:

Events:
  Type    Reason                  Age   From  Message
  ----    ------                  ----  ----  -------
  Normal  EventNewMemberAdded     22m         New member scale-couchbase-cluster-0002 added to cluster

Expand the Persistent Volume

Now let’s modify the CouchbaseCluster resource in order to initiate volume expansion.

$ kubectl edit couchbasecluster scale-couchbase-cluster

In the editor, modify the couchbaseclusters.spec.volumeClaimTemplates section as follows:

and change the requested storage of the data-expanding template from 5Gi to 10Gi.

  volumeClaimTemplates:
  - metadata:
      name: data-expanding
    spec:
      resources:
        requests:
          storage: 10Gi (1)

1	Here we’re changing the requested storage of the `data-expanding` template from `5Gi` to `10Gi`. This will trigger the expansion of all the Couchbase services that reference `data-expanding` template.

Save the configuration changes to initiate the volume expansion task.

Verify Volume Expansion

Run the following command to verify the status of the volume expansion task by checking events on the CouchbaseCluster resource:

$ kubectl describe couchbasecluster expand-couchbase-cluster

If the volume expansion was successful, you should expect output similar to the following:

Normal  ExpandVolumeStarted    46s          Expanding Volume scale-couchbase-cluster-0003-data-01 from 5Gi to 10Gi (1)
Normal  ExpandVolumeSucceeded  41s          Successfully expanded volume scale-couchbase-cluster-0002-default-01
Normal  ExpandVolumeStarted    18s          Expanding Volume scale-couchbase-cluster-0004-data-01 from 5Gi to 10Gi (1)
Normal  ExpandVolumeSucceeded  6s           Successfully expanded volume scale-couchbase-cluster-0002-default-01

1	The persistent volume on each of the Couchbase `data` pods has been expanded in place.

Cleaning up

Running the commands in this section will uninstall all of the resources that were created during the course of this tutorial.

Uninstall both the Autonomous Operator and Couchbase cluster by deleting the Helm release:

$ helm delete scale