Persistent Volumes Zoning Guide (Legacy)

When deploying on a cloud that spans multiple availability zones, certain steps are required in order to use persistent volumes. These steps will vary depending on the version of Kubernetes being provided by your cloud.

Kubernetes 1.11.x requires a storage class to serve volumes within every zone of your cloud provider. Therefore, Couchbase clusters that use server groups with Kubernetes 1.11.x require following steps to be preformed before deployment:

  • Step 1: Label nodes with zone/region selectors

  • Step 2: Add server groups to the cluster configuration

  • Step 3: Create zoned storage classes for each zone

  • Step 4: Add storage classes to the cluster configuration

If your cloud provider is running Kubernetes 1.12+, refer to the non-legacy zoning guide.

Label Nodes

Kubernetes nodes must be labeled according to the zones where they are physically located. Most cloud providers will already have the nodes labeled appropriately, but if this is not the case you will need to do it manually. This ensures that persistent volumes are provisioned into the same zones as the nodes.

The key failure-domain.beta.kubernetes.io/zone must be used when creating the label. For example, to label a node in Amazon EC2 within zone us-east-1a:

kubectl label nodes ip-172-16-0-10 failure-domain.beta.kubernetes.io/zone=us-east-1a

For more information about labeling nodes, see the server groups documentation.

Add Server Groups to the Configuration

Once nodes are labeled according to their associating zones, these zones need to be added to the Couchbase cluster spec under serverGroups. Server groups will ensure that the Operator creates pods in a specific zone. The following snippet demonstrates configuring Couchbase to deploy pods across three zones:

spec:
  serverGroups:
    - us-east-1a
    - us-east-1b
    - us-east-1c
  servers:
    - name: east-1a
      size: 3
      services:
        - data
        - index
      serverGroups:
        - us-east-1a
    - name: east-1b
      size: 3
      services:
        - data
        - index
      serverGroups:
        - us-east-1b
    - name: east-1c
      size: 2
      services:
        - query
        - search
      serverGroups:
        - us-east-1c

Only one server group should be used for each group of spec.servers[*].serverGroups. This is because later in the configuration you’ll notice that there is a 1:1 ratio between storage class and server groups, and having multiple server groups would cause some pods to be created in a different zone than where the storage class is able to create volumes.

Create Zoned Storage Classes

While the previous steps dealt with deploying pods into specific availability zones, the next steps deal with deploying persistent volumes into specific availability zones. This is necessary because the default behavior of a storage class is to create persistent volumes across zones in a round-robin fashion, which unfortunately leads to scheduling conflicts when a pod attempts to mount a volume that gets created outside of its availability zone. For example, a volume created in zone us-east-1a cannot be mounted by pods that are scheduled in zone us-east-1b. Therefore, a Kubernetes storage class must be created for each zone that is designated to host pods with persistent volumes.

You may have to refer to the details of your storage class for information about how to constrain volume creation to a specific zone/region. The following snippet is an example of a storage class that uses the aws-ebs provisioner to restrict volume creation to zone us-east-1a:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: us-east-1a
parameters:
  type: gp2
  zone: us-east-1a
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete

Add Storage Classes to the Configuration

Storage classes are used by the persistent volume claim templates to dynamically create volumes for Pods. Since there is a storage class for each zone, you’ll also need to create a persistent volume claim template for each Zone.

The following snippet is an example configuration that is required for using storage classes across three different zones:

spec:
  volumeClaimTemplates:
    - metadata:
        name: us-east-1a
      spec:
        storageClassName: "us-east-1a"
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: us-east-1b
      spec:
        storageClassName: "us-east-1b"
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: us-east-1c
      spec:
        storageClassName: "us-east-1c"
        resources:
          requests:
            storage: 10Gi

Now that the templates are added, the final step is to pair the volume claim template with server groups according to the zones that they service. For instance, pods within a server group named us-east-1a should use the volumeClaimTemplate named us-east-1a, which itself is using the storage class named us-east-1a.

For example, the following snippet shows the pairing of a server group and its associated volumeClaimTemplate:

spec:
  servers:
    - name: east-1a
      size: 3
      services:
        - data
        - index
      serverGroups:
        - us-east-1a
      pod:
        volumeMounts:
          default: us-east-1a
          data: us-east-1a
  volumeClaimTemplates:
    - metadata:
        name: us-east-1a
      spec:
        storageClassName: "us-east-1a"
        resources:
          requests:
            storage: 1Gi

Notice that the volume claim template is specified within servers.pod.volumeMounts (us-east-1a) and that this claim template uses a storage class spec.volumeClaimTemplates.spec.storageClassName (us-east-1a) which matches the server group specified in servers.serverGroups (also us-east-1a). Namings are arbitrary, but naming claims, groups, and classes according to zones will make it easier to check that everything matches.

Finally, the full spec for deploying across three different zones looks like this:

spec:
  servers:
    - name: east-1a
      size: 3
      services:
        - data
        - index
      pod:
        volumeMounts:
          default: us-east-1a
          data: us-east-1a

    - name: east-1b
      size: 3
      services:
        - data
        - index
      pod:
        volumeMounts:
          default: us-east-1b
          data: us-east-1b

    - name: east-1c
      size: 3
      services:
        - search
        - query
      pod:
        volumeMounts:
          default: us-east-1c

  volumeClaimTemplates:
    - metadata:
        name: us-east-1a
      spec:
        storageClassName: "us-east-1a"
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: us-east-1b
      spec:
        storageClassName: "us-east-1b"
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: us-east-1c
      spec:
        storageClassName: "us-east-1c"
        resources:
          requests:
            storage: 10Gi

See the CouchbaseCluster configuration documentation for more detailed information about defining a cluster with persistent volumes.