Couchbase Server Deployment Best Practices
The Couchbase Autonomous Operator makes deploying Couchbase Server incredibly simple. However, there are some external influences and configurations that can cause issues. This topic outlines some of the deployment best practices that can help you avoid some of the most common pitfalls.
Pod scheduling details how to deploy your Couchbase Server clusters to consistently maintain high performance, and also minimize disruption in the case of failure.
The noisy neighbor problem occurs when multiple VMs are running on the same hypervisor, and one virtual machine (VM) adversely affects the other. This effect may result in any of the following conditions:
- CPU Starvation
A highly CPU-intensive neighbor consumes more than its fair share of CPU resources, which results in fewer cycles allocated to the affected pod, higher latencies when processing data and network traffic, and increased cache latencies as the noisy neighbor flushes your entries to slower storage.
- Memory Starvation
There are two common scenarios when considering memory starvation. In the cloud, a hypervisor VM will typically run without swap space to page out less-used memory pages. The result is that when memory is starved, the operating system has no other choice but to choose a process that is using too much memory and terminate it. When the hypervisor is on physical hardware with a swap file, performance will suffer as memory pages are transferred to and from a storage device, resulting in slower computation as memory accesses page-fault and the memory is swapped back in, higher latencies, and degraded I/O performance.
- Disk I/O Starvation
A storage device has a finite number of input or output operations it can perform per second. Typically for a spinning disk this is in the order of 100 IOPs. For solid state storage, this is far higher — in the order of 10,000s of IOPs. If a noisy neighbor is utilizing a high proportion of that capacity, then an application will see slower reads and writes to and from the file system. In a cloud architecture, where a storage volume may be attached via a network protocol (e.g. iSCSI), you also generate increased network load.
- Network Starvation
Like disk I/O, the network only has a limited capacity. Modern data center network interface cards will typically handle 10 Gbps of bandwidth. The problem is however not limited to just the local machine, as top-of-rack leaf switches are often over-subscribed by a ratio of 6:1 (or higher), so a network-intensive process may be affected by a process on a completely separate hypervisor.
To mitigate as many of these factors as is possible, it is recommended that you deploy a Couchbase Server cluster on a set of Kubernetes nodes that are separate from other processes. The Kubernetes cluster administrator should be responsible for labeling nodes as belonging to a specific application type. Developers may then use the
spec.servers.pod.nodeSelector configuration property to control which nodes Couchbase pods are scheduled onto. Take care to ensure that other pods (e.g. stateless web applications) are not scheduled on those nodes that are allocated for use by Couchbase.
Likewise, it is recommended that you deploy Couchbase Server clusters with anti-affinity enabled so that cluster pods are not scheduled on the same nodes. This can be achieved with the
spec.antiAffinity configuration property.
|Anti-affinity is only scoped to a specific cluster. In order to prevent pods from one cluster interfering with another, you should use a separate node label per Couchbase cluster.|
The server group feature of the Couchbase data platform allows you group Couchbase Server nodes in such a way that data or index replicas are distributed over failure domains that are larger than that of a single server (e.g. rack, data center, availability zone, etc). When server groups are defined, the Server Group Awareness feature distributes vBuckets on a best-effort basis, while ensuring each instance is under an equal load. As such, under certain circumstances, it’s possible for multiple replicas to be co-located in the same server group.
It is recommended that server groups should have an equal number of pods per server class to make it simpler to maintain server group constraints. It is also recommended that you have more server groups than replicas for the same reason.
For more information, refer to About Using Couchbase Server Groups With the Operator.
Storage options may have both beneficial and detrimental impacts on your clusters.
Persistent volumes are recommended for production deployments. This allows for the quick recovery of a failed pod by warming up from existing data and allowing quicker back-filling via delta-recovery, rather than a full re-balance. By having some of the pod’s file system persisted, logging data can be extracted from failed pods where this would normally be lost with a fully ephemeral file system. It also aids in data recovery situations which cannot be rectified by the Operator.
It is also highly recommended that in cloud deployments, storage be located in the same data center as the pods that are running in that server group. If a data center were to suffer an outage with storage randomly distributed across a virtual private cloud, not only would you lose all the pods that are resident in the availability zone, but also any pods with backing storage provided by the availability zone.
|There will be a slight reduction in performance and latency when using a network-based storage backend.|
Keeping your data safe is of paramount importance. This section contains some best practices that you should follow to keep your deployment secure.
When you deploy a cluster, you do so in a namespace. The Operator needs to be given a fairly broad set of permissions in order to dynamically create pods, services, etc. It is recommended that the Operator, and the clusters that it manages, be segregated in their own name space. This limits the set of resources that the Operator can affect, and that support tools can collect data about, which aids in security compliance situations where confidential information may be leaked via the Kubernetes API operations.
How your Kubernetes nodes are configured may also have a detrimental impact on your ability to deploy a Couchbase Server cluster. These are not officially supported but recommended for cluster administrators.
It is recommended that the following services be running on all Kubernetes nodes.
Like all distributed systems, Couchbase Server needs to have synchronized time in order to resolve conflicts. Newer systems should have NTP enabled by default if running
systemd-timesyncd. Older systems should run
ntpdor similar and can be automated via an orchestration tool such as Puppet or Ansible.
Couchbase server has some recommended kernel parameters which should be set. These can be done via an orchestration solution or a Kubernetes
For systems with a swap file, this controls how aggressively the kernel will transfer unused application memory pages from physical RAM onto disk. It is highly recommended that this value be set to
0to improve the data service performance, and by extension dependent services.
Transparent huge pages (THP) attempt to allocate large areas of contiguous memory so that applications which process data sequentially, or that have high data locality, do so without causing a high number of kernel page faults. For a database such as Couchbase, this actually has a detrimental effect, since its access patterns are typically highly random. It is highly recommended that this parameter be set to
mm.transparent_hugepage.enabled, it is highly recommended that this parameter be set to
For example, you could run something similar to the following:
apiVersion: apps/v1 kind: DaemonSet metadata: name: couchbase-sysctls namespace: kube-system spec: selector: matchLabels: name: couchbase-sysctls template: metadata: labels: name: couchbase-sysctls spec: # Apply to nodes marked as belonging to a couchbase cluster only nodeSelector: app: couchbase-cluster containers: - name: couchbase-sysctls image: busybox:latest # Run as root on the host system securityContext: privileged: true # Constantly reconcile the state to revert any changes command: - /bin/sh - -c - | set -o xtrace while true do sysctl -w vm.swappiness=0 sysctl -w mm.transparent_hugepage.enabled=never sysctl -w mm.transparent_hugepage.defrag=never sleep 60 done
These ulimit settings are necessary when running under heavy load. If you are just doing light testing and development, you can omit these settings, and everything will still work.
The Couchbase Pod ulimits are inherited from the default ulimits set on the docker daemon running on each kubernetes node. Configuration of this daemon varies according to the distributions service management system.
Systemd compatible hosts can configure ulimits for the daemon by editing its service configuration file. Run the following command to detect the location of the location of the configuration file:
$ sudo systemctl cat docker # /lib/systemd/system/docker.service
Add the following values to the service file:
LimitNOFILE=40960 LimitCORE=infinity LimitMEMLOCK=infinity
Reload the new configuration and restart the docker daemon:
$ sudo systemctl daemon-reload $ sudo systemctl restart docker
Upstart compatible hosts can configure ulimits for the docker daemon by editing the
$ vi /etc/init/docker.conf
And the following values to the service configuration:
limit nofile 40960 limit core unlimited unlimited limit memlock unlimited unlimited
Restart the docker service to apply ulimits to the daemon:
$ sudo /etc/init.d/docker restart