Memory
Couchbase Server memory-management ensures high performance and scalability.
Service Memory Quotas
Memory-quota allocation on Couchbase Server occurs per service (except in the cases of the Query Service and Backup Service, which do not require specific allocations). This allows the availability of memory-resources to be tuned according to the assignment of services, node by node. Note that the Data Service must run on at least one node; and that on each of those nodes, quotas for buckets, specified at the time of bucket-creation, are subtracted from the quota allocated to the Data Service.
The memory-quota allocation specified for a given service applies to every instance of that service across the cluster. For example, if 2048 MB is specified for the Analytics Service, and the Analytics Service is run on three of a cluster’s five nodes, each of the three instances of the service is duly allocated 2048 MB. Note that it is not possible to have different memory allocations across multiple instances of the same service within a single cluster.
As a guideline, Couchbase recommends that no more than 90% of a node’s memory (80% on nodes with a small amount of total memory) be allocated to the server and its services.
The firm limits on server memory allocation are: \(max(total\_memory - 1, 80\% \times total\_memory)\) where total_memory is the maximum memory on the node in GiB.
Instructions on how to allocate memory quotas to services when initializing a new cluster can be found in the section Create a Cluster.
When a node is added to a cluster, the Default Configuration, as established by the set-up of the first node in the cluster, is available: this covers all configurable elements, including memory quotas. However, if insufficient memory for the default configuration is available on the new node, the default configuration is prohibited: in such cases, settings for the new node can be custom-configured, allowing an appropriate subset of services to be specified.
If, when the initial node of a cluster is set up, only a subset of services is assigned, additional nodes can subsequently be added, in order to host additional services: in which case, each new service can be given its initial memory allocation as its node is added.
Instructions on how to add a new node to a cluster can be found in the section Create a Cluster.
The Memory Quotas panel on the General settings screen of Couchbase Web Console lists all services running on the cluster, and specifies the memory allocation for each. The panel is interactive, and allows the memory allocations to be changed and saved. If a modification attemptedly exceeds a permitted minimum-required or maximum-available level, a notification of the error is displayed, and the modification is disallowed.
Minimum-required memory-quotas (which are the quota-defaults initially provided by Couchbase Server) are:
| Service | Minimum Memory Quota (in MB) |
|---|---|
Data |
256 |
Index |
256 |
Search |
256 |
Analytics |
1024 |
Eventing |
256 |
Note that neither the Query Service nor the Backup Service requires an administrator-specified memory quota.
Initialization and Warmup
When Couchbase Server is restarted on a node, the node goes through a warmup process before it restarts the handling of data requests. During this warmup process, data on disk is sequentially reloaded into memory for each bucket. The time required for the reload depends on the size and configuration of the system, the amount of data persisted on the node, and the ejection policy configured for the buckets.
On a given Data Service node, a bucket’s data is loaded in accordance with determinations made by the access scanner. This program runs by default once every day. It checks the resident ratio — which is the percentage of items in active and replica vBuckets that are currently in memory on the node:
-
If the ratio is below 95%, the access scanner generates a new access log, which records the documents that have been most frequently accessed during the last 24 hours. If and when data-loading subsequently occurs, the new access log is consulted, the recorded document-keys obtained, and the corresponding documents loaded with the highest priority.
-
If the ratio is above 95%, the access scanner does not generate a new access log. Instead, it deletes any existing access log, and exits. If and when data-loading subsequently occurs, since no access log exists, loading occurs with no priority-order (this being, in cases of extremely high resident ratio, the more performative loading procedure).
Configuring the Access Scanner
The access scanner is configurable, via the CLI utility cbepctl, with the flush_param parameter.
This utility also provides the parameters warmup_min_memory_threshold and warmup_min_item_threshold, which can be used to schedule the resumption of traffic before all items have been reloaded into memory.
Note that the access scanner is a highly CPU-intensive process, whose configurable start-time should ideally be established as different for each Data Service node in the cluster. See set flush_param.
Ejection
If a bucket’s memory quota is exceeded, items may be ejected from the bucket by the Data Service. Different ejection methods are available, and are configured per bucket. Note that in some cases, ejection is configured not to occur. For detailed information, see Buckets.
For each bucket, available memory is managed according to two watermarks, which are mem_low_wat and mem_high_wat.
If data is continuously loaded into the bucket, its quantity eventually increases to the value indicated by the mem_low_wat watermark.
At this point, no action is taken.
Then, as still more data is loaded, the data’s quantity increases to the value indicated by the mem_high_wat watermark.
If, based on the bucket’s configuration, items can be ejected from the bucket, the Data Service ejects items from the bucket until the quantity of data has decreased to the mem_low_wat watermark.
In cases where ejection cannot free enough space to support continued data-ingestion, the Data Service stops ingesting data, error messages are sent to clients, and the system displays an insufficient memory notification.
When sufficient memory is again available, data-ingestion resumes.
Items are selected for ejection based on metadata that each contains, indicating whether the item has been recently used. If an item has not been recently used, it is a candidate for ejection.
The relationship of mem_low_wat and mem_high_wat to the bucket’s overall memory quota is illustrated as follows:
The default setting for mem_low_wat is 75%.
The default setting for mem_high_wat is 85%.
The default settings can be changed by means of the cbepctl utility.
See set flush_param.
Expiry Pager
Scans for items that have expired, and erases them from memory and disk; after which, a tombstone remains for a default period of 3 days.
The expiry pager runs every 10 minutes by default: for information on changing the interval, see cbepctl set flush_param.
For more information on item-deletion and tombstones, see Expiration.
Active Memory Defragmenter
Over time, Couchbase Server-memory can become fragmented. Each page in memory is typically responsible for holding documents of a specific size-range. Over time, if memory pages assigned to a specific size-range become sparsely populated (due to documents of that size being ejected, or to items changing in size), the unused space in those pages cannot be used for documents of other sizes, until a complete page is free, and that page is re-assigned to a new size. Such effects, which are highly workload-dependent, may result in memory that cannot be used efficiently.
Couchbase Server provides an Active Memory Defragmenter, which periodically scans the cache, to identify pages that are sparsely used. It then repacks the items on those pages, to free up space.