Couchbase Server memory-management ensures high performance and scalability.
Service Memory Quotas
Memory-quota allocation on Couchbase Server occurs per service (except in the case of the Query Service, which does not require a specific allocation). This allows the availability of memory-resources to be tuned according to the assignment of services, node by node. Note that the Data Service must run on at least one node; and that on each of those nodes, quotas for buckets, specified at the time of bucket-creation, are subtracted from the quota allocated to the Data Service.
The memory-quota allocation specified for a given service applies to every instance of that service across the cluster. For example, if 2048 MB is specified for the Analytics Service, and the Analytics Service is run on three of a cluster’s five nodes, each of the three instances of the service is duly allocated 2048 MB. Note that it is not possible to have different memory allocations across multiple instances of the same service within a single cluster.
By default, Couchbase Server allows 90% of a node’s total available memory to be allocated to the server and its services. Consequently, if a node’s total available memory is 100 GB, any attempt to allocate memory beyond 90 GB produces an error.
Instructions on how to allocate memory quotas to services when initializing a new cluster can be found in the section Create a Cluster.
When a node is added to a cluster, the Default Configuration, as established by the set-up of the first node in the cluster, is available: this covers all configurable elements, including memory quotas. However, if insufficient memory for the default configuration is available on the new node, the default configuration is prohibited: in such cases, settings for the new node can be custom-configured, allowing an appropriate subset of services to be specified.
If, when the initial node of a cluster is set up, only a subset of services is assigned, additional nodes can subsequently be added, in order to host additional services: in which case, each new service can be given its initial memory allocation as its node is added.
Instructions on how to add a new node to a cluster can be found in the section Create a Cluster.
The Memory Quotas panel on the General settings screen of Couchbase Web Console lists all services running on the cluster, and specifies the memory allocation for each. The panel is interactive, and allows the memory allocations to be changed and saved. If a modification attemptedly exceeds a permitted minimum-required or maximum-available level, a notification of the error is displayed, and the modification is disallowed.
Minimum-required memory-quotas (which are the quota-defaults initially provided by Couchbase Server) are:
|Service||Minimum Memory Quota (in MB)|
Initialization and Warmup
When Couchbase Server is restarted on a node, the node goes through a warmup process before it restarts the handling of data requests. During this warmup process, data on disk is sequentially reloaded into memory. The time required for the reload depends on the size and configuration of the system, the amount of data persisted on the node, and the ejection policy configured for the buckets.
Frequently used items are identified via a scanner process, which examines an access log, and obtains the appropriate keys.
Corresponding items are then loaded with the highest priority.
The scanner process is configurable, via the CLI utility
cbepctl, with the
This utility also provides the parameters
warmup_min_item_threshold, which can be used to schedule the resumption of traffic before all items have been reloaded into memory.
See set flush_param.
If a bucket’s memory quota is exceeded, items may be ejected from the bucket by the Data Service. Different ejection methods are available, and are configured per bucket. Note that in some cases, ejection is configured not to occur. For detailed information, see Buckets.
For each bucket, available memory is managed according to two watermarks, which are
If data is continuously loaded into the bucket, its quantity eventually increases to the value indicated by the
At this point, no action is taken.
Then, as still more data is loaded, the data’s quantity increases to the value indicated by the
If, based on the bucket’s configuration, items can be ejected from the bucket, the Data Service ejects items from the bucket until the quantity of data has decreased to the
In cases where ejection cannot free enough space to support continued data-ingestion, the Data Service stops ingesting data, error messages are sent to clients, and the system displays an insufficient memory notification.
When sufficient memory is again available, data-ingestion resumes.
Items are selected for ejection based on metadata that each contains, indicating whether the item has been recently used. If an item has not been recently used, it is a candidate for ejection.
The relationship of
mem_high_wat to the bucket’s overall memory quota is illustrated as follows:
The default setting for
mem_low_wat is 75%.
The default setting for
mem_high_wat is 85%.
The default settings can be changed by means of the
See set flush_param.
Scans for items that have expired, and erases them from memory and disk; after which, a tombstone remains for a default period of 3 days.
The expiry pager runs every 60 minutes by default: for information on changing the interval, see
cbepctl set flush_param.
For more information on item-deletion and tombstones, see Expiration.
Active Memory Defragmenter
Over time, Couchbase Server-memory can become fragmented. Each page in memory is typically responsible for holding documents of a specific size-range. Over time, if memory pages assigned to a specific size-range become sparsely populated (due to documents of that size being ejected, or to items changing in size), the unused space in those pages cannot be used for documents of other sizes, until a complete page is free, and that page is re-assigned to a new size. Such effects, which are highly workload-dependent, may result in memory that cannot be used efficiently.
Couchbase Server provides an Active Memory Defragmenter, which periodically scans the cache, to identify pages that are sparsely used. It then repacks the items on those pages, to free up space.