Data Service Metrics
A list of the metrics provided by the Data Service.
The following Data Service metrics can be queried by means of the REST APIs described in Statistics.
|
Tip
|
The number of audit events dropped due to errors while trying to insert them to the audit trail |
Boolean value to indicate if audit is enabled or not |
The number of authentication commands |
The number of failed authentication requests |
Batch size for background fetches |
Background fetches waiting for disk |
Background fetches waiting in the dispatcher queue |
The number of references to the bucket |
Per-opcode histogram of time taken to execute operations |
The number of lookup operations |
The number of lookup operations performed within the last 10 seconds |
The total duration of lookup operations performed over the last 10 seconds |
The number of mutation operations |
The number of mutation operations performed within the last 10 seconds |
The total duration of mutation operations performed over the last 10 seconds |
The total number of data retrieval operations (all buckets) |
The sum of cmd_total_sets and cmd_total_gets (all buckets) |
The total number of mutation operations (all buckets) |
Per-collection data size on disk |
Whether history (CDC) is enabled for each collection |
Per-collection item count |
Per-collection maxTTL (maximum expiry) if configured |
Per-collection memory usage |
Per-collection counters of sets/gets/deletes |
Counter of all SetWithMeta/DelWithMeta conflict resolution results. The result may be that the incoming operation was: accepted as it is 'ahead', rejected as it is 'behind', or rejected as it appears identical (by metadata, not comparing document bodies) |
The total number all clients in this bucket yield due to using their entire timeslice |
The total number all clients in this bucket yield due to consuming the number of ops allowed for the current timeslice |
Current number of allocated connection structures |
The current number of connections. This includes user, system and daemon connections |
Count of alive (non-deleted) items in active vbuckets, including non-resident items |
Total number of items |
Number of temporary items in memory |
The number of server sockets currently in use |
Total amount of memory allocated (outside the context of a bucket) |
Total amount of memory resident (outside the context of a bucket) |
Count of items in memory with a given datatype combination |
Number of times Consumer DCP connections (i.e., replica) have paused consuming items because memory usage is too high |
Current number of DCP connections (Consumers or Producers) |
Current number of DCP connections |
Number of items pushed into the DCP stream ready queue from a backfill |
Current total number of items remaining for to be sent for all outgoing DCP streams (approximate) |
Total number of items sent out by all currently existing outgoing DCP streams, since each stream was created |
Maximum number of backfills across all DCP connections |
Total number of running backfills across all DCP connections |
Count of how many times the DCP connection has been paused |
Estimated memory usage of items waiting to be sent across all existing DCP connections |
Current number of Streams (Active or Passive) |
Total data sent across all existing DCP connections |
Total equivalent uncompressed size of data sent across all existing DCP connections |
Count of how many times the DCP connection has been unpaused |
time spent waiting for disk |
Current memory used in KV for primary/secondary domain |
True if access scanner task is enabled |
Number of seconds that last Access Scanner task run took to complete. |
Number of items that last Access Scanner task run wrote to the Access Log. |
Time of the next access scanner task (GMT), NOT_SCHEDULED if access scanner has been disabled |
Total number of times a vbucket saw an item with a HLC CAS from too far in the future (indicating the clock is behind) |
Let EPE delete/prepare/del_with_meta prune any invalid body in the payload instead of failing |
Logging block size. |
|
|
Number of minutes between each sweep for the access log |
Hour in GMT time when access scanner task is scheduled to run |
The total memory allocated from the engine's arena |
The resident set size of the engine's arena |
|
Total number of times a vbucket saw an item with a HLC CAS from too far in the past (indicating the clock is ahead) |
|
|
|
|
Average read amplification for all background fetch operations - ratio of read()s to documents fetched. |
Number of items fetched from disk |
The number of bgfetches which are triggered by compaction |
The average time for an item to be loaded from disk |
The total elapsed time for items to be loaded from disk |
The longest load time when loading from disk |
The longest time in the queue waiting to be loaded from disk |
Number of metadata fetches from disk |
The shortest load time when loading from disk |
The shortest time in the queue waiting to be loaded from disk |
The number of samples included in the average |
Number of remaining bg fetch items |
Number of remaining bg fetch jobs |
The average wait time for an item before it's serviced by the dispatcher |
The total elapse time for the wait queue |
The number of blob objects in the cache |
The number of blob object allocations |
The number of blob object deallocations |
Time in seconds between the BucketQuotaChangeTask polling memory usage to attempt to reduce the bucket quota |
Memory quota (in bytes) for this bucket. |
Actual max size in bytes of a single Checkpoint |
Number of tasks responsible for destroying closed unreferenced checkpoints. |
Max size (in bytes) of a single checkpoint. '0' for EPEngine auto-setup. |
Memory of items in all checkpoints |
Memory of checkpoint structures awaiting destruction by a background task |
Max allocation allowed in all checkpoints |
Max ratio of the bucket quota that can be allocated in checkpoints. The system enters a TempOOM phase if hit. |
Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that represents the target of checkpoint memory recovery. Memory recovery yields when reached. |
Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that represents the target of checkpoint memory recovery. Memory recovery yields when reached |
Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that triggers attempt of memory releasing from checkpoint. |
Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that triggers attempt of memory releasing from checkpoint |
Number of concurrent tasks performing ItemExpel and CursorDrop/CheckpointRemoval |
Enable the ability to expel (remove from memory) items from a checkpoint. An item can be expelled if all cursors in the checkpoint have iterated past the item. |
Number of remaining vbuckets for checkpoint persistence |
|
ep_active_ahead_exceptions + ep_replica_ahead_exceptions |
How many milliseconds before compaction runs following the drop of a collection |
Enable the collections functionality, enabling the storage of collection metadata |
Total number of write commits |
Number of milliseconds of most recent commit |
Cumulative milliseconds spent committing |
Counter of how many times compaction aborted, e.g. the vbucket is required to rollback, so compaction is aborted |
Should compaction expire items that were logically deleted at the start of the compaction (true) or at the point in time at which they were visited (false)? |
If compaction requires a bgfetch before attempting expiry to ensure it does not expire an older version of the document, true: fetch it in the compaction thread. false: queue a bgfetch for the bgfetcher task to complete |
Counter of how many times compaction has failed, e.g. a system call error caused compaction to fail |
|
Number of eviction pager tasks to create when memory usage is high |
How often connection manager task should release dead connections (in seconds). |
How often connection manager task should be run (in seconds). |
Maximum number of couchstore files that we will keep open. Default value is 30 * 1024 (i.e. one file for each vBucket and 30 Buckets - the supported limit). |
Should we have to rollback more than half of the seqnos seen by this vBucket we will instead rollback to 0 and re-stream from the active if set to true |
Enable couchstore to mprotect the iobuffer |
Enable couchstore tracing |
Validate couchstore writes |
Allow this Bucket's HashTable quota to be shared with other Buckets which have this setting enabled. |
Number of cursors dropped by the checkpoint remover |
Total number of get failures |
True if we want to enable data traffic after warmup is complete |
Total compaction and commit failures |
Total size of valid data in db files |
Total size of the db files |
The total size of all history currently stored by the bucket |
The timestamp of the oldest document stored in the history window, oldest of all vbuckets |
Total size of SyncWrite prepares in db files |
Max bytes a connection can backfill into memory before backfill is paused |
The maximum number of backfills each connection can have in-progress (i.e. KVStore snapshot open and reading data from) |
Ratio of the BucketQuota that can be allocated by all DCP consumers for buffered messages |
Ratio of freed bytes in the DCP Consumer buffer that triggers a BufferAck message to the Producer |
Max seconds after which a Consumer acks all the remaining freed bytes, regardless of whether dcp_consumer_flow_control_ack_ratio has kicked-in or not |
Whether DCP Consumer on this node enable flow control |
The number of DcpConsumerTask iterations before forcing the task to yield. |
Whether DCP Consumer connections should attempt to negotiate no-ops with the Producer |
The maximum number of seconds between dcp messages before a connection is disconnected |
|
Forces clients to enable noop for v5 features |
The time interval in seconds between noop messages being sent to the consumer |
|
|
|
|
The number of snapshots before ActiveStreamCheckpointProcessorTask::run yields. |
Max bytes that can be read in a single backfill scan before yielding |
Max items that can be read in a single backfill scan before yielding |
Max amount of time for takeover send (in seconds) after which front end ops would return ETMPFAIL |
How old (measured in number of DefragmenterVisitor passes) must a document be to be considered for defragmentation. |
When mode is not static and scored fragmentation is above this value, a sleep time between defragmenter_auto_min_sleep and defragmenter_auto_max_sleep will be used |
The maximum sleep that the auto controller can set |
The minimum sleep that the auto controller can set |
The d term for the PID controller |
The dt (interval) term for the PID controller. Value represents milliseconds |
The i term for the PID controller |
The p term for the PID controller |
When mode is auto_linear and scored fragmentation is above this value, the defragmenter will use defragmenter_auto_min_sleep |
Maximum time (in ms) defragmentation task will run for before being paused (and resumed at the next defragmenter_interval). |
True if defragmenter task is enabled |
How often defragmenter task should be run (in seconds). |
Number of items moved by the defragmentater task. |
Number of items visited (considered for defragmentation) by the defragmenter task. |
The amount of time the defragmenter task will sleep before it is scheduled to run again. |
How old (measured in number of DefragmenterVisitor passes) must a StoredValue be to be considered for defragmentation. |
Number of StoredValues moved by the defragmentater task. |
True if the engine is either warming up or data traffic is disabled |
Total drained items on disk queue |
Total enqueued items on disk queue |
Total items in disk queue |
Total memory used in disk queue |
Total bytes of pending writes |
Maximum time (in ms) ephemeral hash table cleaner task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval). |
Age in seconds after which Ephemeral metadata is purged entirely from memory. Purging disabled if set to -1. |
Time in seconds between automatic, periodic runs of the Ephemeral metadata purge task. Periodic purging disabled if set to 0. |
Maximum time (in ms) ephemeral stale metadata purge task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval). |
True if expiry pager task is enabled |
Hour in GMT time when expiry pager can be scheduled for initial run |
Number of seconds between expiry pager runs. |
Number of times an item was expired on application access |
Number of times an item was expired by the compactor |
Number of times an item was expired by the item pager |
Number of tasks which are created to scan for and delete expired items |
Time of the next expiry pager task (GMT), NOT_SCHEDULED if expiry pager has been disabled |
If true then do not allow traffic to be enabled to the bucket if warmup didn't complete successfully |
Cumulative milliseconds spent flushing |
Number of items currently being written |
Number of items that all flushers can be currently flushing. Each flusher has flusher_total_batch_limit / num_writer_threads individual batch size. Individual batches may be larger than this value, as we cannot split Memory checkpoints across multiple commits. |
The increment factor of the ProbabilisticCounter being used for the frequency counter. The default value of 0.012 is set such that it allows an 8-bit ProbabilisticCounter to mimic a uint16 counter. See the comment on the ProbabilisticCounter class for more information. |
Perform a file sync() operation after every N bytes written. Disabled if set to 0. |
The default timeout for a getl lock in (s) |
The maximum timeout for a getl lock in (s) |
Max bytes of history a bucket should aim to retain on disk. |
Seconds of history the bucket should aim to retain on disk. |
The μs threshold of drift at which we will increment a vbucket's ahead counter. |
The μs threshold of drift at which we will increment a vbucket's behind counter. |
The accumulated number of times the corresponding kv_ep_hlc_drift_count has been updated |
The accumulated drift between this node's HLC and the remote node. For active vbucket's this represents the difference in CAS and local HLC for withMeta operations, for replica vbucket's this represents the difference in CAS and local HLC from DCP replication. |
The total byte size of all items, no matter the vbucket's state, no matter if an item's value is ejected. Tracks the same value as ep_total_cache_size |
|
Interval in seconds to wait between HashtableResizerTask executions. |
Initial number of slots in HashTable objects. |
Accumulated count of read system calls issued by BG fetches, only maintained by couchstore buckets |
Total number of bytes read during compaction |
Total number of bytes written during compaction |
Total number of bytes written. Only maintained by couchstore buckets and includes Couchstore B-Tree and other overheads |
Total number of bytes read |
Total number of bytes written |
Number of times a transaction failed to start due to storage errors |
Number of times a transaction failed to commit due to storage errors |
Maximum time (in ms) item compression task will run for before being paused (and resumed at the next item_compressor_interval). |
How often the item compressor task should run (in milliseconds) |
Number of items compressed by the item compressor task. |
Number of items visited (considered for compression) by the item compressor task. |
The age percentage used when determining the age threshold in the learning_age_and_mfu eviction policy. |
|
Percentile of existing item MFU distribution to use to determine the MFU to give to new items (0 would insert new items with MFU equal to that of the coldest item present; 100 to that of hottest item present) for the upfront_mfu_only eviction strategy. |
Time between updates of the initial MFU given to new items (seconds) |
Number of times an item is not flushed due to the expiry of the item |
Number of times an item failed to flush due to storage errors |
Maximum time (in ms) itemFreqDecayer task will run for before being paused. |
The percent that the frequency counter of a document is decayed when visited by item_freq_decayer. |
The number of item objects allocated |
The number of item object allocations |
The number of item object deallocations |
Number of items expelled from checkpoints. Expelled refers to items that have been ejected from memory but are still considered to be part of the checkpoint. |
Number of items removed from closed unreferenced checkpoints |
Memory used to store items metadata, keys and values in the system, no matter the vbucket's state |
Compressed disk size of latest version of the LSM Trees. This includes history |
Number of block cache hits |
Memory used by block cache. Accounts for allocated size of blocks that includes allocator internal fragmentation and any internal cache overheads due to auxilliary structures |
Number of block cache misses |
|
|
Bloom filter memory usage in all versions of the LSM Trees |
Memory usage for some buffers |
Data written to key, seq, local index as part of the KV frontend writes. |
Total bytes returned via get (excluding bytes returned from sequence iterator) |
Bytes read by get / number of Gets |
Checkpoint overhead |
Frequency of checkpoint interval; in seconds. A checkpoint provides a rollback point to which the data store can rollback to in the event of a failure. |
Threshold of data written before a checkpoint is created; threshold is based on a fraction of the total data size. Checkpoints require data to be retained in order to provide rollback capability. If the amount of data written during a checkpoint interval is large, we need to do more frequent checkpoints to reduce space amplification. |
Count of Magma compactions in key, seq and local index. |
Data blocks compressed size; actual size in storage |
The compression ratio calculated by dividing the uncompressed data size by the compressed data size |
Estimated percentage of space savings in compressed data blocks (0-100) |
Data blocks uncompressed size |
Magma compaction always removes duplicate keys but not all sstables are visited during compaction.. This is the minimum fragmentation ratio threshold for when a compaction will be triggerred. |
Magma uses a lazy update model to maintain the sequence index. It maintains a list of deleted seq #s that were deleted from the key Index. |
The block cache is an LRU policy driven cache that is used to maintain index blocks for the sstable's btrees. |
Using direct IO tells magma to bypass the file system cache when writing or reading sstables. |
|
When enabled, if copying a write batch into memtable results in exceeding the write cache quota, Magma avoids the copy and instead flushes the batch to disk on the writer thread itself. This tradeoffs an increase in write latency for reduced memory consumption and obeys quota limits. If copying a batch keeps us under the quota, Magma will to continue to copy and do the flush in background. |
When true, the kv_engine will utilize Magma's upsert capabiltiy but accurate document counts for the data store or collections can not be maintained. |
WAL ensures Magma's atomicity, durability. Disabling it is useful in performance analysis. |
All compactions perform expiry but not all sstables are visited by compaction. Magma maintains an expiry histogram across the kvstore to help determine which range of sstables need to have compaction run on them because there are a significant number of expired items. The frag threshold is the number of expired keys vs keys in the data store. |
Magma maintains statistics about expired documents to run compaction based on magma_expiry_frag_threshold. This config determines the the expiry purger polling interval in seconds to trigger compaction on eligible sstables |
Number of compactions triggered by file count |
Percentage of storage threads that are flusher threads (i.e. with a value of 20 we will allocate 4 (1/5th) of the storage threads to flushers and the remaining 16 (4/5ths) threads will be compactors). |
Number of write cache flushes performed |
The percentage of fragmentation a magma bucket aims to maintain. A 100 value will disable sequence tree compactions by setting the desired fragmentation percentage to 100%. Smaller compactions of the key and local indexes will still run. |
Fragmentation on disk (excludes history) |
Number of get operations |
|
|
Frequency of heartbeat interval; in seconds. A heartbeat task is scheduled to provide cleanup and maintenance when magma is idle. |
Memory usage for MagmaHistogramStats and file histograms |
The logical data size of history |
The logical disk size of history |
History eviction bytes based on size |
History eviction bytes based on time |
Proportion of keyIndex (data+index blocks) and seqIndex (index blocks) in memory |
The WAL buffer is used to stage items to the write ahead log along with control information like begin and end transaction. This parameter refers to the initial WAL buffer size. The WAL buffer will adjust its size up to a maximum of 4MB or down to a minimum of 64KB depending on the transaction batch size with consideration for other magma components which consume memory such as the block cache, bloom filters, write cache and meta data overhead. |
Number of DocInsert operations |
Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key and metadata for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. |
Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. |
Number of compactions triggered by file count for the KeyIndex |
Number of compaction performed on the writer thread for the KeyIndex |
The logical data size, including history |
The logical disk size, including history |
Memory used by LSMTree objects |
Maximum # of checkpoints retained for rollback. |
If the number of storage threads = 0, then we set the number of storage threads based on the number of writer threads up to a maximum of 20 threads and use magma_flusher_thread_percentage to determine the ratio of flusher and compactor threads. |
Maximum time (in seconds) that data is kept in level 0 before it is merged. |
Maximum amount of data that is replayed from the WAL during magma recovery. When this threshold is reached magma, creates a temporary checkpoint to recover at. This is per kvstore and in bytes. |
Magma uses a common skiplist to buffer all items at the shard level called the write cache. The write cache contains items from all the kvstores that are part of the shard and when it is flushed, each kvstore will receive a few items each. Regardless of how much memory might be available, this would be the maximum amount that could be allocated. |
Fraction of memory quota used by magma as it's low water mark. Magma uses this low watermark to size it's write cache and block cache. This sizing includes bloom filters memory usage but bloom filter eviction is based on the memory quota |
Magma total memory ratio of the Bucket Quota across all shards and Magma limit's it's memory usage to this value. |
Minimum interval between two checkpoints; in seconds. Prevents excessive creation of checkpoints. |
Magma creates value blocks for values larger than this size. Value blocks only contain a single KV item and their reads/writes are optimised for memory as it avoids many value copies. Right now compression is turned off for value blocks to reduce memory consumption while building them. This setting should be at least as large as the SeqIndex block size. |
Apply Snappy compression to each document when persisted (magma only) |
Memory consumed by read ahead buffers. They are used for compactions and sequence iterators. This is included in BufferMemUsed |
Total bytes read from disk as per Magma's manual accounting in various code paths |
Total bytes read from disk by compactors |
Total bytes read from disk by gets |
Bytes Read from disk by only Get threads / Bytes outgoing |
Bytes read from disk / bytes outgoing. Bytes read from disk includes Gets and compactors (excluding WAL) |
Number of read IOs performed |
Number of read IOs performed by GetDocs divided by the number of GetDocs |
Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key, metadata and value for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. |
Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. |
Count of Magma compactions in seq index that compact the data level. This are already accounted in ep_magma_seqindex_compactions hence not part of the magma_compactions Prometheus stat family. |
Data written to seq index delta levels as part of frontend update operations. This is already accounted in ep_magma_seqindex_bytes_incoming hence not part of Prometheus stat family magma_bytes_incoming. |
Bytes written by Magma flushes, compactions of the seq index delta levels. This is already accounted into ep_magma_sequndex_write_bytes hence not part of the Prometheus stat family magma_write_bytes. |
Number of compactions triggered by file count for the SeqIndex |
Number of compaction performed on the writer thread for the SeqIndex |
Number of set operations (DocUpsert) |
Couchstore generates a commit point at the end of every batch of items. During normal operation, Magma checkpoints are taken at every magma_checkpoint_interval. Many of the tests require more frequent checkpoints so this configuration parameter makes sure every batch generates a checkpoint. Each checkpoint generated in this way is a "Sync" checkpoint and isn't going to be useful for rollback as it only the latest checkpoint is a "Sync" checkpoiont. A "Rollback" checkpoint will be made instead if we set magma_checkpoint_interval to 0. The "Rollback" checkpoints are stored in the checkpoint queue as potential rollback points. Should be used for testing only! |
Number of fsyncs performed |
Memory used by sstable metadata |
Memory used by SSTable objects |
Number of files used for tables |
Number of table files created |
Number of table files deleted |
Compressed size of all SSTables in all checkpoints, WAL and any other files on disk |
Total memory used by bloom filters, write cache, block cache and index blocks This account for all versions of the trees |
Memory consumed by all LSMTree TreeSnapshots |
Number of time-to-live based compactions |
Disk usage by the WAL |
Total WAL memory used, including WAL buffer and any auxiliary memory |
Bytes written by Magma flushes, compactions and WAL writes. |
Bytes written by Magma compactions. |
Memory usage of the write cache |
Memory is maintained across 3 magma components; Bloom filters, Block cache and Write cache. The least important of these is the write cache. If there is insufficent memory for the write cache, the write cache will grow to the size of the batch and then be immediately flushed and freed. If there is available memory, the write cache is limited to 20% of the available memory (after bloom filter and block cache get their memory up to magma_max_write_cache (128MB). Bloom filters are the most important and are never paged out. Bloom filter memory can cause magma to go above the memory quota. To allevaite this, the bottom layer where the majority of bloom filter memory is, won't use bloom filters when OptimizeBloomFilterForMisses is on (which it is by default). The block cache grows each time the index sizes change. But its growth is bounded by the available memory or what's left over after the bloom filter memory is subtracted. |
Number of compaction performed on the writer thread |
The expected max number of checkpoints in each VBucket on a balanced system. Note: That is not a hard limit on the single vbucket. That is used (together with checkpoint_memory_ratio) for computing checkpoint_max_size, which triggers checkpoint creation. |
maximum number of failover log entries |
Maximum number of bytes allowed for 'privileged' (system) data for an item in addition to the max_item_size bytes |
Maximum number of bytes allowed for an item |
Maximum number of bg fetcher objects (the number of concurrent bg fetch tasks we can run). 0 = auto-configure which means we use the same number as the number of shards (max_num_shards - for historic reasons). See also num_reader_threads. |
Maximum number of flusher objects (the number of concurrent flusher tasks we can run). 0 = auto-configure which means we use the same number as the number of shards (max_num_shards - for historic reasons). See also num_writer_threads. |
Maximum mumber of shards (0 = auto-configure) |
Bucket Priority relative to other buckets |
Memory quota (in bytes) for this bucket. |
Maximum number of threads of any single class (0 = automatically select based on core count) |
A maximum TTL (in seconds) that will apply to all new documents, documents set with no TTL will be given this value. A value of 0 means this is disabled |
Maximum number of vbuckets expected |
Memory recovered from Checkpoint by expelling clean items (i.e. items processed by all cursors) from the queue |
Amount of memory freed through ckpt removal |
|
High water mark (as a percentage) |
|
Low water mark (as a percentage) |
True if memory usage tracker is enabled |
What percent of max_data size should we allow the estimated total memory to lag by (EPStats::getEstimatedTotalMemoryUsed) |
Estimate of how much metadata has been written to disk since startup |
Total memory used by meta data, including the key |
specifies a minimum compression ratio below which storing the document will be stored as uncompressed. |
|
Number of times we ran accesss scanner to snapshot working set |
Number of times accesss scanner task decided not to generate access log |
The number of checkpoint objects allocated |
The number of checkpoint object allocations |
The number of checkpoint object deallocations |
Number of checkpoints detached from CM and owned by Destroyers |
Number of items that could not be ejected |
Number of times we ran expiry pager loops to purge expired items from memory/disk |
Number of times we ran the freq decayer task because a frequency counter has become saturated |
The number of non-resident items |
Number of times Not My VBucket exception happened during runtime |
Number of times we ran pager loops to seek additional memory |
Number of times item values got ejected from memory to disk |
Global number of shared worker threads |
Number of times unrecoverable OOMs happened while processing operations |
How long in milliseconds the ItemPager will sleep for when not being requested to run |
For persistent buckets, the count of compaction tasks. |
Number of ops awaiting pending vbuckets |
Max ops seen awaiting 1 pending vbucket |
Max time (µs) used waiting on pending vbuckets |
Total blocked pending ops since reset |
Total VB persist state to disk |
Age in seconds after which tombstones may be purged. Defaults to 3 days. Max of 60 days. If this is dynamically changed for a magma bucket then magma may not trigger compactions when it should, this can be avoided by running a full manual compaction after changing this parameter. |
Is PiTR enabled or not |
The granularity (interval between each rollback point) in seconds (up to 5 hours) |
The number of seconds of the oldest entry to keep as part of compaction (up to 48 hours) |
Number of items queued for storage |
The ratio for calculating how many RangeScans can exist, a ratio of total KVStore scans. |
The maximum number of range scan tasks that can exist concurrently. Setting to 0 results in num_auxio_threads - 1 tasks |
The maximum lifetime in seconds for a range-scan. Scans that don't complete before this limit are cancelled |
The size of a buffer used to store data read during the I/O phase of a range-scan-continue. Once the buffer size is >= to this value the data is sent to the connection |
whether erroneous tombstones need to be retain during compaction. Erroneous tombstones are those that have invalid meta data in it. For example, a delete time of 0. |
Number of rollbacks on consumer |
Timeout in seconds after which a pending SeqnoPersistence operation is temp-failed |
System-generated engine startup time |
The number of storedval objects allocated |
The number of storedval object allocations |
The number of blob object deallocations |
The total number of bytes ever allocated for storedval objects |
The total number of bytes ever freed by deallocated storedval objects |
The maximum number of supported replicas for SyncWrites. Attempts to issue SyncWrites against a topology with more replicas than this setting will fail with DurabilityImpossible. |
Number of times temporary OOMs happened while processing operations |
The total byte size of all items, no matter the vbucket's state, no matter if an item's value is ejected. Tracks the same value as ep_ht_item_memory |
Total number of items de-duplicated when queued to CheckpointManager |
Total number of items de-duplicated when flushed to disk |
Total number of persisted deletions |
Total number of items queued for persistence |
Total number of persisted new items |
Total number of items persisted |
The amount of items that have not been written to disk |
The total number of bytes ever allocated for blob objects |
The total number of bytes ever freed by deallocated blob objects |
Total vBuckets (count) |
Number of vbucket deletion events |
Avg wall time (µs) spent by deleting a vbucket |
Number of failed vbucket deletion events |
Max wall time (µs) spent by deleting a vbucket |
Are vBucket mappings (key -> vBucket) checked by the server? This is a sanity checking mode which crc32 hashes the key to ensure that the client is supplying the expected vBucket for each key. |
Is Warmup of existing data enabled |
Number of keys present in access log |
The duration (in ms) after which warmup's backfill scans will yield and re-schedule; allowing other tasks on the same threads to run. |
The size of each batch loaded during warmup. |
Duplicates encountered during warmup |
Total time spent in the estimate item count phase of warmup |
Estimated number of keys in database |
Estimated number of values in database |
Number of keys warmed up |
Time (µs) spent by warming keys |
The minimum number of items that needs to be warmed up before external data mutations is allowed. This is in % of the number of items. |
Percentage of total items warmed up before we enable traffic. |
Percentage of max mem warmed up before we enable traffic. |
OOMs encountered during warmup |
The current status of the warmup thread |
Warmup thread status |
Time (µs) spent by warming data |
Number of values warmed up |
|
Item allocation size counters (in bytes) |
The number of items currently in transit (with a reference into the engine) |
The number of times an operation failed due to accessing a locked document |
The logical size of all user data on disk (per-bucket), with compression applied |
Data written as part of the KV frontend writes. |
Count of Magma compactions |
Number of items returned by iterators |
Bytes written by Magma flushes, compactions. |
Bytes written by Magma compactions due to file count triggers. |
Maximum number of connections to the interfaces marked as system only |
Maximum number of connections to the interfaces marked as user |
Engine's total memory usage |
Engine's total estimated memory usage (this is a faster stat to read, but lags mem_used as it's only updated when a threshold is crossed see mem_used_merge_threshold) |
Number of active items in memory |
The max size of the bucket |
Number of items evicted from the bucket |
Engine's total memory usage |
Number of items allocated by reusing expired objects |
Total number of items in the bucket |
The "unused" memory caused by the allocator returning bigger chunks than requested |
Memory used for various objects |
Num of async high priority requests |
Number of vBuckets |
Number of operations |
Number of operations failed due to conflict resolution |
The number bytes received from all connections bound to this bucket |
The number of connections rejected due to hitting the maximum number of connections |
Num of items rolled back |
Number of collections in the scope |
Timestamp when the stats was reset |
The memory footprint for tracing times spent processing stat requests |
The total number of bytes from the documents being returned as part of subdoc lookup operations |
The total size of all documents used for subdoc lookup operations |
The total number of bytes inserted into the documents via subdoc |
The total number of bytes for the documents mutated via subdoc |
The number of subdoc operations |
The number of times a subdoc mutation had to be retried as the document was changed by a different actor while performing the operation |
Commit duration for SyncWrites |
The number of connections to system ports in memcached |
Number of seconds the given thread has spent running according to the OS |
The number of threads used to serve clients |
The servers current time (seconds since January 1st, 1970 at 00:00:00 UTC) |
The total number of connections to this system since the process started (or reset) |
Extra memory used by transient data like persistence queue, replication queues, checkpoints, etc |
Engine's total memory usage |
The number of error messages returned |
The number of seconds elapsed since process start |
The number of connections to user ports in memcached |
Cumulative count of documents auto-deleted due to NRU ejection (Ephemeral Buckets only) |
The memory usage in bytes of the per-vBucket Bloom filters |
Total memory in checkpoints, sum of kv_vb__checkpoint_memory_overhead_bytes and kv_vb_checkpoint_memory_queue_bytes |
Checkpoints memory overhead. That is the sum of kv_vb_checkpoint_memory_overhead_index_bytes and kv_vb_checkpoint_memory_overhead_queue_bytes. |
Memory overhead in the checkpoints key-index. For every index entry, that accounts the internal structure allocation plus the key-size. |
Memory overhead in the checkpoints internal items queue. For every item in the queue, that accounts the internal structure allocation for holding the item's reference. |
Total memory of all the items queued in checkpoints. For every item in the queue, that accounts the item's key, metadata and value size. The value allocation may have shared ownership HashTable and DCP Stream readyQ. |
Count of alive (non-deleted) items in the vbucket, including non-resident items |
The memory usage in bytes of the per-VBucket Durability Monitor (in-progress SyncWrites) |
The number of items tracked in the per-VBucket Durability Monitor (in-progress SyncWrites) |
Cumulative count of the number of items whose values have been ejected for this vBucket |
Histogram of the per-VBucket tracking MFU values of items which are currently evictable |
Cumulative count of the number of items which have been expired for this vBucket |
The memory usage of items stored in the HashTable |
The memory usage of items stored in the HashTable, if the values were not compressed |
Memory overhead of the HashTable |
Number of purged tombstones (Ephemeral) |
The total size of history stored per-vBucket (the largest size reported by any vBucket) |
Memory recovered from Checkpoint by expelling clean items (i.e. items processed by all cursors) from the queue |
Amount of memory freed through ckpt removal |
Estimate of how much metadata has been written to disk since startup |
Total memory used by meta data, including the key |
The number of non-resident items |
Number of operations where an item has been flushed to disk that did not previously exist |
Number of operations where a delete has been persisted |
Number of successful front-end get operations |
Number of fatal errors in persisting a mutation (including deletion) |
Number of operations where a new version of an item has been persisted |
Percentage of items which are resident in memory |
Sum of disk queue item age in seconds |
Total drained items |
Total items enqueued on disk queues |
Total memory used by the disk queues |
Total bytes of pending writes in the disk queue |
Number of items in the disk queues |
Total number of mutations discarded during rollback |
Number of items in the sequence list |
Number of deleted items in the sequence list |
Number of tombstones purged from the sequence list |
Number of sequence numbers visible according to the sequence list read range |
Number of stale items in the sequence list |
Total bytes of stale metadata (key + fixed metadata) in the sequence list |
Total bytes of stale values in the sequence list |
Total number of aborted SyncWrites |
Total number of accepted SyncWrites (in-progress, aborted or committed) |
Total number of comitted SyncWrites |
The number bytes sent to all connections bound to this bucket |