A newer version of this documentation is available.

View Latest

cbstats

    +
    Retrieve lower level statistics with the cbstats utility, which provides deep insight into what occurs within a cluster.

    Syntax

    The basic syntax is:

    cbstats [host]: port [command] [options]

    Description

    The cbstats tool queries differently depending on the port:

    • When using the Couchbase data port 11210, it will give you operations per node per bucket:

      /opt/couchbase/bin/cbstats -b beer-sample localhost:11210 all | grep \ curr_items:
      curr_items:                                 3711
    • When using the Couchbase legacy port 11211, it will query via MOXI and provide cluster-wide stats:

      /opt/couchbase/bin/cbstats -b beer-sample localhost:11211 all | grep \ curr_items:
      curr_items:                                 7303

    If a named bucket is not provided, the server applies the setting to any default bucket that exists at the specified node.

    The tool is found in the following locations:

    Platform Location

    Linux

    /opt/couchbase/bin/cbstats

    Windows

    C:\Program Files\Couchbase\Server\bin\cbstats.exe

    Mac OS X

    /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbstats

    This following table shows top-level statistics:

    Table 1. Top-level statistics
    Stat Description

    uuid*

    The unique identifier for the bucket.

    ep_version

    Version number of ep_engine.

    ep_storage_age

    Seconds since most recently stored object was initially queued.

    ep_storage_age_highwat

    ep_storage_age high water mark.

    ep_startup_time

    System-generated engine startup time.

    ep_data_age

    Seconds since most recently stored object was modified.

    ep_data_age_highwat

    ep_data_age high water mark.

    ep_too_young

    Number of times an object was not stored due to being too young.

    ep_too_old

    Number of times an object was stored after being dirty too long.

    ep_total_enqueued

    Total number of items queued for persistence.

    ep_total_new_items

    Total number of persisted new items.

    ep_total_del_items

    Total number of persisted deletions.

    ep_total_persisted

    Total number of items persisted.

    ep_item_flush_failed

    Number of times an item failed to flush due to storage errors.

    ep_item_commit_failed

    Number of times a transaction failed to commit due to storage errors.

    ep_item_begin_failed

    Number of times a transaction failed to start due to storage errors.

    ep_expired_access

    Number of times an item was expired on application access.

    ep_expired_pager

    Number of times an item was expired by ep engine item pager.

    ep_item_flush_expired

    Number of times an item is not flushed due to the expiry of the item.

    ep_queue_size

    Number of items queued for storage.

    ep_flusher_todo

    Number of items currently being written.

    ep_flusher_state

    Current state of the flusher thread.

    ep_commit_num

    Total number of write commits.

    ep_commit_time

    Number of milliseconds of most recent commit.

    ep_commit_time_total

    Cumulative milliseconds spent committing.

    ep_vbucket_del

    Number of vbucket deletion events.

    ep_vbucket_del_fail

    Number of failed vbucket deletion events.

    ep_vbucket_del_max_walltime

    Max wall time (µs) spent by deleting a vbucket.

    ep_vbucket_del_avg_walltime

    Avg wall time (µs) spent by deleting a vbucket.

    ep_flush_duration_total

    Cumulative seconds spent flushing.

    ep_flush_all

    True if disk flush_all is scheduled.

    ep_num_ops_get_meta

    Number of getMeta operations.

    ep_num_ops_set_meta

    Number of setWithMeta operations.

    ep_num_ops_del_meta

    Number of delWithMeta operations..

    ep_num_ops_set_meta_res_failed

    Number of setWithMeta ops that failed conflict resolution.

    ep_num_ops_del_meta_res_failed

    Number of delWithMeta ops that failed conflict resolution.

    ep_num_ops_set_ret_meta

    Number of setRetMeta operations.

    ep_num_ops_del_ret_meta

    Number of delRetMeta operations.

    curr_items

    Num items in active vbuckets (temp + live).

    curr_temp_items

    Num temp items in active vbuckets.

    curr_items_tot

    Num current items including those not active (replica, dead and pending states).

    ep_kv_size

    Memory used to store item metadata, keys and values, no matter the vbucket’s state. If an item’s value is ejected, this stats will be decremented by the size of the item’s value.

    ep_value_size

    Memory used to store values for resident keys

    ep_overhead

    Extra memory used by transient data like persistence queues, replication queues, checkpoints, etc.

    ep_mem_low_wat

    Low water mark for auto-evictions.

    ep_mem_high_wat

    High water mark for auto-evictions.

    ep_total_cache_size

    The total byte size of all items, no matter the vbucket’s state, no matter if an item’s value is ejected.

    ep_oom_errors

    Number of times unrecoverable OOMs happened while processing operations.

    ep_tmp_oom_errors

    Number of times temporary OOMs happened while processing operations.

    ep_mem_tracker_enabled

    True if memory usage tracker is enabled.

    ep_bg_fetched

    Number of items fetched from disk.

    ep_bg_meta_fetched

    Number of meta items fetched from disk.

    ep_bg_remaining_jobs

    Number of remaining bg fetch jobs.

    ep_max_bg_remaining_jobs

    Max number of remaining bg fetch jobs that we have seen in the queue so far.

    ep_tap_bg_fetched

    Number of tap disk fetches.

    ep_tap_bg_fetch_requeued

    Number of times a tap bg fetch task is requeued.

    ep_num_pager_runs

    Number of times we ran pager loops to seek additional memory.

    ep_num_expiry_pager_runs

    Number of times we ran expiry pager loops to purge expired items from memory/disk.

    ep_num_access_scanner_runs

    Number of times we ran access scanner to snapshot working set.

    ep_access_scanner_num_items

    Number of items that last access scanner task swept to access log.

    ep_access_scanner_task_time

    Time of the next access scanner task (GMT).

    ep_access_scanner_last_runtime

    Number of seconds that last access scanner task took to complete.

    ep_items_rm_from_checkpoints

    Number of items removed from closed unreferenced checkpoints.

    ep_num_value_ejects

    Number of times item values got ejected from memory to disk.

    ep_num_eject_failures

    Number of items that could not be ejected.

    ep_num_not_my_vbuckets

    Number of times Not My vBucket exception happened during runtime.

    ep_tap_keepalive

    Tap keepalive time.

    ep_dbname

    DB path.

    ep_io_num_read

    Number of I/O read operations.

    ep_io_num_write

    Number of I/O write operations.

    ep_io_read_bytes

    Number of bytes read (key + values).

    ep_io_write_bytes

    Number of bytes written (key + values).

    ep_io_total_read_bytes

    Number of bytes read (total, including Couchstore B-Tree and other overheads).

    ep_io_total_write_bytes

    Number of bytes written (total, including Couchstore B-Tree and other overheads).

    ep_io_compaction_read_bytes

    Number of bytes read (compaction only, including Couchstore B-Tree and other overheads).

    ep_io_compaction_write_bytes

    Number of bytes written (compaction only, including Couchstore B-Tree and other overheads).

    ep_pending_ops

    Number of ops awaiting pending vbuckets.

    ep_pending_ops_total

    Total blocked pending ops since reset.

    ep_pending_ops_max

    Max ops seen awaiting 1 pending vbucket.

    ep_pending_ops_max_duration

    Max time (µs) used waiting on pending vbuckets.

    ep_bg_num_samples

    The number of samples included in the average.

    ep_bg_min_wait

    The shortest time (µs) in the wait queue.

    ep_bg_max_wait

    The longest time (µs) in the wait queue.

    ep_bg_wait_avg

    The average wait time (µs) for an item before it’s serviced by the dispatcher.

    ep_bg_min_load

    The shortest load time (µs).

    ep_bg_max_load

    The longest load time (µs).

    ep_bg_load_avg

    The average time (µs) for an item to be loaded from the persistence layer.

    ep_num_non_resident

    The number of non-resident items.

    ep_bg_wait

    The total elapse time for the wait queue.

    ep_bg_load

    The total elapse time for items to be loaded from the persistence layer.

    ep_allow_data_loss_during_shutdown

    Whether data loss is allowed during server shutdown.

    ep_alog_block_size

    Access log block size.

    ep_alog_path

    Path to the access log.

    ep_alog_sleep_time

    Interval between access scanner runs in minutes.

    ep_alog_task_time

    Hour in GMT time when access scanner task is scheduled to run.

    ep_backend

    The backend that is being used for data persistence.

    ep_bg_fetch_delay

    The amount of time to wait before doing a background fetch.

    ep_chk_max_items

    The number of items allowed in a checkpoint before a new one is created.

    ep_chk_period

    The maximum lifetime of a checkpoint before a new one is created.

    ep_chk_persistence_remains

    Number of remaining vbuckets for checkpoint persistence.

    ep_chk_persistence_timeout

    Timeout for vbucket checkpoint persistence.

    ep_chk_remover_stime

    The time interval for purging closed checkpoints from memory.

    ep_config_file

    The location of the ep-engine config file.

    ep_couch_bucket

    The name of this bucket.

    ep_couch_host

    The hostname that the CouchDB views server is listening on.

    ep_couch_port

    The port the CouchDB views server is listening on.

    ep_couch_reconnect_sleeptime

    The amount of time to wait before reconnecting to CouchDB.

    ep_couch_response_timeout

    Length of time to wait for a response from CouchDB before reconnecting.

    ep_data_traffic_enabled

    Whether or not data traffic is enabled for this bucket.

    ep_degraded_mode

    True if the engine is either warming up or data traffic is disabled.

    ep_exp_pager_stime

    The time interval for purging expired items from memory.

    ep_failpartialwarmup

    True if we want kill the bucket if warmup fails.

    ep_flushall_enabled

    True if this bucket enables the use of the flush_all command.

    ep_getl_default_timeout

    The default getl lock duration.

    ep_getl_max_timeout

    The maximum getl lock duration.

    ep_ht_locks

    The amount of locks per vb hashtable.

    ep_ht_size

    The initial size of each vb hashtable.

    ep_item_num_based_new_chk

    True if the number of items in the current checkpoint plays a role in a new checkpoint creation.

    ep_keep_closed_chks

    True if we want to keep the closed checkpoints for each vbucket unless the memory usage is above high water mark.

    ep_max_checkpoints

    The maximum amount of checkpoints that can be in memory per vbucket.

    ep_max_item_size

    The maximum value size.

    ep_max_size

    The maximum amount of memory this bucket can use.

    ep_max_vbuckets

    The maximum amount of vbuckets that can exist in this bucket.

    ep_mutation_mem_threshold

    The ratio of total memory available that we should start sending temp oom or oom message when hitting.

    ep_pager_active_vb_pcnt

    Active vbuckets paging percentage.

    ep_tap_ack_grace_period

    The amount of time to wait for a tap acks before disconnecting.

    ep_tap_ack_initial_sequence_number

    The initial sequence number for a tap ack when a tap stream is created.

    ep_tap_ack_interval

    The amount of messages a tap producer should send before requesting an ack.

    ep_tap_ack_window_size

    The maximum amount of ack requests that can be sent before the consumer sends a response ack. When the window is full the tap stream is paused..

    ep_tap_backfill_resident

    The resident ratio for deciding how to do backfill. If under the ratio we schedule full disk backfill. If above the ratio then we do bg fetches for non-resident items.

    ep_tap_backlog_limit

    The maximum amount of backfill items that can be in memory waiting to be sent to the tap consumer.

    ep_tap_backoff_period

    The number of seconds the tap connection.

    ep_tap_bg_fetch_requeued

    Number of times a tap bg fetch task is requeued.

    ep_tap_bg_max_pending

    The maximum number of bg jobs a tap connection may have.

    ep_tap_noop_interval

    Number of seconds between a noop is sent on an idle connection.

    ep_tap_requeue_sleep_time

    The amount of time to wait before a failed tap item is requeued.

    ep_tap_throttle_cap_pcnt

    Percentage of total items in write queue at which we throttle tap input.

    ep_tap_throttle_queue_cap

    Max size of a write queue to throttle incoming tap input.

    ep_tap_throttle_threshold

    Percentage of max mem at which we begin NAKing tap input.

    ep_uncommitted_items

    The amount of items that have not been written to disk.

    ep_vb0

    Whether vbucket 0 should be created by default.

    ep_waitforwarmup

    True if we should wait for the warmup process to complete before enabling traffic.

    ep_warmup

    Shows if warmup is enabled / disabled.

    ep_warmup_batch_size

    The size of each batch loaded during warmup.

    ep_warmup_dups

    Number of Duplicate items encountered during warmup.

    ep_warmup_min_items_threshold

    Percentage of total items warmed up before we enable traffic.

    ep_warmup_min_memory_threshold

    Percentage of max mem warmed up before we enable traffic.

    ep_warmup_oom

    The amount of oom errors that occurred during warmup.

    ep_warmup_thread

    The status of the warmup thread.

    ep_warmup_time

    The amount of time warmup took.

    • uuid is specified per bucket, therefore you must specify the bucket you are interested in. If you don’t specify a bucket (-b XXX), then you will get information for the default bucket, if it exists. If the default bucket does not exist, the output will be empty.

    The following table shows replica vbucket statistics.

    Stat Description

    vb_replica_num

    Number of replica vBuckets.

    vb_replica_curr_items

    Number of in memory items.

    vb_replica_num_non_resident

    Number of non-resident items.

    vb_replica_perc_mem_resident

    % memory resident.

    vb_replica_eject

    Number of times item values got ejected..

    vb_replica_expired

    Number of times an item was expired.

    vb_replica_ht_memory

    Memory overhead of the hashtable.

    vb_replica_itm_memory

    Total item memory.

    vb_replica_meta_data_memory

    Total metadata memory.

    vb_replica_ops_create

    Number of create operations.

    vb_replica_ops_update

    Number of update operations.

    vb_replica_ops_delete

    Number of delete operations.

    vb_replica_ops_reject

    Number of rejected operations.

    vb_replica_queue_size

    Replica items in disk queue.

    vb_replica_queue_memory

    Memory used for disk queue.

    vb_replica_queue_age

    Sum of disk queue item age in milliseconds.

    vb_replica_queue_pending

    Total bytes of pending writes.

    vb_replica_queue_fill

    Total enqueued items.

    vb_replica_queue_drain

    Total drained items.

    The following table shows active vbucket statistics:

    Stat Description

    vb_active_num

    Number of active vBuckets.

    vb_active_curr_items

    Number of in memory items.

    vb_active_num_non_resident

    Number of non-resident items.

    vb_active_perc_mem_resident

    % memory resident.

    vb_active_eject

    Number of times item values got ejected.

    vb_active_expired

    Number of times an item was expired.

    vb_active_ht_memory

    Memory overhead of the hashtable.

    vb_active_itm_memory

    Total item memory.

    vb_active_meta_data_memory

    Total metadata memory.

    vb_active_ops_create

    Number of create operations.

    vb_active_ops_update

    Number of update operations.

    vb_active_ops_delete

    Number of delete operations.

    vb_active_ops_reject

    Number of rejected operations.

    vb_active_queue_size

    Active items in disk queue.

    vb_active_queue_memory

    Memory used for disk queue.

    vb_active_queue_age

    Sum of disk queue item age in milliseconds.

    vb_active_queue_pending

    Total bytes of pending writes.

    vb_active_queue_fill

    Total enqueued items.

    vb_active_queue_drain

    Total drained items.

    The following table shows pending vbucket statistics:

    Stat Description

    vb_pending_num

    Number of pending vBuckets.

    vb_pending_curr_items

    Number of in memory items.

    vb_pending_num_non_resident

    Number of non-resident items.

    vb_pending_perc_mem_resident

    % memory resident.

    vb_pending_eject

    Number of times item values got ejected.

    vb_pending_expired

    Number of times an item was expired.

    vb_pending_ht_memory

    Memory overhead of the hashtable.

    vb_pending_itm_memory

    Total item memory.

    vb_pending_meta_data_memory

    Total metadata memory.

    vb_pending_ops_create

    Number of create operations.

    vb_pending_ops_update

    Number of update operations.

    vb_pending_ops_delete

    Number of delete operations.

    vb_pending_ops_reject

    Number of rejected operations.

    vb_pending_queue_size

    Pending items in disk queue.

    vb_pending_queue_memory

    Memory used for disk queue.

    vb_pending_queue_age

    Sum of disk queue item age in milliseconds.

    vb_pending_queue_pending

    Total bytes of pending writes.

    vb_pending_queue_fill

    Total enqueued items.

    vb_pending_queue_drain

    Total drained items.

    Options

    The majority of cbstats commands are predominantly used by Couchbase internally and to help resolve customer support incidents.

    The following are the command options:

    Options Description

    -h, --help

    Shows the help message and exits.

    -j

    Outputs the result in JSON format

    -b [BUCKETNAME]

    The bucket to get the status from. Default: default.

    -p [PASSWORD]

    The password for the bucket, if one exists.

    Examples

    Example with two buckets, default and c:

    $ cbstats localhost:12000 -b default uuid
    	uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f
    	$ cbstats localhost:12000 uuid
    	uuid: 3cd3d2efb20f3b9d7d484925b6d8c03f
    	$ cbstats localhost:12000 -b c uuid
    	uuid: 1de74b8dc0981abbc6481c5276b91be1

    Example after the default bucket was deleted, and only the c bucket was left:

    $ cbstats localhost:12000 -b default uuid
    	Authentication error for default
    	$ cbstats localhost:12000 uuid
    	<nothing>
    	$ cbstats localhost:12000 -b c uuid
    	uuid: 1de74b8dc0981abbc6481c5276b91be1

    Example for getting statistics for timings:

    To get statistics, for example, timings on host 10.5.2.117:

    cbstats 10.5.2.117:11210 timings

    Response:

     disk_commit (1024 total)
        0 - 1s : (100.00%) 1024 ###################################################
        Avg    : (     1s)
     get_stats_cmd (30663276 total)
        0 - 1us       : (  0.05%)    14827
        1us - 2us     : (  6.56%)  1995778 ##
        2us - 4us     : ( 41.79%) 10804626 ##############
        4us - 8us     : ( 45.20%)  1044043 #
        8us - 16us    : ( 45.49%)    89929
        16us - 32us   : ( 45.90%)   124472
        32us - 64us   : ( 46.38%)   148935
        64us - 128us  : ( 56.17%)  2999690 ###
        128us - 256us : ( 68.57%)  3804009 ####
        256us - 512us : ( 69.91%)   411281
        512us - 1ms   : ( 78.77%)  2717402 ###
        1ms - 2ms     : ( 96.36%)  5391526 #######
        2ms - 4ms     : ( 99.05%)   826345 #
        4ms - 8ms     : ( 99.96%)   278727
        8ms - 16ms    : (100.00%)    11443
        16ms - 32ms   : (100.00%)      217
        32ms - 65ms   : (100.00%)       19
        65ms - 131ms  : (100.00%)        7
        Avg           : (  347us)
     disk_vbstate_snapshot (93280 total)
        32us - 64us   : ( 15.34%) 14308 ######
        64us - 128us  : ( 74.74%) 55413 #########################
        128us - 256us : ( 91.39%) 15532 #######
        256us - 512us : ( 95.69%)  4007 #
        512us - 1ms   : ( 99.49%)  3546 #
        1ms - 2ms     : ( 99.95%)   423
        2ms - 4ms     : ( 99.99%)    43
        4ms - 8ms     : (100.00%)     4
        2s - 4s       : (100.00%)     4
        Avg           : (  190us)
     notify_io (4 total)
        4us - 8us   : ( 25.00%) 1 ############
        16us - 32us : ( 75.00%) 2 ########################
        32us - 64us : (100.00%) 1 ############
        Avg         : (   17us)

    Example for cbstats output:

    The cbstats output can be used with other command-line tools to sort and filter the data, for example, the watch command.

    watch --diff "cbstats \
        ip-10-12-19-81:11210 -b bucket_name -p bucket_password all | egrep 'item|mem|flusher|ep_queue|bg|eje|resi|warm'"

    Example for disk_insert:

    The following sample statistics show that disk_insert took 8–16µs 9,488 times, 16–32µs 290 times, and so on.

    STAT disk_insert_8,16 9488
    STAT disk_insert_16,32 290
    STAT disk_insert_32,64 73
    STAT disk_insert_64,128 86
    STAT disk_insert_128,256 48
    STAT disk_insert_256,512 2
    STAT disk_insert_512,1024 12
    STAT disk_insert_1024,2048 1

    The same statistics displayed by the CLI cbstats tool looks like the following:

    disk_insert (10008 total)
    8us - 16us : ( 94.80%) 9488 ###########################################
    16us - 32us : ( 97.70%) 290 #
    32us - 64us : ( 98.43%) 73
    64us - 128us : ( 99.29%) 86
    128us - 256us : ( 99.77%) 4
    256us - 512us : ( 99.79%) 2
    512us - 1ms : ( 99.91%) 12
    1ms - 2ms : ( 99.92%) 1