CouchbaseCluster Status

Kubernetes users often need to check the status of various objects (such as pods, deployments, and StatefulSets) that have been deployed inside their Kubernetes clusters. Status checking is done by describing one of the objects in the cluster using kubectl describe command.

The kubectl describe command outputs the configuration for the specified object, a status section, and an event section. The status section varies depending on the object type and shows various health metrics related to the object described. The events sections is printed last and shows major events that have happened during the life of the object.

Since the Couchbase Operator registers a CouchbaseCluster Custom Resource Definition (CRD) with Kubernetes, it allows Kubernetes to know about Couchbase clusters natively. This means that the CouchbaseCluster becomes another object that can be described to get the configuration, status, and events specific to a particular Couchbase cluster.

To describe a Couchbase cluster named cb-example, run the following command:

On Kubernetes:

kubectl describe couchbasecluster cb-example

Note that the object name in the above command is couchbasecluster. You can also use the shorthand name cbc as the object name to save some typing.

On Kubernetes:

kubectl describe cbc cb-example

On OpenShift:

oc describe cbc cb-example

Below is an example of the output of the kubectl describe command. The various parts of the output and their significance will be discussed in detail in the following sections.

Name:		cb-example
Kind:		CouchbaseCluster
Spec:   ...
Status:
  Admin Console Port:		30239
  Admin Console Port SSL:	31628
  Buckets:
    Default:
      Conflict Resolution:	seqno
      Enable Flush:		true
      Enable Index Replica:	false
      Eviction Policy:		fullEviction
      Io Priority:		high
      Memory Quota:		128
      Name:			default
      Replicas:			1
      Type:			couchbase
  Cluster Id:			9ae7fe4634e0360cf9c9245cb4ebb27b
  Conditions:
    Balanced:
      Message:			Data is equally distributed across all nodes in the cluster
      Reason:			Cluster is balanced
      Status:			True
    ...
  Control Paused:		false
  Current Version:		enterprise-6.0.1
  Members:
    Ready:
      cb-example-0000
      cb-example-0001
      cb-example-0002
  Phase:		Running
  Reason:
  Size:			3
  Events:
  FirstSeen	LastSeen	Count	From			Type		Reason			Message
  ---------	--------	-----	----			--------	------			-------
  21m		21m		1	couchbase-operator	Normal		NewMemberAdded		New member cb-example-0000 added to cluster

Status And Events

Often when the status spec of the schema is updated, an associated event is generated which details the reason why the change occurred. For instance, increasing the size of the cluster causes an add member event to appear in the Events section of the status result. The following section explains the information provided in the status response, along with any events associated with value updates.

Admin Console

Admin Console Port: 30239
Admin Console Port SSL:	31628

These ports are used for exposing the Couchbase cluster’s web console. This section of the status is only visible when the exposeAdminConsole setting of the cluster spec is enabled. See admin-console-access.adoc for information on how to expose and access the Couchbase Server Web Console.

Enabling and disabling the exposeAdminConsole setting produces the following events, respectively:

Events:
  ...  ServiceCreated		Service for admin console `cb-example-ui` was created
  ...  ServiceDeleted		Service for admin console `cb-example-ui` was deleted

Buckets

Buckets:
  Default:
    Memory Quota:		128
    Name:			default
    ...

A list of buckets currently active within the Couchbase cluster. The values of the bucket status are the same as the values provided in the bucket specification. See CouchbaseCluster Configuration for details about the bucket specification. When buckets from the specification are added, updated, or removed the following events are generated, respectively:

Events:
  ... BucketCreated		A new bucket `default` was created
  ... BucketEdited		Bucket `default` was edited
  ... BucketDeleted		Bucket `default` was deleted

The bucket section of the status only reflects valid changes that have been made to the cluster. In the case of attempts to update the bucket section with an invalid specification, the following message will be seen in the Conditions section of the Status:

Conditions:
  ManageBuckets:
    Message:	Bucket: default cannot change (default) bucket param='conflictResolution' from 'seqno' to 'timestamp'
    Reason:		Bucket edit failed
    Status:		False

To resolve errors resulting from invalid updates, edit the cluster specification according to the message shown under Conditions and apply the updated specification.

Cluster ID

Cluster Id:			9ae7fe4634e0360cf9c9245cb4ebb27b

The cluster identifier as provided by the Couchbase cluster. This value is provided directly by the Couchbase cluster and is therefore never updated by any changes to the cluster specification.

Members

Members:
  Ready:
    cb-example-0000
    cb-example-0001
    cb-example-0002

Members represent pods that are managed by the Couchbase Operator. All members in the Ready section represent pods that make up the Couchbase cluster. The status of cluster members is directly affected by the value of Size in the cluster spec. When Size is increased, a member add event is generated, followed by rebalance:

Events:
  ... NewMemberAdded		New member cb-example-0003 added to cluster
  ... RebalanceStarted	A rebalance has been started to balance data across the cluster

Removing a member generates the expected removal event, followed by a rebalance:

Events:
  ... RebalanceStarted	A rebalance has been started to balance data across the cluster
  ... MemberRemoved		Existing member cb-example-0003 removed from the cluster
It is also possible for a MemberRemoved event to be generated when an auto-failover occurs and a member is replaced by a new member.

The Conditions section of the status is also updated as the operator works to resolve changes to cluster spec size or failures. Important conditions to check when scaling are:

  • Balanced: Denoting whether data is equally distributed across all nodes in the cluster.

  • Available: Denoting whether all members are up and all vBuckets are available.

  • Scaling: Denoting whether the cluster is currently scaling.

See Conditions and Events for more information about these conditions and their statuses.

Phase

Phase:		Running

The current phase of the cluster displayed as one of the following phases:

  • Creating: When a cluster is first deployed.

  • Running: After the couchbasecluster resource object is available and the first orchestrator pod is running.

  • Failed: When a failure occurs within the creation or running phase of the cluster.

  • None: Initial state denoted by an empty string.

Size

Size:			3

The size of the Couchbase cluster. When you change this value in the cluster specification, this value is also updated as members are added to the Couchbase cluster. If an error occurs while the Operator is attempting to conform the cluster to the desired size, check the Conditions section for information about the current state of the cluster. See admin-console-access.adoc for information on how to manually access the cluster and collect logs when additional troubleshooting is needed.

Conditions

Conditions:
  Balanced:
    Message:			Data is equally distributed across all nodes in the cluster
    Reason:			Cluster is balanced
    Status:			True
  ...

A list of conditions reflecting the current state of the Couchbase cluster. Each Condition item is denoted by a Type along with an associated Status. The various conditions and their statuses are documented in Conditions and Events.

Events

Events:
  29m		29m		1	couchbase-operator-1917615544-j1mg8			Normal		NewMemberAdded		New member cb-example-0003 added to cluster
  29m		29m		1	couchbase-operator-1917615544-j1mg8			Normal		RebalanceStarted	A rebalance has been started to balance data across the cluster
  51m		51m		1	couchbase-operator-1917615544-j1mg8			Normal		BucketEdited		Bucket `default` was edited

Events generated during cluster reconciliation. The last 10 events are recorded and timestamped as the Operator works to reconcile the cluster to its desired cluster state. The types of events that can occur throughout the lifecycle of a CouchbaseCluster are documented in Conditions and Events.