Logs and Troubleshooting
This section provides information about how to diagnose and troubleshoot problems with the Couchbase Operator or your deployment.
When troubleshooting the Couchbase Operator, it is important to rule out Kubernetes itself as the root cause of the problem you are experiencing. See the Kubernetes Troubleshooting Guide for information about debugging applications within a Kubernetes cluster.
The following topics are also helpful to understand when troubleshooting the Operator:
Full Deployment Logs
The Operator is distributed with a support tool which can automatically collect resources, logs and events from the Kubernetes cluster for use in support cases. It is also capable of collecting logs from Couchbase server instances via cbcollect_info. Please see the documentation.
Operator Logs
The Couchbase Operator generates logs that can help troubleshoot your deployment. Using kubectl
or oc
, you can choose to print the Operator logs to stdout
.
On Kubernetes:
# Get name of operator pod
$ kubectl get po -lapp=couchbase-operator
NAME READY STATUS RESTARTS AGE
couchbase-operator-1917615544-h20bm 1/1 Running 0 20h
# Get logs
$ kubectl logs couchbase-operator-1917615544-h20bm
time="2018-01-23T22:56:34Z" level=info msg="Obtaining resource lock" module=main
time="2018-01-23T22:56:34Z" level=info msg="Starting event recorder" module=main
time="2018-01-23T22:56:34Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
time="2018-01-23T22:56:51Z" level=info msg="I'm the leader, attempt to start the operator" module=main
time="2018-01-23T22:56:51Z" level=info msg="Creating the couchbase-operator controller" module=main
On OpenShift:
# Get name of operator pod
$ oc get po -lapp=couchbase-operator
Watch for the following messages which indicate that the Operator is unable to reconcile your cluster into a desired state:
-
Logs with
level=error
-
Operator is unable to get cluster state after N retries
Profiling the operator
The Couchbase operator serves profiling data on it’s default listenAddress localhost:8080
. You can access this endpoint by performing a kubectl exec
into the operator pod, for example:
$ kubectl exec -it couchbase-operator-599bcf47f-8wswh sh
$ wget -O- "localhost:8080/debug/pprof/trace" | less
Another option is to create a custom service or forward the service directly to your localhost.
$ kubectl port-forward couchbase-operator-599bcf47f-8wswh 8080:8080
$ go tool pprof localhost:8080/debug/pprof/heap
$ (pprof) traces
See Also
Refer to the Couchbase Server Troubleshooting guide for additional information about reporting issues.