Manage Autonomous Operator Logging
The Autonomous Operator generates logs that can be used for auditing and troubleshooting purposes.
This page describes logging that is specific to the Autonomous Operator itself. For information about Couchbase cluster logging, refer to Manage Couchbase Server Logging.
Overview
The Autonomous Operator generates logs that include information about itself and the various other Kubernetes components that make up the Operator deployment. These logs are distinct from the logs that are generated by the Couchbase Server application.
This page provides information about how to collect and scrutinize logging information that is produced by the Autonomous Operator. When troubleshooting the Autonomous Operator, it is important to first rule out Kubernetes itself as the root cause of the problem. The Kubernetes Troubleshooting Guide contains a great deal of helpful information about debugging applications within a Kubernetes cluster.
Familiarity with the Operator’s configuration settings can be helpful when troubleshooting the Autonomous Operator.
Collecting Autonomous Operator Logs
Using kubectl
or oc
, you can choose to print the Autonomous Operator logs to to standard console output.
-
Kubernetes
-
OpenShift
Start by getting the name of the Autonomous Operator pod.
$ kubectl get po -lapp=couchbase-operator
NAME READY STATUS RESTARTS AGE couchbase-operator-1917615544-h20bm 1/1 Running 0 20h
Use the pod name to get the logs.
$ kubectl logs couchbase-operator-1917615544-h20bm
time="2018-01-23T22:56:34Z" level=info msg="couchbase-operator v1.1.0 (release)" module=main time="2018-01-23T22:56:34Z" level=info msg="Obtaining resource lock" module=main time="2018-01-23T22:56:34Z" level=info msg="Starting event recorder" module=main time="2018-01-23T22:56:34Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main time="2018-01-23T22:56:51Z" level=info msg="I'm the leader, attempt to start the operator" module=main time="2018-01-23T22:56:51Z" level=info msg="Creating the couchbase-operator controller" module=main
Alternatively, you can specify the Autonomous Operator deployment to get the logs.
$ kubectl logs deployment/couchbase-operator
Since there is only one instance of the Autonomous Operator in the deployment, the the underlying command will automatically select the correct pod and print the logs.
Start by getting the name of the Autonomous Operator pod.
$ oc get po -lapp=couchbase-operator
NAME READY STATUS RESTARTS AGE couchbase-operator-1917615544-h20bm 1/1 Running 0 20h
Use the pod name to get the logs.
$ oc logs couchbase-operator-1917615544-h20bm
time="2018-01-23T22:56:34Z" level=info msg="couchbase-operator v1.1.0 (release)" module=main time="2018-01-23T22:56:34Z" level=info msg="Obtaining resource lock" module=main time="2018-01-23T22:56:34Z" level=info msg="Starting event recorder" module=main time="2018-01-23T22:56:34Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main time="2018-01-23T22:56:51Z" level=info msg="I'm the leader, attempt to start the operator" module=main time="2018-01-23T22:56:51Z" level=info msg="Creating the couchbase-operator controller" module=main
Alternatively, you can specify the Autonomous Operator deployment to get the logs.
$ oc logs deployment/couchbase-operator
Since there is only one instance of the Autonomous Operator in the deployment, the the underlying command will automatically select the correct pod and print the logs.
If you’re troubleshooting the Autonomous Operator, watch for the following messages which indicate that the Operator is unable to reconcile a Couchbase cluster into a desired state:
-
Logs with
level=error
-
Operator is unable to get cluster state after N retries
Profiling the Autonomous Operator
For more advanced troubleshooting, the Autonomous Operator supports the Go language pprof feature and serves profiling data on its default listen address localhost:8080
.
You can access this endpoint by running a remote shell or forwarding the port to your local system.
-
Kubernetes
-
OpenShift
To access goroutine stack traces using a shell:
$ kubectl exec -it couchbase-operator-599bcf47f-8wswh sh
$ wget -O- 'http://localhost:8080/debug/pprof/goroutine?debug=1' | less
To access Go memory usage using a port forward:
$ kubectl port-forward couchbase-operator-599bcf47f-8wswh 8080:8080
$ go tool pprof localhost:8080/debug/pprof/heap
(pprof) traces
To access goroutine stack traces using a shell:
$ oc exec -it couchbase-operator-599bcf47f-8wswh sh
$ wget -O- 'http://localhost:8080/debug/pprof/goroutine?debug=1' | less
To access Go memory usage using a port forward:
$ oc port-forward couchbase-operator-599bcf47f-8wswh 8080:8080
$ go tool pprof localhost:8080/debug/pprof/heap
(pprof) traces