Quick Start with Prometheus Monitoring
Enable and setup Prometheus Monitoring for the Couchbase Autonomous Operator.
Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.
This guide walks through recommended procedures for enabling and configuring Prometheus monitoring of the Couchbase Autonomous Operator.
Clone the kube-prometheus repository from GitHub, but do not create any manifests just yet.
$ git clone https://github.com/coreos/kube-prometheus
Make sure you have a Kubernetes cluster running the Autonomous Operator with monitoring enabled and follow the Prerequisites section in the
kube-prometheus project includes a folder called
manifests that includes all the resources necessary to
run the Prometheus Operator. The Prometheus Operator creates our Prometheus deployment which scrapes endpoints continuously for Prometheus metrics.
We will be creating these manifests in a later step.
The Autonomous Operator, with monitoring enabled, exposes the Couchbase Prometheus metrics on the sidecar containers running in each Couchbase Server pod. Our task is then to get Prometheus to discover and scrape these endpoints in order to monitor the overall cluster through the Prometheus UI and with custom Grafana dashboards.
In order for our Prometheus deployment to recognise and scrape Couchbase endpoints, we need to create a Couchbase specific ServiceMonitor, and a Couchbase metrics specific service.
This tutorial works on the basis that the manifests which bring up the relevant resources for Prometheus Operator are still located in the folder
The Couchbase Metrics
Service will define the set of pods we want to monitor and the port to scrape on each.
apiVersion: v1 kind: Service metadata: name: couchbase-metrics namespace: default (1) labels: app: couchbase spec: ports: - name: metrics port: 9091 (2) protocol: TCP selector: app: couchbase couchbase_cluster: cb-example (3)
|1||Make sure that the
|2||Keep this port as its default value of 9091 as this is the default port the Couchbase Exporter will be writing to.|
Copy this YAML into a file named, for example,
couchbase-service.yaml and make sure it is placed in the
ServiceMonitor tells Prometheus to monitor the
Service resource we just defined which then enables Prometheus to scrape for metrics provided by the couchbase-exporter.
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: couchbase namespace: default (1) labels: app: couchbase spec: endpoints: - port: metrics (2) interval: 5s (3) bearerTokenSecret: (4) key: token (5) name: cb-metrics-token (6) namespaceSelector: matchNames: - default (7) selector: matchLabels: app: couchbase (8)
|1||You may wish to include our Couchbase
|4||If it is specified in the CouchbaseCluster spec, by
|5||Pass in the value of the
|7||Here we want to match the namespace of the Couchbase Metrics
|8||Similar to the
Copy this YAML into a file named
couchbase-serviceMonitor.yaml for example, and save it in the
If we were to just run with this
ServiceMonitor and without any defined
Service we would see on our Prometheus targets page a section for couchbase endpoints but none showing.
Follow the specific commands given in the GitHub README to bring up our created resources along with the other provided default manifests.
Components such as Prometheus, AlertManager, NodeExporter and Grafana should then startup and we can confirm this by inspecting the pods in the namespace
Check that our
Service have been created.
$ kubectl get servicemonitor couchbase $ kubectl get service couchbase-metrics
$ oc get servicemonitor couchbase $ oc get service couchbase-metrics
To check that all is working correctly with the Prometheus Operator deployment, run the following command to view the logs:
$ kubectl logs -f deployments/prometheus-operator -n monitoring prometheus-operator
$ oc logs -f deployments/prometheus-operator -n monitoring prometheus-operator
Any potential issues that may arise should be fairly straight forward to debug and understand.
One you may run into is the
ServiceMonitor reporting problems due to your bearer token Secret not being created.
Once all pods are ready and running, in order to access Prometheus and Grafana, follow the relevant steps in the GitHub README.
It is also possible to instead use the Prometheus Operator Helm chart which closely matches the kube-prometheus project. To do so we must make Prometheus aware of the additional Service Monitor to monitor our Couchbase metrics service.
We use a YAML file to include in the
values argument in
helm install which will introduce additional configuration into the Helm Chart.
The following example YAML is almost an exact replica of the Service Monitor YAML example in the earlier section, but with a few important differences.
These differences exist due to the way that the Helm Chart structure has been defined.
For further information see the defined
values.yaml in the Helm chart GitHub repository.
prometheus: prometheusSpec: additionalServiceMonitors: - name: couchbase additionalLabels: - app: couchbase endpoints: - port: metrics interval: 5s bearerTokenFile: key: token name: cb-metrics-token namespaceSelector: matchNames: - default selector: matchLabels: app: couchbase
Once this YAML file is created and saved, we can then deploy the Prometheus Operator with Helm using the following command.
helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace
To make sure the Helm chart will be deployed with the additional configuration, you can add the argument
--dry-run to the command to view the intended configuration.
helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace --dry-run
If you wish to deploy the Prometheus Operator in a separate namespace, make sure to create that namespace if it does not already exist.
Now all we need to do is make sure our Couchbase service and Service Monitor are deployed for the Prometheus Operator to recognise.
However the Prometheus Operator Helm chart deployment requires an extra bit of configuration to the Service Monitor for the Service Monitor to show up in Prometheus.
The Prometheus Operator has a
labelSelector defined looking for the label
release: prometheus-operator so we need to add this to our Couchbase Service Monitor.
With this label added the Service Monitor will look something like the following.
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: couchbase namespace: default labels: app: couchbase release: prometheus-operator spec: endpoints: - port: metrics interval: 5s bearerTokenSecret: key: token name: cb-metrics-token namespaceSelector: matchNames: - default selector: matchLabels: app: couchbase
Deploy this slightly modified
ServiceMonitor and the Couchbase metrics service.
We can now use commands similar to the ones found in the kube-prometheus README, to access Prometheus Server and Grafana.
$ kubectl port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090 $ kubectl port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000
$ oc port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090 $ oc port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000
An important distinction however is that the Grafana default password will be different.
At time of writing the default password is
prom-operator rather than
admin as it is in
The default password can be discovered in the
prometheus-operator-grafana secret through base64 decoding.