Quick Start with Prometheus Monitoring

Enable and setup Prometheus Monitoring for the Couchbase Autonomous Operator.

Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

This guide walks through recommended procedures for enabling and configuring Prometheus monitoring of the Couchbase Autonomous Operator.

Prerequisites

Clone the kube-prometheus repository from GitHub, but do not create any manifests just yet.

$ git clone https://github.com/coreos/kube-prometheus

Make sure you have a Kubernetes cluster running the Autonomous Operator with monitoring enabled and follow the Prerequisites section in the kube-prometheus README.

Manifests Setup

Currently, the kube-prometheus project includes a folder called manifests that includes all the resources necessary to run the Prometheus Operator. The Prometheus Operator creates our Prometheus deployment which scrapes endpoints continuously for Prometheus metrics. We will be creating these manifests in a later step.

The Autonomous Operator, with monitoring enabled, exposes the Couchbase Prometheus metrics on the sidecar containers running in each Couchbase Server pod. Our task is then to get Prometheus to discover and scrape these endpoints in order to monitor the overall cluster through the Prometheus UI and with custom Grafana dashboards.

In order for our Prometheus deployment to recognise and scrape Couchbase endpoints, we need to create a Couchbase specific ServiceMonitor, and a Couchbase metrics specific service.

This tutorial works on the basis that the manifests which bring up the relevant resources for Prometheus Operator are still located in the folder manifests. Please adjust accordingly if changes have been made since as the repository is experimental and subject to change.

Create the Couchbase Metrics Service

The Couchbase Metrics Service will define the set of pods we want to monitor and the port to scrape on each.

apiVersion: v1
kind: Service
metadata:
  name: couchbase-metrics
  namespace: default (1)
  labels:
    app: couchbase
spec:
  ports:
  - name: metrics
    port: 9091 (2)
    protocol: TCP
  selector:
    app: couchbase
    couchbase_cluster: cb-example (3)
1 Make sure that the Service is in the same namespace as the Couchbase Cluster that you wish to scrape metrics from, otherwise no pods will be selected and no endpoints will be displayed on the Prometheus targets page (at http://localhost:9090/targets).
2 Keep this port as its default value of 9091 as this is the default port the Couchbase Exporter will be writing to.
3 app:couchbase will select all couchbase server pods in the specified namespace regardless of which cluster they belong to. Adding the `couchbase_cluster: cb-example" selector will constrain the LabelSelector to a specific cluster.

Copy this YAML into a file named, for example, couchbase-service.yaml and make sure it is placed in the manifests directory.

Create the Couchbase ServiceMonitor

The ServiceMonitor tells Prometheus to monitor the Service resource we just defined which then enables Prometheus to scrape for metrics provided by the couchbase-exporter.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: couchbase
  namespace: default (1)
  labels:
    app: couchbase
spec:
  endpoints:
  - port: metrics       (2)
    interval: 5s        (3)
    bearerTokenSecret:  (4)
      key: token (5)
      name: cb-metrics-token (6)
  namespaceSelector:
    matchNames:
    - default (7)
  selector:
    matchLabels:
      app: couchbase (8)
1 You may wish to include our Couchbase ServiceMonitor in the monitoring namespace along with the other ServiceMonitors included in the provided kube-prometheus manifests. For examples of this tutorial we have just left it in the default namespace for ease of use.
2 The port may be a literal port number as defined in the couchbase-metrics service, or may reference the port in the same service by name.
3 interval tells Prometheus how often to scrape the endpoint.
4 If it is specified in the CouchbaseCluster spec, by spec.monitoring.prometheus.authorizationSecret, we then need to provide the correct Secret details in order for Prometheus to be able to gain access to the scrape endpoints. Please see Configure Prometheus Metrics Collection on how to create said Secret.
5 Pass in the value of the key field in the secret data. In this example the key value is token if we have the corresponding Secret with value {data: {token: base64encodedTokenHere}}.
6 name is simply the name of your Secret containing the bearer token.
7 Here we want to match the namespace of the Couchbase Metrics Service we have just created.
8 Similar to the namespaceSelector, this is a simple labelSelector that will look for our Service through label discovery.

Copy this YAML into a file named couchbase-serviceMonitor.yaml for example, and save it in the manifests directory. If we were to just run with this ServiceMonitor and without any defined Service we would see on our Prometheus targets page a section for couchbase endpoints but none showing.

Create the Manifests

Follow the specific commands given in the GitHub README to bring up our created resources along with the other provided default manifests.

Components such as Prometheus, AlertManager, NodeExporter and Grafana should then startup and we can confirm this by inspecting the pods in the namespace monitoring.

  • Kubernetes

  • OpenShift

$ kubectl get pods -n monitoring
$ oc get pods -n monitoring

Check that our ServiceMonitor and Service have been created.

  • Kubernetes

  • OpenShift

$ kubectl get servicemonitor couchbase
$ kubectl get service couchbase-metrics
$ oc get servicemonitor couchbase
$ oc get service couchbase-metrics

To check that all is working correctly with the Prometheus Operator deployment, run the following command to view the logs:

  • Kubernetes

  • OpenShift

$ kubectl logs -f deployments/prometheus-operator -n monitoring prometheus-operator
$ oc logs -f deployments/prometheus-operator -n monitoring prometheus-operator

Any potential issues that may arise should be fairly straight forward to debug and understand. One you may run into is the ServiceMonitor reporting problems due to your bearer token Secret not being created.

Once all pods are ready and running, in order to access Prometheus and Grafana, follow the relevant steps in the GitHub README.

Working with the Prometheus Operator Helm Chart

It is also possible to instead use the Prometheus Operator Helm chart which closely matches the kube-prometheus project. To do so we must make Prometheus aware of the additional Service Monitor to monitor our Couchbase metrics service.

We use a YAML file to include in the values argument in helm install which will introduce additional configuration into the Helm Chart. The following example YAML is almost an exact replica of the Service Monitor YAML example in the earlier section, but with a few important differences. These differences exist due to the way that the Helm Chart structure has been defined. For further information see the defined values.yaml in the Helm chart GitHub repository.

prometheus:
  prometheusSpec:
    additionalServiceMonitors:
      - name: couchbase
        additionalLabels:
          - app: couchbase
        endpoints:
          - port: metrics
            interval: 5s
             bearerTokenFile:
               key: token
               name: cb-metrics-token
        namespaceSelector:
          matchNames:
            - default
        selector:
          matchLabels:
            app: couchbase

Once this YAML file is created and saved, we can then deploy the Prometheus Operator with Helm using the following command.

helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace

To make sure the Helm chart will be deployed with the additional configuration, you can add the argument --dry-run to the command to view the intended configuration.

helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace --dry-run

If you wish to deploy the Prometheus Operator in a separate namespace, make sure to create that namespace if it does not already exist.

  • Kubernetes

  • OpenShift

$ kubectl create namespace your-namespace
$ oc create namespace your-namespace

Now all we need to do is make sure our Couchbase service and Service Monitor are deployed for the Prometheus Operator to recognise. However the Prometheus Operator Helm chart deployment requires an extra bit of configuration to the Service Monitor for the Service Monitor to show up in Prometheus. The Prometheus Operator has a labelSelector defined looking for the label release: prometheus-operator so we need to add this to our Couchbase Service Monitor. With this label added the Service Monitor will look something like the following.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: couchbase
  namespace: default
  labels:
    app: couchbase
    release: prometheus-operator
spec:
  endpoints:
  - port: metrics
    interval: 5s
    bearerTokenSecret:
      key: token
      name: cb-metrics-token
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: couchbase

Deploy this slightly modified ServiceMonitor and the Couchbase metrics service.

We can now use commands similar to the ones found in the kube-prometheus README, to access Prometheus Server and Grafana.

  • Kubernetes

  • OpenShift

$ kubectl port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090
$ kubectl port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000
$ oc port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090
$ oc port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000

An important distinction however is that the Grafana default password will be different. At time of writing the default password is prom-operator rather than admin as it is in kube-prometheus. The default password can be discovered in the prometheus-operator-grafana secret through base64 decoding.