Quick Start with Prometheus Monitoring

      +
      Enable and setup Prometheus Monitoring for the Couchbase Autonomous Operator.

      Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

      This guide walks through recommended procedures for enabling and configuring Prometheus monitoring of the Couchbase Autonomous Operator.

      Prerequisites

      Clone the kube-prometheus repository from GitHub, but do not create any manifests just yet.

      $ git clone https://github.com/coreos/kube-prometheus

      Make sure you have a Kubernetes cluster running the Autonomous Operator with monitoring enabled and follow the Prerequisites section in the kube-prometheus documentation.

      Manifests Setup

      Currently, the kube-prometheus project includes a folder called manifests that includes all the resources necessary to run the Prometheus Operator. The Prometheus Operator creates our Prometheus deployment which scrapes endpoints continuously for Prometheus metrics. We will be creating these manifests in a later step.

      The Autonomous Operator, with monitoring enabled, exposes the Couchbase Prometheus metrics on the sidecar containers running in each Couchbase Server pod. Our task is then to get Prometheus to discover and scrape these endpoints in order to monitor the overall cluster through the Prometheus UI and with custom Grafana dashboards.

      In order for our Prometheus deployment to recognize and scrape Couchbase endpoints, we need to create a Couchbase specific service monitor, and a Couchbase metrics specific service.

      This tutorial works on the basis that the manifests which bring up the relevant resources for Prometheus Operator are still located in the folder manifests. Please adjust accordingly if changes have been made since as the repository is experimental and subject to change.

      Create the Couchbase Metrics Service

      The Couchbase Metrics Service will define the set of pods we want to monitor and the port to scrape on each.

      apiVersion: v1
      kind: Service
      metadata:
        name: couchbase-metrics
        namespace: default (1)
        labels:
          app: couchbase
      spec:
        ports:
        - name: metrics
          port: 9091 (2)
          protocol: TCP
        selector:
          app: couchbase
          couchbase_cluster: cb-example (3)
      1 Make sure that the Service is in the same namespace as the Couchbase Cluster that you wish to scrape metrics from, otherwise no pods will be selected and no endpoints will be displayed on the Prometheus targets page (at http://localhost:9090/targets).
      2 Keep this port as its default value of 9091 as this is the default port the Couchbase Exporter will be writing to.
      3 app:couchbase will select all Couchbase server pods in the specified namespace regardless of which cluster they belong to. Adding the couchbase_cluster: cb-example selector will constrain the label selector to a specific cluster.

      Copy this YAML into a file named, for example, couchbase-service.yaml and make sure it is placed in the manifests directory.

      Create the Couchbase ServiceMonitor

      The ServiceMonitor tells Prometheus to monitor the Service resource we just defined which then enables Prometheus to scrape for metrics provided by the Couchbase exporter.

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        name: couchbase
        namespace: default (1)
        labels:
          app: couchbase
      spec:
        endpoints:
        - port: metrics       (2)
          interval: 5s        (3)
          bearerTokenSecret:  (4)
            key: token (5)
            name: cb-metrics-token (6)
        namespaceSelector:
          matchNames:
          - default (7)
        selector:
          matchLabels:
            app: couchbase (8)
      1 You may wish to include our Couchbase ServiceMonitor in the monitoring namespace along with the other ServiceMonitors included in the provided kube-prometheus manifests. For examples of this tutorial we have just left it in the default namespace for ease of use.
      2 The port may be a literal port number as defined in the couchbase-metrics service, or may reference the port in the same service by name.
      3 interval tells Prometheus how often to scrape the endpoint.
      4 If it is specified in the CouchbaseCluster spec, by spec.monitoring.prometheus.authorizationSecret, we then need to provide the correct Secret details in order for Prometheus to be able to gain access to the scrape endpoints. Please see Configure Prometheus Metrics Collection on how to create said Secret.
      5 Pass in the value of the key field in the secret data. In this example the key value is token if we have the corresponding Secret with value {data: {token: base64encodedTokenHere}}.
      6 name is simply the name of your Secret containing the bearer token.
      7 Here we want to match the namespace of the Couchbase Metrics Service we have just created.
      8 Similar to the namespaceSelector, this is a simple labelSelector that will look for our Service through label discovery.

      Copy this YAML into a file named couchbase-serviceMonitor.yaml for example, and save it in the manifests directory. If we were to just run with this ServiceMonitor and without any defined Service we would see on our Prometheus targets page a section for Couchbase endpoints but none showing.

      Create the Manifests

      Follow the specific commands given in the kube-prometheus documentation to bring up our created resources along with the other provided default manifests.

      Components such as Prometheus, Alertmanager, Node Exporter and Grafana should then startup and we can confirm this by inspecting the pods in the namespace monitoring.

      • Kubernetes

      • OpenShift

      $ kubectl get pods -n monitoring
      $ oc get pods -n monitoring

      Check that our ServiceMonitor and Service have been created.

      • Kubernetes

      • OpenShift

      $ kubectl get servicemonitor couchbase
      $ kubectl get service couchbase-metrics
      $ oc get servicemonitor couchbase
      $ oc get service couchbase-metrics

      To check that all is working correctly with the Prometheus Operator deployment, run the following command to view the logs:

      • Kubernetes

      • OpenShift

      $ kubectl logs -f deployments/prometheus-operator -n monitoring prometheus-operator
      $ oc logs -f deployments/prometheus-operator -n monitoring prometheus-operator

      Any potential issues that may arise should be fairly straight forward to debug and understand. One you may run into is the ServiceMonitor reporting problems due to your bearer token Secret not being created.

      Once all pods are ready and running, in order to access Prometheus and Grafana, follow the relevant steps in the kube-prometheus documentation.

      Working with the Prometheus Operator Helm Chart

      It is also possible to instead use the Prometheus Operator Helm chart which closely matches the kube-prometheus project. To do so we must make Prometheus aware of the additional Service Monitor to monitor our Couchbase metrics service.

      We use a YAML file to include in the values argument in helm install which will introduce additional configuration into the Helm Chart. The following example YAML is almost an exact replica of the Service Monitor YAML example in the earlier section, but with a few important differences. These differences exist due to the way that the Helm Chart structure has been defined. For further information see the defined values.yaml in the Helm chart GitHub repository.

      prometheus:
        prometheusSpec:
          additionalServiceMonitors:
            - name: couchbase
              additionalLabels:
                - app: couchbase
              endpoints:
                - port: metrics
                  interval: 5s
                   bearerTokenFile:
                     key: token
                     name: cb-metrics-token
              namespaceSelector:
                matchNames:
                  - default
              selector:
                matchLabels:
                  app: couchbase

      Once this YAML file is created and saved, we can then deploy the Prometheus Operator with Helm using the following command.

      helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace

      To make sure the Helm chart will be deployed with the additional configuration, you can add the argument --dry-run to the command to view the intended configuration.

      helm install prometheus-operator stable/prometheus-operator --values your-helm-config.yaml -n your-namespace --dry-run

      If you wish to deploy the Prometheus Operator in a separate namespace, make sure to create that namespace if it does not already exist.

      • Kubernetes

      • OpenShift

      $ kubectl create namespace your-namespace
      $ oc create namespace your-namespace

      Now all we need to do is make sure our Couchbase service and Service Monitor are deployed for the Prometheus Operator to recognize. However the Prometheus Operator Helm chart deployment requires an extra bit of configuration to the Service Monitor for the Service Monitor to show up in Prometheus. The Prometheus Operator has a labelSelector defined looking for the label release: prometheus-operator so we need to add this to our Couchbase Service Monitor. With this label added the Service Monitor will look something like the following.

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        name: couchbase
        namespace: default
        labels:
          app: couchbase
          release: prometheus-operator
      spec:
        endpoints:
        - port: metrics
          interval: 5s
          bearerTokenSecret:
            key: token
            name: cb-metrics-token
        namespaceSelector:
          matchNames:
          - default
        selector:
          matchLabels:
            app: couchbase

      Deploy this slightly modified ServiceMonitor and the Couchbase metrics service.

      We can now use commands similar to the ones found in the kube-prometheus documentation, to access Prometheus Server and Grafana.

      • Kubernetes

      • OpenShift

      $ kubectl port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090
      $ kubectl port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000
      $ oc port-forward -n your-namespace pod/prometheus-prometheus-operator-prometheus-0 9090:9090
      $ oc port-forward -n your-namespace deployment/prometheus-operator-grafana 3000:3000

      An important distinction however is that the Grafana default password will be different. At time of writing the default password is prom-operator rather than admin as it is in kube-prometheus. The default password can be discovered in the prometheus-operator-grafana secret through base64 decoding.