Using a Certificate Manager

    +
    This tutorial demonstrates how to set up a certificate manager (cert-manager) to be used with the Operator dynamic admission controller (DAC) and managed Couchbase clusters. Certificate managers like cert-manager provide native Kubernetes support for TLS certificate generation and rotation.

    Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

    What is cert-manager?

    cert-manager is a Kubernetes controller — much like the Autonomous Operator — that defines Kubernetes resource types that represent X.509 primitives. The key resources we will examine in further depth are:

    Issuer

    This type represents a certificate issuer. This may be a root certificate authority (CA), or a signing certificate (intermediate CA). It is your responsibility to provide the certificate and private key for this resource.

    Certificate

    This type represents a certificate and private key pair. It allows the specification of the private key size and algorithm. If allows the specification of the certificate name, lifetime, usages, and any subject alternative addresses (SANs). A Certificate references, and is signed by, an Issuer.

    Certificates are initially issued by cert-manager, then at a defined time before the expiry, are automatically rotated.

    Certificates are stored in a Kubernetes Secret resource, broadly similar to the standard kubernetes.io/tls secret type. The one major difference is that, along with tls.crt and tls.key secret data, there is also a ca.crt copied in from the Issuer.

    (You may notice that this has heavily influenced the design of the Autonomous Operator’s TLS interface.)

    Before We Begin

    Before continuing with this tutorial, please ensure the following:

    • You have installed cert-manager. Follow the official installation guides at cert-manager.io.

    • You have installed Autonomous Operator 2.2 or higher. This tutorial assumes that the installed resources are present, and also leverages the cbopcfg command line tool that comes with the Autonomous Operator binary package.

      You can actually integrate/perform the steps in this tutorial as part of the process of installing the Autonomous Operator. However, for the sake of making the tutorial more straightforward, it is assumed that you’ve already performed a basic installation of the Autonomous Operator.

    Another thing to note is that all of the commands in this tutorial are run from the same, default, namespace. cert-manager runs cluster scoped, and can see Issuers and Certificates in any namespace, so you can use any namespace you desire. Before you begin the tutorial, make sure to configure your Kubernetes context to point to the namespace where you deploy Couchbase resources, or ensure you use the --namespace configuration flag with the given commands.

    Creating a Common Configuration

    The best way to explain cert-manager is by example. The following subsections describe how to create a common configuration that will be used to sign certificates for both the dynamic admission controller (DAC) and our Couchbase clusters.

    Create a Root CA

    First, we need to create a root CA. The following commands demonstrate how to create a root CA using easy-rsa:

    $ git clone https://github.com/OpenVPN/easy-rsa
    $ cd easy-rsa/easyrsa3
    $ ./easyrsa init-pki
    $ ./easyrsa build-ca nopass
    $ cat pki/private/ca.key | base64 -w 0 (1)
    $ cat pki/ca.crt | base64 -w 0 (2)
    1 The output of this command is the encoded CA key, and will be used in the next step.
    2 The output of this command is the encoded CA certificate, and will be used in the next step.

    Create an Issuer

    Next, we’ll create the CA’s Secret and Issuer. Start by creating two files — ca-secret.yaml and ca-issuer.yaml — with the following configurations:

    ca-secret.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: ca
    data:
      tls.key: LS0tLS1C... (1)
      tls.crt: LS0tLS1C... (2)
    1 This is the CA key from the previous step; textual representation has been shortened for brevity.
    2 This is the CA certificate from the previous step; textual representation has been shortened for brevity.
    ca-issuer.yaml
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ca (1)
    spec:
      ca:
        secretName: ca (2)
    1 This is the issuer name; take a note of it, as it will be referred to by Certificate resources and used to sign certificates.
    2 This is a reference to the CA secret we have just created containing the CA certificate and key pair.

    Run the following commands to create the CA’s Secret and Issuer:

    $ kubectl create -f ca-secret.yaml
    $ kubectl create -f ca-issuer.yaml

    Using cert-manager with the DAC

    The DAC component of the Autonomous Operator distribution has built-in certificate rotation detection and handling. This makes it a perfect candidate for automation.

    Uninstall the Existing DAC

    As mentioned in the prerequisites section, this tutorial assumes that you’ve already performed a basic installation of the Autonomous Operator. As part of the installation, you would have also installed the DAC. Before we can continue with the tutorial, we need to uninstall the DAC. (Don’t worry, we’ll be re-installing it soon.)

    $ cbopcfg delete admission

    Create the DAC Certificate

    We need to create a Certificate resource to issue and rotate the DAC certificate and key. Paste the following configuration into your editor of choice and save the file as admission-certificate-resource.yaml:

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: couchbase-operator-admission
    spec:
      secretName: couchbase-operator-admission (1)
      duration: 6h (2)
      renewBefore: 1h (3)
      commonName: couchbase-operator-admission
      isCA: false
      privateKey:
        algorithm: RSA
        encoding: PKCS8
        size: 2048
      usages:
      - server auth (4)
      dnsNames:
      - couchbase-operator-admission.default.svc (5)
      issuerRef:
        group: cert-manager.io
        kind: Issuer
        name: ca (6)
    1 This is the secret that will be generated containing the TLS certificate and key. The name used in this example is the same as the default one created by cbopcfg during standard installation, so we’ll just reuse it to make integration simpler.
    2 The certificate will be issued immediately and will be valid for 6 hours.
    3 The certificate will be reissued (rotated) 1 hour before the certificate expiration time.
    4 The certificate can be used as server authentication — the DAC is a HTTPS server.
    5 The certificate needs a subject alternative name that matches the network name used to connect to it. This name is comprised of <service-name>.<namespace>.svc. The cbopcfg tool usually handles this for you, however, as TLS generation is delegated to a 3rd party, we need to be aware of it. Again, like the secret name, this is the default service name created by cbopcfg, so we’ll just keep this as is.
    6 The issuer, ca, refers to the CA we defined in the section Creating a Common Configuration.

    Run the following command to create the Certificate resource:

    $ kubectl create -f admission-certificate-resource.yaml
    This command will not succeed if the DAC is already installed. Ensure that you’ve followed the instructions in the section Uninstall the Existing DAC before running this command.

    You can run the following command to check the status of the resource:

    $ kubectl get certificates

    If it was created/applied properly, the output should look like the following:

    NAME                           READY   SECRET                         AGE
    couchbase-operator-admission   True    couchbase-operator-admission   4s

    Update the DAC Configuration Settings

    When you install the Autonomous Operator, the cbopcfg tool automatically creates a default TLS configuration for you. However, we don’t have to worry about this default configuration because we removed it entirely when we uninstalled the DAC in the section Uninstall the Existing DAC. Therefore, in this next step, we will generate a brand new DAC configuration, modify it to make use of our managed certificates, and then submit it to Kubernetes to redeploy the DAC.

    Start by generating a copy of the default configuration template for the DAC:

    $ cbopcfg generate admission > admission.yaml

    Open admission.yaml with your editor of choice.

    The first thing we’re going to do is remove (delete) the Secret resource configuration from this file, as cert-manager is now managing it for us. The configuration that you need to remove looks like the following:

    apiVersion: v1
    data:
      tls-cert-file: LS0tLS1CRU...
      tls-private-key-file: LS0tLS1CRU...
    kind: Secret
    metadata:
      annotations:
        config.couchbase.com/version: 2.2.0
      creationTimestamp: null
      name: couchbase-operator-admission

    Next, locate the MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources. These need to be updated to include the signing CA’s certificate. If you recall, when we created the certificate issuer, we configured its Secret with a tls.crt key. We simply need to copy this same tls.crt key into each instance of caBundle in the webhook resources.

    apiVersion: admissionregistration.k8s.io/v1beta1
    kind: MutatingWebhookConfiguration
    metadata:
      creationTimestamp: null
      name: couchbase-operator-admission
    webhooks:
    - clientConfig:
        caBundle: LS0tLS1CRU... (1)
    1 Replace the existing key with the tls.crt key of the certificate issuer.
    apiVersion: admissionregistration.k8s.io/v1beta1
    kind: ValidatingWebhookConfiguration
    metadata:
      creationTimestamp: null
      name: couchbase-operator-admission
    webhooks:
    - clientConfig:
        caBundle: LS0tLS1CRU... (1)
    1 Replace the existing key with the tls.crt key of the certificate issuer.

    Now save all of your changes to admission.yaml and exit your editor — the configuration is ready to to be submitted to Kubernetes. Run the following command to deploy the DAC:

    $ kubectl create -f admission.yaml

    If all goes well, you can create Couchbase resources, and the DAC will reject any misconfiguration:

    $ kubectl create -f example/couchbase-cluster.yaml
    couchbasebucket.couchbase.com/default created
    Error from server: error when creating "example/couchbase-cluster.yaml": admission webhook "couchbase-operator-admission.default.svc" denied the request: validation failure list:
    secret cb-example-auth referenced by spec.security.adminSecret must exist

    However, if you see something similar to the following, it means that the Kubernetes API isn’t able to validate your server certificate:

    $ kubectl create -f example/couchbase-cluster.yaml
    Error from server (InternalError): error when creating "example/couchbase-cluster.yaml": Internal error occurred: failed calling webhook "couchbase-operator-admission.default.svc": Post "https://couchbase-operator-admission.default.svc:443/couchbaseclusters/mutate?timeout=10s": x509: certificate signed by unknown authority

    If this is the case, check that the correct CA is installed in the webhooks, and ensure that the server certificate issued by cert-manager is valid and verifies against the CA.

    Using cert-manager with a Couchbase Cluster

    The Autonomous Operator has been capable of rotating Couchbase Server certificates since version 2.0. With the 2.2 release, we introduced the ability to use new certificate formats, in particular the standard form used by cert-manager. This allows the database’s TLS configuration to be specified in code, along with security policies. The key benefits of using this are 1.) oversight — the ability to peer review configuration; and 2.) auditing — providing simple, policy driven security constraints.

    Create the Couchbase Server Certificate

    Using our common issuer, we will create a certificate for use with Couchbase Server. Paste the following configuration into your editor of choice and save the file as server-certificate-resource.yaml:

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: cb-example
    spec:
      secretName: server-tls (1)
      duration: 6h (2)
      renewBefore: 1h (3)
      commonName: couchbase-server
      isCA: false
      privateKey:
        algorithm: RSA
        encoding: PKCS8 (4)
        size: 2048
      usages:
        - server auth (5)
      dnsNames: (6)
        - "*.cb-example"
        - "*.cb-example.default"
        - "*.cb-example.default.svc"
        - "*.cb-example.default.svc.cluster.local"
        - "cb-example-srv"
        - "cb-example-srv.default"
        - "cb-example-srv.default.svc"
        - "*.cb-example-srv.default.svc.cluster.local"
        - localhost
      issuerRef:
        name: ca (7)
        group: cert-manager.io
        kind: Issuer
    1 This is the secret that will be generated containing the TLS certificate and key.
    2 The certificate will be issued immediately and will be valid for 6 hours.
    3 The certificate will be reissued (rotated) 1 hour before the certificate expiration time.
    4 Couchbase Server only works with PKCS#1 private keys, but, because we have to create a shadow copy of the certificates with hard coded names, we can also translate private keys from one format to another.
    5 The certificate can be used as server authentication.
    6 The certificate needs a set of subject alternative names (SAN) that cover all possible methods of addressing the cluster by clients. The canonical list of SANs can be found in the TLS tutorial.
    7 The issuer, ca, refers to the CA we defined in the section Creating a Common Configuration.

    Run the following command to create the Certificate resource:

    $ kubectl create -f server-certificate-resource.yaml

    You can run the following command to check the status of the resource:

    $ kubectl get certificates

    If it was created/applied properly, the output should look like the following:

    NAME                           READY   SECRET                         AGE
    cb-example                     True    server-tls                     10s
    couchbase-operator-admission   True    couchbase-operator-admission   32m

    Create the Couchbase Cluster

    Consuming certificates issued by cert-manager is fairly straightforward. Using the couchbase-cluster.yaml template from the Autonomous Operator binary package, make the following edits to the CouchbaseCluster custom resource:

    apiVersion: v1
    kind: Secret
    metadata:
      name: cb-example-auth
    type: Opaque
    data:
      username: QWRtaW5pc3RyYXRvcg==
      password: cGFzc3dvcmQ=
    ---
    apiVersion: couchbase.com/v2
    kind: CouchbaseBucket
    metadata:
      name: default
    ---
    apiVersion: couchbase.com/v2
    kind: CouchbaseCluster
    metadata:
      name: cb-example
    spec:
      image: couchbase/server:6.6.0
      security:
        adminSecret: cb-example-auth
      networking:
        tls:
          secretSource: (1)
            serverSecretName: server-tls (2)
      buckets:
        managed: true
      servers:
        - name: basic
          size: 3
          services:
            - data
            - index
            - query
    1 The secretSource TLS provider tells the Autonomous Operator that the secrets will be in kubernetes.io/tls format. This also requires a ca.crt key, which cert-manager provides automatically.
    2 The serverSecretName tells the Autonomous Operator to use the cert-manager issued secret defined in the previous section [create-the-Couchbase-server-certificate].

    Run the following command to create the Certificate resource:

    $ kubectl create -f couchbase-cluster.yaml

    You can run the following command to check the status of the deployment:

    $ kubectl get cbc

    If the Couchbase cluster was deployed successfully, the output should look like the following:

    NAME         VERSION   SIZE   STATUS      UUID                               AGE
    cb-example   6.6.0     3      Available   8c531b1cc11680c286d04616cc7a8185   3m31s

    Summary

    By embracing the kubernetes.io/tls format, the Autonomous Operator is able to draw upon great 3rd-party tools, like cert-manager, that offer simplicity, security, and regulation within your Kubernetes environment.