Configuring TLS

Couchbase supports transport layer security (TLS) in order to encrypt communications on the wire and provide mutual authentication between peers. Couchbase clients (including the Couchbase Autonomous Operator) require the usage of TLS in order to communicate with the Couchbase cluster.

The basic requirements are:

A certificate authority (CA) certificate which will be used by all actors to validate that peer certificates have been digitally signed by a trusted CA.
A server certificate/key pair for all nodes in the Couchbase cluster. If you are using a hierarchy of intermediate CAs, these must be appended to the client certificate, ending in the intermediate CA that is signed by the top-level CA. Server certificates must have a subject alternative name (SAN) set so that a client can assert that the host name that it’s connecting to is the same as that in the certificate.

Couchbase currently supports using wildcard entries only. For example, *.cluster.domain.svc where cluster is the name of the CouchbaseCluster resource, and domain is the namespace that the cluster is running in (typically default).

TLS certificates are installed as part of the pod creation process, and cannot be enabled on an existing cluster.

Configuration

An example configuration is as follows:

spec:
  ...
  tls:
    static:
      member:
        serverSecret: couchbase-server-tls
      operatorSecret: couchbase-operator-tls
  ...

For a static TLS configuration, serverSecret and operatorSecret need to be specified. These refer to named Kubernetes secrets which are described below.

Creating Secrets

Secrets are specified in the CouchbaseCluster resource, therefore they may have any name you choose. The format of individual secrets is discussed below.

spec.tls.static.member.serverSecret

Server secrets need to be mounted as a volume within the Couchbase Server pod with specific names. The certificate chain must be named chain.pem and the private key pkey.key.

kubectl create secret generic couchbase-server-tls \
  --from-file example/tls/certs/chain.pem \
  --from-file example/tls/certs/pkey.key

spec.tls.static.operatorSecret

The Operator client secrets are read directly from the API. It expects only a single value to be present; ca.crt is the top-level CA which is used to authenticate all TLS server certificate chains.

kubectl create secret generic couchbase-operator-tls \
  --from-file example/tls/certs/ca.crt

Creating Certificates

Creating X.509 certificates is beyond the scope of this documentation and is given only for illustrative purposes only.

EasyRSA

EasyRSA by OpenVPN makes operating a public key infrastructure (PKI) relatively simple, and is the recommended method to get up and running quickly.

First clone the repository:

git clone https://github.com/OpenVPN/easy-rsa

Initialize and create the CA certificate/key. You will be prompted for a private key password and the CA common name (CN), something like Couchbase CA is sufficient. The CA certificate will be available as pki/ca.crt.

cd easy-rsa/easyrsa3

./easyrsa init-pki

./easyrsa build-ca

Create a server wildcard certificate and key to be used on Couchbase Server pods. The Operator/clients will access the pods via Kubernetes services (cb-example-0000.cb-example.default.svc, for example) so this needs to be in the SAN list in order for a client to verify the certificate belongs to the host that is being connected to. To this end, we add in a --subject-alt-name, this can be specified multiple times in case your client uses a different method of addressing. The key/certificate pair can be found in pki/private/couchbase-server.key and pki/issued/couchbase-server.crt and used as pkey.pem and chain.pem, respectively, in the serverSecret.

./easyrsa --subject-alt-name=DNS:*.cb-example.default.svc build-server-full couchbase-server nopass

Note, password-protected keys are not supported by Couchbase Server or the Operator.

Private Key Formatting

Due to an issue with Couchbase Server’s private key handling, server keys need to be PKCS#1 formatted. This can be achieved with the following commands:

openssl rsa -in pkey.key -out pkey.key.der -outform DER

openssl rsa -in pkey.key.der -inform DER -out pkey.key -outform PEM

Certificate Rotation

Certificates can go out of date, or the private keys of the server certificate and the signing CA can become compromised. The Operator supports Kubernetes certificate rotation in order to enable the replacement of these expired certificates or compromised keys.

The following are some examples of certificate rotation:

Replacement of the certificate chain and key pair (server secret)

Use when:
- Certificates expire
- Server or intermediate CA keys have been compromised
Replacement of the whole PKI (both Operator and server secrets)

Use when:
- The root CA has been compromised

The relevant errors will be shown when a certificate is found to be invalid or compromised.

Expired certificate

certificate cannot be verified: x509: certificate has expired or is not yet valid

Compromised certificate

certificate cannot be verified: x509: certificate signed by unknown authority

Replacing Server Certificates

In the event of server certificate expiry or compromise, a new certificate and key pair generated by the same CA can be used to create a new server TLS secret to be loaded onto Couchbase Server pods, replacing the existing secret. If a certificate chain is being used, it’s possible to just generate a new leaf certificate from the intermediate CA.

First, get the existing secret and direct it into a new YAML file (resource.yaml in this example):

kubectl get secret couchbase-server-tls -o yaml > resource.yaml

The contents of the new file will look similar to the following:

apiVersion: v1
kind: Secret
metadata:
  name: couchbase-server-tls
type: Opaque
data:
  chain.pem: Q2VydGlmaWXRhO...
  pkey.key: LS0tLS1CRUdJTi...

Next, replace the relevant data fields in the YAML file. To do this, start by base64-encoding the PKI files in question.

Certificate chain:

base64 -i chain.pem

Private key:

base64 -i pkey.key

Then replace the relevant fields under data with the new base64 values:

apiVersion: v1
kind: Secret
metadata:
  name: couchbase-server-tls
type: Opaque
data:
  chain.pem: NhdGU6CiAgICBEY...
  pkey.key: LSUlFcFFJQkFBS0N...

Finish the certificate rotation by pushing the new secret to Kubernetes:

kubectl replace -f resource.yaml

You can then check that the new secret has been applied:

kubectl get secret couchbase-server-tls -o yaml

apiVersion: v1
data:
  chain.pem: NhdGU6CiAgICBEY...
  pkey.key: LSUlFcFFJQkFBS0N...
kind: Secret
metadata:
  creationTimestamp: "2019-01-09T09:42:35Z"
  name: couchbase-server-tls
  namespace: default
  resourceVersion: "116380"
  selfLink: /api/v1/namespaces/default/secrets/couchbase-server-tls
  uid: e5111d54-13f2-11e9-baeb-0ac70bf4cf44
type: Opaque

Each pod will have a copy of the new certificate and key in a mounted inbox on each node.

Since the Operator secret (couchbase-operator-tls) contains the root CA, there should be no need to rotate the Operator secret when a server certificate has been compromised or expired. However, if the root CA has been compromised, you will need to replace the Operator secret, as described in the next section.

Replacing the entire PKI

If the root CA is compromised, a full rotation and replacement of the PKI will be required. When you replace the PKI, you’ll also need to replace the server and Operator TLS secrets with ones that include certificates and keys that align to the new CA.

Replacing the Operator secret (couchbase-operator-tls) involves the same procedure as replacing the server secret.