Couchbase supports transport layer security (TLS) in order to encrypt communications on the wire and provide mutual authentication between peers. Couchbase clients (including the Couchbase Autonomous Operator) require the usage of TLS in order to communicate with the Couchbase cluster.
The basic requirements are:
A certificate authority (CA) certificate which will be used by all actors to validate that peer certificates have been digitally signed by a trusted CA.
A server certificate/key pair for all nodes in the Couchbase cluster. If you are using a hierarchy of intermediate CAs, these must be appended to the client certificate, ending in the intermediate CA that is signed by the top-level CA. Server certificates must have a subject alternative name (SAN) set so that a client can assert that the host name that it’s connecting to is the same as that in the certificate.
Couchbase currently supports using wildcard entries only. For example,
*.cluster.domain.svcwhere cluster is the name of the CouchbaseCluster resource, and domain is the namespace that the cluster is running in (typically
TLS certificates are installed as part of the pod creation process, and cannot be enabled on an existing cluster.
An example configuration is as follows:
spec: ... tls: static: member: serverSecret: couchbase-server-tls operatorSecret: couchbase-operator-tls ...
For a static TLS configuration,
operatorSecret need to be specified. These refer to named Kubernetes secrets which are described below.
Secrets are specified in the CouchbaseCluster resource, therefore they may have any name you choose. The format of individual secrets is discussed below.
Server secrets need to be mounted as a volume within the Couchbase Server pod with specific names. The certificate chain must be named
chain.pem and the private key
kubectl create secret generic couchbase-server-tls \ --from-file example/tls/certs/chain.pem \ --from-file example/tls/certs/pkey.key
The Operator client secrets are read directly from the API. It expects only a single value to be present;
ca.crt is the top-level CA which is used to authenticate all TLS server certificate chains.
kubectl create secret generic couchbase-operator-tls \ --from-file example/tls/certs/ca.crt
Creating X.509 certificates is beyond the scope of this documentation and is given only for illustrative purposes only.
EasyRSA by OpenVPN makes operating a public key infrastructure (PKI) relatively simple, and is the recommended method to get up and running quickly.
First clone the repository:
git clone https://github.com/OpenVPN/easy-rsa
Initialize and create the CA certificate/key. You will be prompted for a private key password and the CA common name (CN), something like Couchbase CA is sufficient. The CA certificate will be available as
Create a server wildcard certificate and key to be used on Couchbase Server pods. The Operator/clients will access the pods via Kubernetes services (
cb-example-0000.cb-example.default.svc, for example) so this needs to be in the SAN list in order for a client to verify the certificate belongs to the host that is being connected to. To this end, we add in a
--subject-alt-name, this can be specified multiple times in case your client uses a different method of addressing. The key/certificate pair can be found in
pki/issued/couchbase-server.crt and used as
chain.pem, respectively, in the
./easyrsa --subject-alt-name=DNS:*.cb-example.default.svc build-server-full couchbase-server nopass
Note, password-protected keys are not supported by Couchbase Server or the Operator.
Due to an issue with Couchbase Server’s private key handling, server keys need to be PKCS#1 formatted. This can be achieved with the following commands:
openssl rsa -in pkey.key -out pkey.key.der -outform DER
openssl rsa -in pkey.key.der -inform DER -out pkey.key -outform PEM
Certificates can go out of date, or the private keys of the server certificate and the signing CA can become compromised. The Operator supports Kubernetes certificate rotation in order to enable the replacement of these expired certificates or compromised keys.
The following are some examples of certificate rotation:
Replacement of the certificate chain and key pair (server secret)
Server or intermediate CA keys have been compromised
Replacement of the whole PKI (both Operator and server secrets)
The root CA has been compromised
The relevant errors will be shown when a certificate is found to be invalid or compromised.
certificate cannot be verified: x509: certificate has expired or is not yet valid
certificate cannot be verified: x509: certificate signed by unknown authority
In the event of server certificate expiry or compromise, a new certificate and key pair generated by the same CA can be used to create a new server TLS secret to be loaded onto Couchbase Server pods, replacing the existing secret. If a certificate chain is being used, it’s possible to just generate a new leaf certificate from the intermediate CA.
First, get the existing secret and direct it into a new YAML file (
resource.yaml in this example):
kubectl get secret couchbase-server-tls -o yaml > resource.yaml
The contents of the new file will look similar to the following:
apiVersion: v1 kind: Secret metadata: name: couchbase-server-tls type: Opaque data: chain.pem: Q2VydGlmaWXRhO... pkey.key: LS0tLS1CRUdJTi...
Next, replace the relevant data fields in the YAML file. To do this, start by base64-encoding the PKI files in question.
base64 -i chain.pem
base64 -i pkey.key
Then replace the relevant fields under
data with the new base64 values:
apiVersion: v1 kind: Secret metadata: name: couchbase-server-tls type: Opaque data: chain.pem: NhdGU6CiAgICBEY... pkey.key: LSUlFcFFJQkFBS0N...
Finish the certificate rotation by pushing the new secret to Kubernetes:
kubectl replace -f resource.yaml
You can then check that the new secret has been applied:
kubectl get secret couchbase-server-tls -o yaml
apiVersion: v1 data: chain.pem: NhdGU6CiAgICBEY... pkey.key: LSUlFcFFJQkFBS0N... kind: Secret metadata: creationTimestamp: "2019-01-09T09:42:35Z" name: couchbase-server-tls namespace: default resourceVersion: "116380" selfLink: /api/v1/namespaces/default/secrets/couchbase-server-tls uid: e5111d54-13f2-11e9-baeb-0ac70bf4cf44 type: Opaque
Each pod will have a copy of the new certificate and key in a mounted inbox on each node.
Since the Operator secret (
If the root CA is compromised, a full rotation and replacement of the PKI will be required. When you replace the PKI, you’ll also need to replace the server and Operator TLS secrets with ones that include certificates and keys that align to the new CA.
Replacing the Operator secret (
couchbase-operator-tls) involves the same procedure as replacing the server secret.