Rotate TLS Certificates
How-to rotate TLS certificates.
Certificates can go out of date, or the private keys of the server certificate and the signing CA can become compromised. The Operator supports Kubernetes certificate rotation in order to enable the replacement of these expired certificates or compromised keys.
The following are some examples of certificate rotation:
-
Replacement of the certificate chain and key pair (server secret)
Use when:
-
Certificates expire
-
Server or intermediate CA keys have been compromised
-
-
Replacement of the whole PKI (both Operator and server secrets)
Use when:
-
The root CA has been compromised
-
The relevant errors will be shown when a certificate is found to be invalid or compromised.
certificate cannot be verified: x509: certificate has expired or is not yet valid
certificate cannot be verified: x509: certificate signed by unknown authority
Certificate rotation is fully supported when using client certificate authentication. Due to technical reasons all TLS reconciliation occurs over plain text. This is however still secure as private keys are never exposed by the Operator over the wire. Private keys are securely mounted on Couchbase Server pods by Kubernetes. |
Replacing Server Certificates
In the event of server certificate expiry or compromise, a new certificate and key pair generated by the same CA can be used to create a new server TLS secret to be loaded onto Couchbase Server pods, replacing the existing secret. If a certificate chain is being used, it’s possible to just generate a new leaf certificate from the intermediate CA.
First, get the existing secret and direct it into a new YAML file (resource.yaml
in this example):
$ kubectl get secret couchbase-server-tls -o yaml > resource.yaml
The contents of the new file will look similar to the following:
apiVersion: v1
kind: Secret
metadata:
name: couchbase-server-tls
type: Opaque
data:
chain.pem: Q2VydGlmaWXRhO...
pkey.key: LS0tLS1CRUdJTi...
Next, replace the relevant data fields in the YAML file. To do this, start by base64-encoding the PKI files in question.
Certificate chain:
$ base64 -i chain.pem
Private key:
$ base64 -i pkey.key
Then replace the relevant fields under data
with the new base64 values:
apiVersion: v1
kind: Secret
metadata:
name: couchbase-server-tls
type: Opaque
data:
chain.pem: NhdGU6CiAgICBEY...
pkey.key: LSUlFcFFJQkFBS0N...
Finish the certificate rotation by pushing the new secret to Kubernetes:
$ kubectl replace -f resource.yaml
You can then check that the new secret has been applied:
$ kubectl get secret couchbase-server-tls -o yaml
apiVersion: v1
data:
chain.pem: NhdGU6CiAgICBEY...
pkey.key: LSUlFcFFJQkFBS0N...
kind: Secret
metadata:
creationTimestamp: "2019-01-09T09:42:35Z"
name: couchbase-server-tls
namespace: default
resourceVersion: "116380"
selfLink: /api/v1/namespaces/default/secrets/couchbase-server-tls
uid: e5111d54-13f2-11e9-baeb-0ac70bf4cf44
type: Opaque
Each pod will have a copy of the new certificate and key in a mounted inbox on each node.
Since the Operator secret (couchbase-operator-tls ) contains the root CA, there should be no need to rotate the Operator secret when a server certificate has been compromised or expired.
However, if the root CA has been compromised, you will need to replace the Operator secret, as described in the next section.
|
Replacing the entire PKI
If the root CA is compromised, a full rotation and replacement of the PKI will be required. When you replace the PKI, you’ll also need to replace the server and Operator TLS secrets with ones that include certificates and keys that align to the new CA.
Replacing the Operator secret (couchbase-operator-tls
) involves the same procedure as replacing the server secret.