Using a Certificate Manager
This tutorial demonstrates how to set up a certificate manager (cert-manager) to be used with the Operator dynamic admission controller (DAC) and managed Couchbase clusters. Certificate managers like cert-manager provide native Kubernetes support for TLS certificate generation and rotation.
Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer. |
What is cert-manager?
cert-manager is a Kubernetes controller — much like the Autonomous Operator — that defines Kubernetes resource types that represent X.509 primitives. The key resources we will examine in further depth are:
- Issuer
-
This type represents a certificate issuer. This may be a root certificate authority (CA), or a signing certificate (intermediate CA). It is your responsibility to provide the certificate and private key for this resource.
- Certificate
-
This type represents a certificate and private key pair. It allows the specification of the private key size and algorithm. If allows the specification of the certificate name, lifetime, usages, and any subject alternative addresses (SANs). A
Certificate
references, and is signed by, anIssuer
.
Certificates are initially issued by cert-manager, then at a defined time before the expiry, are automatically rotated.
Certificates are stored in a Kubernetes Secret
resource, broadly similar to the standard kubernetes.io/tls
secret type.
The one major difference is that, along with tls.crt
and tls.key
secret data, there is also a ca.crt
copied in from the Issuer
.
(You may notice that this has heavily influenced the design of the Autonomous Operator’s TLS interface.)
Before We Begin
Before continuing with this tutorial, please ensure the following:
-
You have installed cert-manager. Follow the official installation guides at
cert-manager.io
. -
You have installed Autonomous Operator 2.2 or higher. This tutorial assumes that the installed resources are present, and also leverages the
cao
command line tool that comes with the Autonomous Operator binary package.You can actually integrate/perform the steps in this tutorial as part of the process of installing the Autonomous Operator. However, for the sake of making the tutorial more straightforward, it is assumed that you’ve already performed a basic installation of the Autonomous Operator.
Another thing to note is that all of the commands in this tutorial are run from the same, default, namespace.
cert-manager runs cluster scoped, and can see Issuers and Certificates in any namespace, so you can use any namespace you desire.
Before you begin the tutorial, make sure to configure your Kubernetes context to point to the namespace where you deploy Couchbase resources, or ensure you use the --namespace
configuration flag with the given commands.
Creating a Common Configuration
The best way to explain cert-manager is by example. The following subsections describe how to create a common configuration that will be used to sign certificates for both the dynamic admission controller (DAC) and our Couchbase clusters.
Create a Root CA
First, we need to create a root CA.
The following commands demonstrate how to create a root CA using easy-rsa
:
$ git clone https://github.com/OpenVPN/easy-rsa
$ cd easy-rsa/easyrsa3
$ ./easyrsa init-pki
$ ./easyrsa build-ca nopass
$ cat pki/private/ca.key | base64 -w 0 (1)
$ cat pki/ca.crt | base64 -w 0 (2)
1 | The output of this command is the encoded CA key, and will be used in the next step. |
2 | The output of this command is the encoded CA certificate, and will be used in the next step. |
Create an Issuer
Next, we’ll create the CA’s Secret
and Issuer
.
Start by creating two files — ca-secret.yaml
and ca-issuer.yaml
— with the following configurations:
apiVersion: v1
kind: Secret
metadata:
name: ca
data:
tls.key: LS0tLS1C... (1)
tls.crt: LS0tLS1C... (2)
1 | This is the CA key from the previous step; textual representation has been shortened for brevity. |
2 | This is the CA certificate from the previous step; textual representation has been shortened for brevity. |
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: ca (1)
spec:
ca:
secretName: ca (2)
1 | This is the issuer name; take a note of it, as it will be referred to by Certificate resources and used to sign certificates. |
2 | This is a reference to the CA secret we have just created containing the CA certificate and key pair. |
Run the following commands to create the CA’s Secret
and Issuer
:
$ kubectl create -f ca-secret.yaml
$ kubectl create -f ca-issuer.yaml
Using cert-manager with the DAC
The DAC component of the Autonomous Operator distribution has built-in certificate rotation detection and handling. This makes it a perfect candidate for automation.
Uninstall the Existing DAC
As mentioned in the prerequisites section, this tutorial assumes that you’ve already performed a basic installation of the Autonomous Operator. As part of the installation, you would have also installed the DAC. Before we can continue with the tutorial, we need to uninstall the DAC. (Don’t worry, we’ll be re-installing it soon.)
$ cao delete admission
Create the DAC Certificate
We need to create a Certificate
resource to issue and rotate the DAC certificate and key.
Paste the following configuration into your editor of choice and save the file as admission-certificate-resource.yaml
:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: couchbase-operator-admission
spec:
secretName: couchbase-operator-admission (1)
duration: 6h (2)
renewBefore: 1h (3)
commonName: couchbase-operator-admission
isCA: false
privateKey:
algorithm: RSA
encoding: PKCS8
size: 2048
usages:
- server auth (4)
dnsNames:
- couchbase-operator-admission.default.svc (5)
issuerRef:
group: cert-manager.io
kind: Issuer
name: ca (6)
1 | This is the secret that will be generated containing the TLS certificate and key.
The name used in this example is the same as the default one created by cao during standard installation, so we’ll just reuse it to make integration simpler. |
2 | The certificate will be issued immediately and will be valid for 6 hours. |
3 | The certificate will be reissued (rotated) 1 hour before the certificate expiration time. |
4 | The certificate can be used as server authentication — the DAC is a HTTPS server. |
5 | The certificate needs a subject alternative name that matches the network name used to connect to it.
This name is comprised of <service-name>.<namespace>.svc .
The cao tool usually handles this for you, however, as TLS generation is delegated to a 3rd party, we need to be aware of it.
Again, like the secret name, this is the default service name created by cao , so we’ll just keep this as is. |
6 | The issuer, ca , refers to the CA we defined in the section Creating a Common Configuration. |
Run the following command to create the Certificate
resource:
$ kubectl create -f admission-certificate-resource.yaml
This command will not succeed if the DAC is already installed. Ensure that you’ve followed the instructions in the section Uninstall the Existing DAC before running this command. |
You can run the following command to check the status of the resource:
$ kubectl get certificates
If it was created/applied properly, the output should look like the following:
NAME READY SECRET AGE
couchbase-operator-admission True couchbase-operator-admission 4s
Update the DAC Configuration Settings
When you install the Autonomous Operator, the cao
tool automatically creates a default TLS configuration for you.
However, we don’t have to worry about this default configuration because we removed it entirely when we uninstalled the DAC in the section Uninstall the Existing DAC.
Therefore, in this next step, we will generate a brand new DAC configuration, modify it to make use of our managed certificates, and then submit it to Kubernetes to redeploy the DAC.
Start by generating a copy of the default configuration template for the DAC:
$ cao generate admission > admission.yaml
Open admission.yaml
with your editor of choice.
The first thing we’re going to do is remove (delete) the Secret
resource configuration from this file, as cert-manager is now managing it for us.
The configuration that you need to remove looks like the following:
apiVersion: v1
data:
tls-cert-file: LS0tLS1CRU...
tls-private-key-file: LS0tLS1CRU...
kind: Secret
metadata:
annotations:
config.couchbase.com/version: 2.2.0
creationTimestamp: null
name: couchbase-operator-admission
Next, locate the ValidatingWebhookConfiguration
resource.
This needs to be updated to include the signing CA’s certificate.
If you recall, when we created the certificate issuer, we configured its Secret
with a tls.crt
key.
We simply need to copy this same tls.crt
key into each instance of caBundle
in the webhook resources.
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
creationTimestamp: null
name: couchbase-operator-admission
webhooks:
- clientConfig:
caBundle: LS0tLS1CRU... (1)
1 | Replace the existing key with the tls.crt key of the certificate issuer. |
Now save all of your changes to admission.yaml
and exit your editor — the configuration is ready to to be submitted to Kubernetes.
Run the following command to deploy the DAC:
$ kubectl create -f admission.yaml
If all goes well, you can create Couchbase resources, and the DAC will reject any misconfiguration:
$ kubectl create -f example/couchbase-cluster.yaml
couchbasebucket.couchbase.com/default created
Error from server: error when creating "example/couchbase-cluster.yaml": admission webhook "couchbase-operator-admission.default.svc" denied the request: validation failure list:
secret cb-example-auth referenced by spec.security.adminSecret must exist
However, if you see something similar to the following, it means that the Kubernetes API isn’t able to validate your server certificate:
$ kubectl create -f example/couchbase-cluster.yaml
Error from server (InternalError): error when creating "example/couchbase-cluster.yaml": Internal error occurred: failed calling webhook "couchbase-operator-admission.default.svc": Post "https://couchbase-operator-admission.default.svc:443/couchbaseclusters/validate?timeout=10s": x509: certificate signed by unknown authority
If this is the case, check that the correct CA is installed in the webhooks, and ensure that the server certificate issued by cert-manager is valid and verifies against the CA.
Using cert-manager with a Couchbase Cluster
The Autonomous Operator has been capable of rotating Couchbase Server certificates since version 2.0. With the 2.2 release, we introduced the ability to use new certificate formats, in particular the standard form used by cert-manager. This allows the database’s TLS configuration to be specified in code, along with security policies. The key benefits of using this are 1.) oversight — the ability to peer review configuration; and 2.) auditing — providing simple, policy driven security constraints.
Create the Couchbase Server Certificate
Using our common issuer, we will create a certificate for use with Couchbase Server.
Paste the following configuration into your editor of choice and save the file as server-certificate-resource.yaml
:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cb-example
spec:
secretName: server-tls (1)
duration: 6h (2)
renewBefore: 1h (3)
commonName: couchbase-server
isCA: false
privateKey:
algorithm: RSA
encoding: PKCS8 (4)
size: 2048
usages:
- server auth (5)
dnsNames: (6)
- "*.cb-example"
- "*.cb-example.default"
- "*.cb-example.default.svc"
- "*.cb-example.default.svc.cluster.local"
- "cb-example-srv"
- "cb-example-srv.default"
- "cb-example-srv.default.svc"
- "*.cb-example-srv.default.svc.cluster.local"
- localhost
issuerRef:
name: ca (7)
group: cert-manager.io
kind: Issuer
1 | This is the secret that will be generated containing the TLS certificate and key. |
2 | The certificate will be issued immediately and will be valid for 6 hours. |
3 | The certificate will be reissued (rotated) 1 hour before the certificate expiration time. |
4 | Couchbase Server only works with PKCS#1 private keys, but, because we have to create a shadow copy of the certificates with hard coded names, we can also translate private keys from one format to another. |
5 | The certificate can be used as server authentication. |
6 | The certificate needs a set of subject alternative names (SAN) that cover all possible methods of addressing the cluster by clients. The canonical list of SANs can be found in the TLS tutorial. |
7 | The issuer, ca , refers to the CA we defined in the section Creating a Common Configuration. |
Run the following command to create the Certificate
resource:
$ kubectl create -f server-certificate-resource.yaml
You can run the following command to check the status of the resource:
$ kubectl get certificates
If it was created/applied properly, the output should look like the following:
NAME READY SECRET AGE
cb-example True server-tls 10s
couchbase-operator-admission True couchbase-operator-admission 32m
Create the Couchbase Cluster
Consuming certificates issued by cert-manager is fairly straightforward.
Using the couchbase-cluster.yaml
template from the Autonomous Operator binary package, make the following edits to the CouchbaseCluster
custom resource:
apiVersion: v1
kind: Secret
metadata:
name: cb-example-auth
type: Opaque
data:
username: QWRtaW5pc3RyYXRvcg==
password: cGFzc3dvcmQ=
---
apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
name: default
---
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: cb-example
spec:
image: couchbase/server:6.6.0
security:
adminSecret: cb-example-auth
networking:
tls:
secretSource: (1)
serverSecretName: server-tls (2)
buckets:
managed: true
servers:
- name: basic
size: 3
services:
- data
- index
- query
1 | The secretSource TLS provider tells the Autonomous Operator that the secrets will be in kubernetes.io/tls format.
This also requires a ca.crt key, which cert-manager provides automatically. |
2 | The serverSecretName tells the Autonomous Operator to use the cert-manager issued secret defined in the previous section [create-the-Couchbase-server-certificate]. |
Run the following command to create the Certificate
resource:
$ kubectl create -f couchbase-cluster.yaml
You can run the following command to check the status of the deployment:
$ kubectl get cbc
If the Couchbase cluster was deployed successfully, the output should look like the following:
NAME VERSION SIZE STATUS UUID AGE
cb-example 6.6.0 3 Available 8c531b1cc11680c286d04616cc7a8185 3m31s