Installing and Using the Admission Controller

The admission controller is a required component of the Couchbase Autonomous Operator and needs to be installed separately. The primary purpose of the admission controller is to validate Couchbase cluster configuration changes before the Operator acts on them, thus protecting your Couchbase deployment (and the Operator) from any accidental damage that might arise from an invalid configuration.

The admission controller replaces the cbopctl command line client that was distributed in Operator releases prior to version 1.2. Errors raised by the admission controller are exactly the same as would have been raised by cbopctl.

Overview

The Operator itself monitors CouchbaseCluster custom resources and ensures that the cluster exists in Kubernetes as specified. If a configuration error is detected at run time, it manifests as events, conditions, or log entries.

The admission controller, on the other hand, intercepts a configuration change that you make to a CouchbaseCluster resource as it’s submitted to the Kubernetes API and verifies that the resource is valid. Kubernetes and OpenShift native commands such as kubectl create, kubectl edit, and kubectl apply are all caught and handled by the admission controller.

When the admission controller detects an invalid cluster configuration, it notifies you via the same kubectl or oc command that you used to submit the resource update. The invalid resource update is never seen or processed by the Operator. This makes the Operator far simpler to use because configuration errors are reported interactively, rather than requiring you to examine cluster events and Operator logs to detect misconfigurations.

The admission controller is not just responsible for catching errors specific to the CouchbaseCluster resource. It also examines references to other resources defined in the CouchbaseCluster and ensures they exist and are valid. For example, if the cluster is deployed with TLS enabled, the admission controller checks that the private key is correctly formatted for Couchbase server, that the certificate chain validates correctly, and that the DNS Subject Alternative Names (SANs) are correctly configured for the cluster name and namespace.

Architecture

The following is a simplified illustration of how the admission controller works:

Admission Controller Architecture

A client connects to the Kubernetes API and sends a request to create a resource. The resource specification is encoded as JSON.
The API forwards the JSON to the mutating endpoint of the admission controller. A mutating webhook is responsible for altering the resource (applying default values, for example). It may optionally choose to accept or reject the create request.
The API forwards the JSON to the validating endpoint of the admission controller. A validating webhook is responsible for validating specification constraints above and beyond those offered by JSON schema validation provided by the custom resource definition. It may optionally choose to accept or reject the create request.
Once all admission checks have passed, the resource is persisted in the database (etcd).
The API responds to the client that the create request has been accepted.

If either of the admission checks in stages 2 and 3 respond that the resource is not acceptable, the API will go directly to stage 5 and return any errors returned by the admission controller.

API to Admission Controller Interface

The admission controller is implemented as a simple web server. The application layer protocol is HTTP over TLS. The admission controller is usually deployed using Kubernetes native primitives, such as a Deployment, providing high availability and fault tolerance. The admission controller is stateless so it can be deployed multiple times.

The admission controller Deployment is associated with a service account that grants the admission controller permissions to access other resources with a role. Access to resource types allows the admission controller to check that any resources, such as Secrets, are present for the Operator to access and use. It also allows the admission controller to poll for existing CouchbaseCluster resources to check for invariance of certain specification attributes. For a list of the required permissions, see RBAC Considerations

A service endpoint is exposed with a Kubernetes Service resource that provides a stable DNS name, fault tolerance, and load balancing. The service endpoint is finally bound to the Kubernetes API with MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources. The webhooks identify the resource type and version, and the types of operation to respond to. They also define the TLS CA certificate to use for validation of the service endpoint and the HTTP path to route requests to.

Platform Compatibility

The admission controller is supported on Kubernetes 1.11.0+ and OpenShift 3.11.0+.

Deploying the Admission Controller

Before you can deploy the admission controller, you must download and unpack the Couchbase Autonomous Operator package on the same computer where you normally run kubectl or oc. Contained in this package are a set of YAML files that define the admission controller.

The admission controller is only deployed once for the whole Kubernetes cluster. If the following installation steps fail, you can verify the admission controller is already installed and working by testing it.

Use the following steps to deploy the the admission controller:

Kubernetes
OpenShift

Open a Terminal window and go to the directory where you unpacked the Operator package. For example:
```
cd couchbase-autonomous-operator-kubernetes_x.x.x-linux_x86_64/
```
Run the following command to create the admission controller:
```
kubectl create -f admission.yaml
```

Confirm the admission controller has deployed successfully:

kubectl get deployments

NAME                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
couchbase-operator-admission   1         1         1            1           1m

Test the admission controller to see if it successfully rejects invalid CouchbaseCluster resources.

Open a Terminal window and go to the directory where you unpacked the Operator package. For example:
```
cd couchbase-autonomous-operator-openshift_x.x.x-linux_x86_64/
```
Log in to the system:admin account and switch to the default project.
The admission.yaml file in the Operator package contains an field that will need to manually updated before proceeding. You must define a secret in the project that grants OpenShift permission to access the RedHat container registry. You can either do this manually with the oc create secret docker-registry command or configure a service account and download the secret YAML definition directly from the RedHat container registry website. Find the REDHAT_CONTAINER_REGISTRY_PULL_SECRET token an replace it with the name of the secret you have just created.
Run the following command to create the admission controller:
```
oc create -f admission.yaml
```

Confirm the admission controller has deployed successfully:

oc get deployments

NAME                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
couchbase-operator-admission   1         1         1            1           1m

Enable webhook plug-ins

Before the admission controller can start intercepting requests made to the master API, you need to enable the mutating webhook and validating webhook plug-ins in the master configuration file of your OpenShift cluster.

Open the master configuration file for your OpenShift cluster: /etc/origin/master/master-config.yaml.

(It’s recommended that you make a backup of this file before making edits.)

Locate (or create) the the top-level admissionConfig.pluginConfig element and add the MutatingAdmissionWebhook and ValidatingAdmissionWebhook stanzas as follows:

admissionConfig:
  pluginConfig:
   MutatingAdmissionWebhook:
     configuration:
       apiVersion: v1
       disable: false
       kind: DefaultAdmissionConfig
   ValidatingAdmissionWebhook:
     configuration:
       apiVersion: v1
       disable: false
       kind: DefaultAdmissionConfig

Save your changes and close the file.
Restart the master for the changes to take effect:
```
master-restart api
```
For rpm-based installs, use the Ansible playbook to restart the api server:
```
ansible-playbook  playbooks/openshift-master/restart.yml
```
Test the admission controller to see if it successfully rejects invalid CouchbaseCluster resources.

If you encounter a ErrImagePull or ImagePullBackOff event associated with the admission controller pod, check that the container image URL in admission.yaml is reachable, and that the specified image is available.

The images that are specified in the provided YAML files are hosted in common public container registries. Couchbase regularly publishes new container images that incorporate the latest security updates and patches. In rare cases, a registry’s security policy may force the removal of older container images, causing the provided YAML definitions to become invalid. It’s recommended that you check the registry for new and discontinued container images, and update your deployment accordingly.

To use a newer container image, simply update the container image URL in admission.yaml with the new image name, and then push the definition to your Kubernetes environment. Container images are available in the following registries:

The provided YAML definitions will only work in the default namespace. In order to run in a different namespace you must first regenerate TLS certificates to contain the correct DNS Subject Alternative Name (SAN). The DNS alternate name must be in the form ${service-name}.${namespace}.svc. The admission controller certificate and key pair are defined in the admission controller secret. The admission controller certificate issuer’s CA certificate is defined in each of the webhooks. All certificates and keys are base64 encoded in the YAML definitions.

You can also choose to use the Operator Helm Chart for generating TLS certificates and installing the admission controller into custom namespaces.

Testing the Admission Controller

It’s important to check that the dynamic admission controller is working. A good way to verify functionality is to submit an invalid CouchbaseCluser configuration and see if the admission controller intercepts and rejects it.

First ensure that the CouchbaseCluster custom resource definition (CRD) is installed:

Kubernetes
OpenShift

kubectl create -f crd.yaml

oc create -f crd.yaml

Next, open the couchbase-cluster.yaml file that’s provided with the Operator package and change the bucket type in the example from couchbase to memcached. Use the edited file to create the CouchbaseCluster resource:

Kubernetes
OpenShift

kubectl create -f couchbase-cluster.yaml

oc create -f couchbase-cluster.yaml

If the admission controller isn’t correctly installed, this operation will succeed. If correctly installed, you will see something similar to the following:

Error from server: error when creating "couchbase-cluster.yaml": admission webhook "couchbase-operator-admission.default.svc" denied the request: validation failure list:
secret cb-example-auth must exist
replicas in spec.buckets[0] must be of type nil: "Bucket type is memcached"
conflictResolution in spec.buckets[0] must be of type nil: "Bucket type is memcached"
evictionPolicy in spec.buckets[0] must be of type nil: "Bucket type is memcached"
ioPriority in spec.buckets[0] must be of type nil: "Bucket type is memcached"

RBAC Considerations

The admission controller requires read-only access to several resource types in order to function.

couchbase.com/couchbaseclusters: Used by the admission controller to look for existing instances of a resource. If present, it will validate immutable attributes that have not been modified.
secrets: Used by the admission controller to look for secrets references in the CouchbaseCluster specification. It will ensure that the cluster admin username and password secrets exist. It will ensure that, if specified, the cluster TLS secrets are present and correct, and are valid for the cluster.
storage.k8s.io/storageclasses: Used by the admission controller to look for storage class references in the CouchbaseCluster specification. It will ensure that, if present, any storage class templates reference existing storage classes.

Secret and StorageClass resources are only interrogated — as described — for existence and correctness. The admission controller only performs get operations based on the names specified in the CouchbaseCluster specification. These resources will never be leaked through logs and are never persisted by the admission controller. If, however, your security policies declare that such permissions cannot be granted to an application, then they can be safely removed from the admission controller’s role. Permissions errors will be silently ignored by the admission controller. You will then no longer be informed about missing secrets and storage classes, incorrectly formatted secrets, and invalid TLS configurations.

Uninstalling the Admission Controller

Run the following command to uninstall the admission controller:

Kubernetes
OpenShift

kubectl delete -f admission.yaml

oc delete -f admission.yaml