Kubernetes Network Policies Using Deny-All Default

The Autonomous Operator and Couchbase Server can be used with Kubernetes network policies although this is not officially supported currently.

Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

Overview

Refer to the concepts page on Kubernetes networking for an introduction to network policies and the rules required for Couchbase.

Network policies should work with Couchbase Autonomous Operator deployments but are currently unsupported officially. This information is provided to document a functional set up for those that need it prior to official support. The assumption is standard Kubernetes configuration is used rather than any specific to a particular network plugin.

This example uses the Calico network plugin to show how to enable network policies on a local development cluster. It shows examples of how to configure the various information required, actual deployments may differ.

No rules are set up for any external traffic for Couchbase SDKs, helm or kubectl usage.

We use the Helm chart here only to simplify the examples.

Cluster setup

The Kubernetes cluster needs to be configured to support network policies with an appropriate network plugin. Refer to the official documentation for details on this.

As an example, the following script will create a Kubernetes-in-Docker (KIND) cluster locally with two Kubernetes nodes.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true    # disable kindnet
  podSubnet: 192.168.0.0/16  # set to Calico's default subnet
nodes:
  - role: control-plane
  - role: worker
EOF

Once we have our Kubernetes cluster available, we can deploy the Calico network plugin to it which will manage the network policies for us.

$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
$ kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=true
$ kubectl -n kube-system set env daemonset/calico-node FELIX_XDPENABLED=false

The example above is purely for demonstration purposes and is not guaranteed to be functional or recommended to be used in production.

Couchbase Dynamic Admission Controller

The typical and recommended deployment is a single cluster-wide dynamic admission controller (DAC) for Couchbase. For the purposes of the example we therefore deploy the DAC to the default namespace with no network policies applied.

$ helm upgrade --install couchbase-dac couchbase/couchbase-operator --set install.couchbaseCluster=false,install.couchbaseOperator=false --namespace default --wait

Couchbase Autonomous Operator

We will need to supply the Kubernetes API server details along with the DAC endpoint used by the operator to network policy rules. We also need to supply the Kubernetes namespace to use, here we simply use test as the name.

$ export API_SERVER_IP=$(kubectl get endpoints --namespace default kubernetes --output=json|jq '.subsets[0].addresses[0].ip' -r)
$ export API_SERVER_PORT=$(kubectl get endpoints --namespace default kubernetes --output=json|jq '.subsets[0].ports[0].port' -r)
$ export NAMESPACE=test

The variables above will be used for substitution in the following steps.

To provide a functional network policy we need to enable the rules specified in the Kubernetes networking page.

---
apiVersion: v1
kind: Namespace
metadata:
  name: $NAMESPACE
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-except-dns
  namespace: $NAMESPACE
  # Prevent any traffic for all pods apart from that we specifically allow.
  # This is taken from the Kubernetes documentation with a tweak to allow DNS.
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  egress:
  # Allow DNS resolution, primarily for Couchbase pods but it triggers other
  # hard to debug connection issues sometimes too so be aware if this is moved to
  # the later rules.
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: couchbase-operator-policy
  namespace: $NAMESPACE
  # The operator needs to talk to server pods in its namespace,
  # plus the K8S API and the DAC admission hook.
  #
  # If the operator is in another namespace (cluster wide) then the rules
  # will need a namespace selector to allow that.
  # Remember that any rules need to match on both ends to allow traffic, e.g.
  # if you have a deny-all default and only allow traffic out of the operator
  # but not in on the server pods then it will be prevented.
spec:
  # Select operator pods
  podSelector:
    matchLabels:
      app.kubernetes.io/name: couchbase-operator
  # No ingress required as traffic is always out to other components.
  policyTypes:
    - Egress
  egress:
  # Allow all traffic to Couchbase Server pods
  - to:
    - podSelector:
        matchLabels:
          app: couchbase
  # Allow traffic to the K8S API server IP and port only
  - ports:
    - port: $API_SERVER_PORT
      protocol: TCP
    to:
    - ipBlock:
        cidr: $API_SERVER_IP/32
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: couchbase-namespace-policy
  namespace: $NAMESPACE
  # Couchbase server pods need to talk to each other in the same namespace.
  # They also need to allow the operator to control them.
spec:
  # Select all Couchbase Server pods
  podSelector:
    matchLabels:
      app: couchbase
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow all traffic from other Couchbase Server pods in the same namespace,
    - from:
      - podSelector:
          matchLabels:
            app: couchbase
    # Allow all traffic from the operator.
    - from:
      - podSelector:
          matchLabels:
            app.kubernetes.io/name: couchbase-operator
  egress:
    # Allow all traffic to other Couchbase Server pods in the same namespace.
    - to:
      - podSelector:
          matchLabels:
            app: couchbase

Make sure to use the variables defined previously as required. Once we have this network policy applied we can deploy Couchbase - again using the Helm chart.

$ helm upgrade --install couchbase couchbase/couchbase-operator --set install.admissionController=false --namespace "$NAMESPACE"

As seen above, only deploy the DAC once - we disable it for the second Helm chart deployment of the operator and server pods.