Kubernetes Network Policies Using Deny-All Default

    +
    The Autonomous Operator and Couchbase Server can be used with Kubernetes network policies although this is not officially supported currently.

    Tutorials are accurate at the time of writing but rely heavily on third party software. Tutorials are provided to demonstrate how a particular problem may be solved. Use of third party software is not supported by Couchbase. For further help in the event of a problem, contact the relevant software maintainer.

    Overview

    Refer to the concepts page on Kubernetes networking for an introduction to network policies and the rules required for Couchbase.

    Network policies should work with Couchbase Autonomous Operator deployments but are currently unsupported officially. This information is provided to document a functional set up for those that need it prior to official support. The assumption is standard Kubernetes configuration is used rather than any specific to a particular network plugin.

    This example uses the Calico network plugin to show how to enable network policies on a local development cluster. It shows examples of how to configure the various information required, actual deployments may differ.

    No rules are set up for any external traffic for Couchbase SDKs, helm or kubectl usage.

    We use the Helm chart here only to simplify the examples.

    Cluster setup

    The Kubernetes cluster needs to be configured to support network policies with an appropriate network plugin. Refer to the official documentation for details on this.

    As an example, the following script will create a Kubernetes-in-Docker (KIND) cluster locally with two Kubernetes nodes.

    $ kind create cluster --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      disableDefaultCNI: true    # disable kindnet
      podSubnet: 192.168.0.0/16  # set to Calico's default subnet
    nodes:
      - role: control-plane
      - role: worker
    EOF

    Once we have our Kubernetes cluster available, we can deploy the Calico network plugin to it which will manage the network policies for us.

    $ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
    $ kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=true
    $ kubectl -n kube-system set env daemonset/calico-node FELIX_XDPENABLED=false

    The example above is purely for demonstration purposes and is not guaranteed to be functional or recommended to be used in production.

    Couchbase Dynamic Admission Controller

    The typical and recommended deployment is a single cluster-wide dynamic admission controller (DAC) for Couchbase. For the purposes of the example we therefore deploy the DAC to the default namespace with no network policies applied.

    $ helm upgrade --install couchbase-dac couchbase/couchbase-operator --set install.couchbaseCluster=false,install.couchbaseOperator=false --namespace default --wait

    Couchbase Autonomous Operator

    We will need to supply the Kubernetes API server details along with the DAC endpoint used by the operator to network policy rules. We also need to supply the Kubernetes namespace to use, here we simply use test as the name.

    $ export API_SERVER_IP=$(kubectl get endpoints --namespace default kubernetes --output=json|jq '.subsets[0].addresses[0].ip' -r)
    $ export API_SERVER_PORT=$(kubectl get endpoints --namespace default kubernetes --output=json|jq '.subsets[0].ports[0].port' -r)
    $ export NAMESPACE=test
    The variables above will be used for substitution in the following steps.

    To provide a functional network policy we need to enable the rules specified in the Kubernetes networking page.

    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: $NAMESPACE
    ---
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: default-deny-all-except-dns
      namespace: $NAMESPACE
      # Prevent any traffic for all pods apart from that we specifically allow.
      # This is taken from the Kubernetes documentation with a tweak to allow DNS.
    spec:
      podSelector: {}
      policyTypes:
      - Ingress
      - Egress
      egress:
      # Allow DNS resolution, primarily for Couchbase pods but it triggers other
      # hard to debug connection issues sometimes too so be aware if this is moved to
      # the later rules.
      - ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    ---
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: couchbase-operator-policy
      namespace: $NAMESPACE
      # The operator needs to talk to server pods in its namespace,
      # plus the K8S API and the DAC admission hook.
      #
      # If the operator is in another namespace (cluster wide) then the rules
      # will need a namespace selector to allow that.
      # Remember that any rules need to match on both ends to allow traffic, e.g.
      # if you have a deny-all default and only allow traffic out of the operator
      # but not in on the server pods then it will be prevented.
    spec:
      # Select operator pods
      podSelector:
        matchLabels:
          app.kubernetes.io/name: couchbase-operator
      # No ingress required as traffic is always out to other components.
      policyTypes:
        - Egress
      egress:
      # Allow all traffic to Couchbase Server pods
      - to:
        - podSelector:
            matchLabels:
              app: couchbase
      # Allow traffic to the K8S API server IP and port only
      - ports:
        - port: $API_SERVER_PORT
          protocol: TCP
        to:
        - ipBlock:
            cidr: $API_SERVER_IP/32
    ---
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: couchbase-namespace-policy
      namespace: $NAMESPACE
      # Couchbase server pods need to talk to each other in the same namespace.
      # They also need to allow the operator to control them.
    spec:
      # Select all Couchbase Server pods
      podSelector:
        matchLabels:
          app: couchbase
      policyTypes:
        - Ingress
        - Egress
      ingress:
        # Allow all traffic from other Couchbase Server pods in the same namespace,
        - from:
          - podSelector:
              matchLabels:
                app: couchbase
        # Allow all traffic from the operator.
        - from:
          - podSelector:
              matchLabels:
                app.kubernetes.io/name: couchbase-operator
      egress:
        # Allow all traffic to other Couchbase Server pods in the same namespace.
        - to:
          - podSelector:
              matchLabels:
                app: couchbase

    Make sure to use the variables defined previously as required. Once we have this network policy applied we can deploy Couchbase - again using the Helm chart.

    $ helm upgrade --install couchbase couchbase/couchbase-operator --set install.admissionController=false --namespace "$NAMESPACE"

    As seen above, only deploy the DAC once - we disable it for the second Helm chart deployment of the operator and server pods.