Managing a Sync Gateway Cluster

Troubleshooting and Log Collection

This section provides information about how to diagnose and troubleshoot problems with the Sync Gateway deployment. When troubleshooting, it is important to rule out Kubernetes itself as the root cause of the problem you are experiencing. See the Kubernetes Troubleshooting guide for information about debugging applications within a Kubernetes cluster. For troubleshooting a Couchbase Server deployment using the Couchbase operator, please refer to our guide.

Prerequisites

The location of the Sync Gateway logs within a pod is specified with the "logging" key in the Sync Gateway config file (see logging guide)
To view or fetch the logs from a Sync Gateway pod, you must first identify the Sync Gateway pods in your deployment using the following command.
```
kubectl get pods
```
Your response will include a list of sync gateway pods in your deployment

Viewing Logs in Console

You can use kubectl to view logs for a specific pod using following command. This command will output logs to the console.

kubectl logs -f <pod_id>

Using sgcollect_info

SGCollect Info is a command line utility with detailed statistics for a specific Sync Gateway node. This tool must be run on each node individually. The sgcollect-info tool is bundled as part of the Sync Gateway docker image.

Collecting logs on a pod

To collect logs on a specific pod, run sgcollect_info using the kubectl exec command. The following command runs sgcollect_info on the specified pod and outputs the results to an out.zip file.

kubectl exec <pod_id> -- /opt/couchbase-sync-gateway/tools/sgcollect_info /tmp/out.zip

Coping log output from pod

To copy the sgcollect_info generated output from the pod, use the kubectl cp command.

kubectl cp <pod_id>:/tmp/out.zip .

Convenience Scripts

Collecting sgcollect_info logs from all pods

The following script runs sgcollect_info on all the pods and to copy the zipped files over to the current folder.

#! /bin/sh
OUTFOLDER=${1:-/tmp}
for pod in `kubectl get pods -o=name | grep sync-gateway | sed "s/^.\{4\}//"`
do
    LOGFILE=$OUTFOLDER/$pod'_sgcollect_info_out.zip'
    echo "Running sgcollect_info to generate $LOGFILE"
    kubectl exec $pod -- /opt/couchbase-sync-gateway/tools/sgcollect_info $LOGFILE
    echo "copying $pod:$LOGFILE to current folder"
    kubectl cp $pod:$LOGFILE .
done

Rolling Upgrades of Sync Gateway Nodes

Since we use a deployment controller, we can do a rolling upgrade of the Sync Gateway nodes in the cluster and the load balancer will automatically take care of redirecting the traffic to the remaining nodes.

Note that if you have configured your deployment to have a single Sync Gateway import node, then you will experience some latency in processing write requests until the import node comes back up again.

For example, if you want to upgrade the version of the Sync Gateway, you can update the corresponding deployment config file and re-apply that to the cluster
```
kubectl apply -f sgw-deployment-upgrade.yaml
```
You can verify the status of the rollout as follows
```
kubectl rollout status deployment sync-gateway
```
The deployment controller will ensure that at least one replica is available to serve the incoming requests while the other replicas are being upgraded. The load balancer will automatically redirect all requests to the remaining nodes.

If the upgrade fails for any reason, you can rollback or undo the rollout with the following command.
```
kubectl rollout undo  deployment sync-gateway
```

Upgrading the Kubernetes Cluster

You can refer to the following Couchbase Operator guide for a discussion about upgrading the Kubernetes cluster on which your Couchbase deployment resides.