OS Level Tuning

OS level parameters you can use to tune _Couchbase Sync Gateway

Introduction

To get the most out of Sync Gateway, you may need to tune OS-level parameters that control the maximum number of open file descriptors. This directly affects the maximum number of sockets Sync Gateway can have open, and therefore the maximum number of endpoints it can support.

The host-level instructions on this page apply to Sync Gateway installed directly on a host OS (for example, a bare-metal server or virtual machine). For container environments (Docker, Kubernetes, or similar), see Container Environments.

The host-level instructions below are geared towards CentOS deployments but are broadly applicable to other Linux distributions.

Container Environments

In a container environment, the host OS controls the upper bound on file descriptors, but each container gets its own ulimit set by the container runtime at startup — not by the host-level /etc/security/limits.conf or sysctl configuration. This means your Infra or Platform team, rather than the application team, typically owns this configuration.

File descriptor limits for Sync Gateway scale with the number of connected clients. There is no single fixed minimum, but the default OS limit of 1024 is insufficient for any production deployment. The recommended starting value for nofile (maximum open files) is 250000, which provides headroom for high client concurrency.

The exact value your deployment needs depends on your expected peak concurrent client connections and the resources available on the host. Start with 250000 and adjust based on observed behavior. If you continue to see too many open files errors after raising the limit, see TCP Keepalive Parameters for additional tuning options.

Docker

Pass the --ulimit flag when starting the Sync Gateway container to set the nofile limit for that container:

docker run -d \
  --ulimit nofile=250000:250000 \ (1)
  --name sync-gateway \
  -p 4984:4984 -p 4985:4985 \
  couchbase/sync-gateway

1	Sets both the soft and hard `nofile` limit to 250,000 for this container. The `--ulimit` flag requires Docker 1.6 or later.

Verify the effective limit inside the running container:

docker exec sync-gateway sh -c "ulimit -n"

The output should be 250000.

Kubernetes

Kubernetes does not expose a direct ulimit field in the Pod spec. The nofile limit for containers is inherited from the node’s container runtime configuration (containerd or CRI-O). To raise the limit, you have two options:

Configure the container runtime on each node — edit the runtime configuration (for example, containerd’s config.toml) to raise DefaultRlimits for NOFILE.
Use a privileged init container via a DaemonSet — set the limit at the node level before the Sync Gateway pod starts.

Work with your Infra or Platform team to determine which approach fits your cluster policy. The required changes depend on your runtime, distribution, and cluster security policy, and are therefore outside the scope of Couchbase documentation.

After applying changes, verify the effective limit from inside the running pod:

kubectl exec <sync-gateway-pod> -- sh -c "ulimit -n"

Operating System File Descriptor Limits

Increase the max number of file descriptors available to all processes.

Set the number of system wide file descriptors

Edit the /etc/sysctl.conf file
Add the following line.
```
fs.file-max = 500000
```
Apply the changes by running the following command.
```
$ sysctl -p (1)
```
1 The -p will persist the change across reboots

Sync Gateway File Descriptor Limits

Configure the maximum number of open file descriptors in Sync Gateway in line with the above changes.

See the max_file_descriptors and Example 1.

Example 1. Set the Maximum File Descriptors

{
  "max_file_descriptors": 250000 (1)
}

1	Default = 5000. Setting `max_file_descriptors` higher than the OS or container `nofile` limit has no effect — Sync Gateway cannot exceed the limit imposed by the runtime.

Service File Descriptor Limits

For systemd config: The /usr/lib/systemd/system/sync_gateway.service has a hardcoded limit specified by LimitNOFILE=65535.

To increase that, edit the /sync_gateway.service file to your desired value and restart the service.

Process File Descriptor Limits

This section applies only to host-based Sync Gateway deployments running outside of systemd. For container environments, see Container Environments. For systemd deployments, see Service File Descriptor Limits.

If you’re using systemd, you can skip this section.

Increase the ulimit setting for max number of file descriptors available to a single process. For example, setting it to 250K will allow Sync Gateway to have 250K connections open at any given time, and leave 250K remaining file descriptors available for the rest of the processes on the machine. These settings are an example — tune them for your own use case.
```
$ ulimit -n 250000
```
To persist the ulimit change across reboots, add the following lines to /etc/security/limits.conf:
```
* soft nofile 250000
* hard nofile 250000
```
Verify your changes by running the following commands:
```
$ cat /proc/sys/fs/file-max
$ ulimit -n
```
The output value of both commands above should be 250000.

TCP Keepalive Parameters

If you have already raised the maximum number of file descriptors available to Sync Gateway, but you are still seeing "too many open files" errors, you may need to tune the TCP Keepalive parameters.

Understanding the Problem

Mobile endpoints tend to abruptly disconnect from the network without closing their side of the connection, as described in Section 2.3. (Checking for dead peers) of the TCP-Keepalive-HOWTO.

By default, these connections will hang around for approximately 7200 seconds (2 hours) before they are detected to be dead and cleaned up by the tcp/ip stack of the Sync Gateway process. If enough of these connections accumulate, you can end up seeing "too many open files" errors on Sync Gateway.

If you are seeing "too many open files" errors, you can count the number of established connections coming into your sync gateway with the following command:

$ lsof -p <sync_gw_pid> | grep -i established | wc -l

If the value returned is near your max file descriptor limit, you can either try increasing the max file descriptor limit even further, or tune the TCP Keepalive parameters to reduce the amount of time that dead peers hold a socket open on their behalf.

Linux Instructions (CentOS)

Tuning the TCP Keepalive settings is not without its downsides — it will increase the amount of overall network traffic on your system, because the tcp/ip stack will be sending more frequent keepalive packets in order to detect dead peers faster.

The following settings will reduce the amount of time that dead peer connections hang around from approximately 2 hours down to approximately 30 minutes. Add the following lines to your /etc/sysctl.conf file:

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 20

This translates to:

The keepalive routines wait initially for 10 minutes (600 secs) before sending the first keepalive probe.
Resend the probe every minute (60 seconds).
If no ACK response is received for 20 consecutive times, the connection is marked as broken.

To reduce the amount of time even further, you can reduce the tcp_retries2 value. Add the following line to your /etc/sysctl.conf file:

net.ipv4.tcp_retries2 = 8

To activate the changes and persist them across reboots, run:

$ sysctl -p

See Using TCP keepalive under Linux for more details on setting these parameters.

OS Level Tuning

Introduction

Container Environments

Docker

Kubernetes

Operating System File Descriptor Limits

Sync Gateway File Descriptor Limits

Service File Descriptor Limits

Process File Descriptor Limits

TCP Keepalive Parameters

Understanding the Problem

Linux Instructions (CentOS)

Related Content