Logging

This page describes the Sync Gateway logging API and how to enable different log settings.

Continuous Logging

Continuous logging is a new feature in Sync Gateway 2.1 that provides more flexibility in how logs are generated and retained, whilst maintaining the level of logging required by Couchbase Support for investigation of issues.

Sync Gateway logging is now written to separate files by log level, with each log level supporting individual retention policies.

Console logging can also be configured independently, providing additional flexibility for system administrators depending on their needs. This allows system administrators running Sync Gateway to tweak the log level, and log keys for the console output to suit their needs, whilst maintaining the level of logging required by Couchbase Support for investigation of issues.

The previous logging configuration (logging.default) is being deprecated, and Sync Gateway 2.1 will display warnings on startup of what is required to update your configuration.

Console Log Output

The console output of Sync Gateway can be filtered down via Log Levels and Log Keys, and you can tweak this as much as you like without impacting Support’s ability to analyze the log files.

Log Levels

The console log output can be configured with the following log levels, ordered from least verbose, to most. Note that log levels are additive, so if you enable info level, warn and error logs are also enabled. By default, the log level is set to info.

Log Level Appearance Description

Log Level	Appearance	Description
`none`	-	Disables log output
`error`	`[ERR]`	Displays errors that need urgent attention
`warn`	`[WRN]`	Displays warnings that need some attention
`info`	`[INF]`	Displays information about normal operations that don’t need attention
`debug`	`[DBG]`	Displays verbose output that might be useful when debugging
`trace`	`[TRC]`	Displays extremely verbose output that might be useful when debugging

none

Disables log output

error

[ERR]

Displays errors that need urgent attention

warn

[WRN]

Displays warnings that need some attention

info

[INF]

Displays information about normal operations that don’t need attention

debug

[DBG]

Displays verbose output that might be useful when debugging

trace

[TRC]

Displays extremely verbose output that might be useful when debugging

Log levels can be set in the configuration file (see the logging.$level reference).

Log Keys

Log keys are used to provide finer-grained control over what is logged. By default, only HTTP is enabled.

All log keys and descriptions are described in the logging.console.log_level property reference.

Console Output Color

There is an option to color log output based on log level if logging.console.color_enabled is set to true. Note: this setting is always disabled on Windows for compatibility reasons.

Console Output Redirection

The log files described below are intended for Couchbase Support, and users are urged not to rely on these.

If you have special requirements for logs, such as centralized logging, you can redirect the console output to a file, and apply your own log collection mechanism to feed that data elsewhere. For example:

# Start Sync Gateway and redirect console output to a file
./sync-gateway > my_sg_logs.txt 2>&1

# Start log collection to send to a centralized log aggregator.
logcollector my_sg_logs.txt

Log File Outputs

These are 4 log files split by log level, with a guaranteed retention period for each. The log files can be collected with SGCollect Info, and can be analyzed by Couchbase Support for diagnosing issues in Sync Gateway. As described above, it is recommended to use Console Output redirection if you require special handling of log files from Sync Gateway, as these files are intended for Couchbase Support.

Log File Default enabled Default max_age Minimum max_age

Log File	Default enabled	Default max_age	Minimum max_age
`sg_error.log`	`true`	360 Days	180 Days
`sg_warn.log`	`true`	180 Days	90 Days
`sg_info.log`	`true`	6 Days	3 Days
`sg_debug.log`	`false`	2 Days	1 Day

sg_error.log

true

360 Days

180 Days

sg_warn.log

true

180 Days

90 Days

sg_info.log

true

6 Days

3 Days

sg_debug.log

false

2 Days

1 Day

Each log level and its parameters are described in the logging.$level property reference.

Log File Rotation

These four log files will be rotated once each exceeds a threshold size, defined by max_size in megabytes. Once rotated, the log files will be compressed with gzip, to reduce the disk space taken up by older logs. These old logs will then be cleaned up once the age exceeds max_age in days.

Log Redaction

All log outputs can be redacted, this means that user-data, considered to be private, is removed. This feature is optional and can be enabled in the configuration with the logging.redaction_level property.

Log Rotation [DEPRECATED]

Built-in log rotation

By default, Sync Gateway outputs the logs to standard out with the "HTTP" log key and can also output logs to a file. Prior to 1.4, the two main configuration options were log and logFilePath at the root of the configuration file.

{
    "log": ["*"],
    "logFilePath": "/var/log/sync_gateway/sglogfile.log"
}

In Couchbase Mobile 1.4, Sync Gateway can now be configured to perform log rotation in order to minimize disk space usage.

Log rotation configuration

The log rotation configuration is specified under the logging key. The following example demonstrates where the log rotation properties reside in the configuration file.

{
  "logging": {
    "default": {
      "logFilePath": "/var/log/sync_gateway/sglogfile.log",
      "logKeys": ["*"],
      "logLevel": "debug",
      "rotation": {
        "maxsize": 1,
        "maxage": 30,
        "maxbackups": 2,
        "localtime": true
      }
    }
  },
  "databases": {
    "db": {
      "server": "walrus:data",
      "bucket": "default",
      "users": {"GUEST": {"disabled": false,"admin_channels": ["*"]}}
    }
  }
}

As shown above, the logging property must contain a single named logging appender called default. Note that if the "logging" property is specified, it will override the top level log and logFilePath properties.

The descriptions and default values for each logging property can be found on the Sync Gateway configuration page.

Example Output

If Sync Gateway is running with the configuration shown above, after a total of 3.5 MB of log data, the contents of the /var/log/sync_gateway directory would have 3 files because maxsize is set to 1 MB.

/var/log/sync_gateway
├── sglogfile.log
├── sglogfile-2017-01-25T23-35-23.671.log
└── sglogfile-2017-01-25T22-25-39.662.log

Windows Configuration

On MS Windows logFilePath supports the following path formats.

"C:/var/tmp/sglogfile.log"
`C:\var\tmp\sglogfile.log`
`/var/tmp/sglogfile.log`
"/var/tmp/sglogfile.log"

Log rotation will not work if logFilePath is set to the path below as it is reserved for use by the Sync Gateway Windows service wrapper.

C:\Program Files (x86)\Couchbase\var\lib\couchbase\logs\sync_gateway_error.log

Deprecation notice

The current proposal is to remove the top level log and logFilePath properties in Sync Gateway 2.0. For users that want to migrate to the new logging config to write to a log file but do not need log rotation they should use a default logger similar to the following:

{
    "logging": {
        "default": {
            "logFilePath": "/var/log/sync_gateway/sglogfile.log",
            "logKeys": ["*"],
            "logLevel": "debug"
        }
    }
}

OS log rotation

In production environments it is common to rotate log files to prevent them from taking too much disk space, and to support log file archival.

By default Sync gateway will write log statements to stderr, normally stderr is redirected to a log file by starting Sync Gateway with a command similar to the following:

sync_gateway sync_gateway.json 2>> sg_error.log

On Linux the logrotate tool can be used to monitor log files and rotate them at fixed time intervals or when they reach a certain size. Below is an example of a logrotate configuration that will rotate the Sync Gateway log file once a day or if it reaches 10M in size.

/home/sync_gateway/logs/*.log {
    daily
    rotate 1
    size 10M
    delaycompress
    compress
    notifempty
    missingok

The log rotation is achieved by renaming the log file with an appended timestamp. The idea is that Sync Gateway should recreate the default log file and start writing to it again. The problem is Sync Gateway will follow the renamed file and keep writing to it until Sync gateway is restarted. By adding the copy truncate option to the logrotate configuration, the log file will be rotated by making a copy of the log file, and then truncating the original log file to zero bytes.

/home/sync_gateway/logs/*.log {
    daily
    rotate 1
    size 10M
    copytruncate
    delaycompress
    compress
    notifempty
    missingok
}

Using this approach there is a possibility of loosing log entries between the copy and the truncate, on a busy Sync Gateway instance or when verbose logging is configured the number of lost entries could be large.

In Sync Gateway 1.1.0 a new configuration option has been added that gives Sync Gateway control over the log file rather than relying on stderr. To use this option call Sync Gateway as follows:

sync_gateway -logFilePath=sg_error.log sync_gateway.json

The logFilePath property can also be set in the configuration file at the server level.

If the option is not used then Sync Gateway uses the existing stderr logging behavior. When the option is passed Sync Gateway will attempt to open and write to a log file at the path provided. If a Sync Gateway process is sent the SIGHUP signal it will close the open log file and then reopen it, on Linux the SIGHUP signal can be manually sent using the following command:

pkill -HUP sync_gateway

This command can be added to the logrotate configuration using the 'postrotate' option:

/home/sync_gateway/logs/*.log {
    daily
    rotate 1
    size 10M
    delaycompress
    compress
    notifempty
    missingok
    postrotate
        /usr/bin/pkill -HUP sync_gateway > /dev/null
    endscript
}

After renaming the log file logrotate will send the SIGHUP signal to the sync_gateway process, Sync Gateway will close the existing log file and open a new file at the original path, no log entries will be lost.