Logging

    +

    Description — Introducing Couchbase Sync Gateway’s logging functionality
    Abstract — Sync Gateway’s _Continuous Logging_ feature delivers flexible log generation and retention, without compromising the availability of diagnostic information necessary to provide effective support and maintenance.
    Related Content — Configuration File | Admin REST API | SG Collect

    Overview

    Sync Gateway provides a robust Continuous Logging[1] solution that delivers flexibility in terms of how logs are generated and retained, whilst also maintaining the level of logging required by Couchbase Support for investigation of issues. Its logs are written to separate log files filtered by log level, with each log level supporting individual retention policies. You control what is logged using the sync-gateway-config.json configuration file settings for logging.

    In addition to the log files, you can also independently configure and control Console Logging, which is a convenient method of accessing diagnostic information during debugging scenarios. With console logging, system administrators can easily fine-tune diagnostic output to suit specific debug scenarios. All without interfering with the logging required by Couchbase Support for the investigation of issues.

    Configuration

    You configure your continuous and console logging requirements in the sync-gateway-config.json file, using the logging properties — see: Example 1.

    Example 1. Sample Logging Configuration
    {
      "logging": {
        "log_file_path": "/var/tmp/sglogs", (1)
        "redaction_level": "partial", (2)
        "console": { (3)
          "log_level": "debug",
          "log_keys": ["*"]
          }, (4)
        "error": { (5)
          "enabled": true,
          "rotation": {
            "max_size": 20,
            "max_age": 180
            }
          },
        "warn": { (6)
          "enabled": true,
          "rotation": {
            "max_size": 20,
            "max_age": 90
            }
          },
        "info": { (7)
          "enabled": false
        },
        "debug": { (8)
          "enabled": false
          }
        },
      "databases": {
        "db": {
          "server": "couchbase://localhost",
          "username": "username",
          "password": "password",
          "bucket": "default",
          "users": {"GUEST": {"disabled": false,"admin_channels": ["*"]}},
          "allow_conflicts": false,
          "revs_limit": 20
          }
        }
      }
    1 Set the path to the log file(s)
    2 Define the optional redaction level, here we select "partial" redaction — see: Log Redaction
    3 Here we define the Console Logging levels we require for debugging. In this instance turning on debug level output for all available log_keys — see: console log configuration properties
    4 The following logging properties enable, or disable, Continuous Logging at the specified level — see: continuous logging configuration properties:
    5 Here we set error level logging on and define the log-file rotation, for errors messages
    6 Here we set warn level logging on and define the log-file rotation, for warnings messages
    7 Here we set info level logging off; we do not define the log-file rotation
    8 Here we set debug level logging off; we do not define the log-file rotation

    See: Configuration File for more information on these settings.

    Log Redaction

    All log outputs — console or continuous — can optionally be redacted, which will remove any user-data considered private.

    You enable this feature by setting the `logging.redaction_level` property.

    Console Logging

    By default only HTTP logging is enabled

    Console logs are your go-to resource for diagnostic information. You can easily fine-tune their diagnostic content to meet the needs of a particular debugging scenario, perhaps by increasing the verbosity and filtering out unnecessary log_keys to better focus on the problem area.

    Changes to console logging are independent of continuous logging, so you can, for example, tweak any of the following without compromising the core continuous logging streams:

    • Increase the verbosity using Log Levels to generate additional diagnostic information

    • Focus on the area under investigation by enabling or disabling specific Log Keys

    • Enhance readability by setting a color for log output based on log level

    Admin REST API

    You can define console log settings in the configuration file, or more conveniently, you can use the Admin REST API to adjust them — see: Example 2.

    Example 2. Setting log_level and log_keys with API
    curl --location --request POST 'http://localhost:4985/_logging?logLevel=trace' \ (1)
    --header 'Content-Type: application/json' \
    --data-raw '{"HTTP":false, "WS": true, "WSFrame": true, "Replicate": true}' (2)
    1 Here we define the log_level to be trace for maximum verbosity
    2 Here we specify the particular log_keys we want to focus on; perhaps we suspect a websocket or replication issue?

    The console log will show the following after this command:

    2021-01-08T13:26:23.884Z [INF] HTTP:  #110: POST /_logging?logLevel=trace (as ADMIN)
    2021-01-08T13:26:23.885Z [INF] Setting log level to: trace
    2021-01-08T13:26:23.885Z [INF] Setting log keys to: [DCP Replicate WS WSFrame]
    ---

    Log Levels

    When debugging, setting the console log’s log-level to debug or trace can provide valuable additional information

    Console logs have six levels of verbosity — see: Table 1. The default log level is info

    Note that the log levels are inclusive, so if you enable info level, then warn and error logs are also enabled.

    You can define console log levels using the configuration file (see logging-console-log_level) and by using the Admin REST API (see: Example 2).

    One approach might be to set your base level in the configuration file and then use the Admin REST API for specific debugging scenarios.

    Table 1. Console Logging — Available Log Levels

    Log Level

    Appearance

    Description

    none

    -

    Disables log output

    error

    [ERR]

    Displays errors that need urgent attention

    warn

    [WRN]

    Displays warnings that need some attention

    info

    [INF]

    Displays information about normal operations that don’t need attention

    debug

    [DBG]

    Displays verbose output that might be useful when debugging

    trace

    [TRC]

    Displays extremely verbose output that might be useful when debugging

    Log Keys

    Select log keys relevant to the area you are debugging, providing them as a comma-delimited list, such as: "log_keys": ["HTTP", "CRUD", "Import"] in the config or see Example 2 for how to provide them using the Admin REST API.

    Log keys provide fine-grained control over the information types that Sync Gateway outputs to the console log. By default, only HTTP related information is enabled, but a range of other keys are available to meet specific diagnostic needs — see: Table 2.

    You can define the required `logging.console.log_keys` within your configuration file and-or use the Admin REST API (see: Example 2).

    Table 2. List of Available Log Keys

    Log Key

    Description

    *

    This wildcard log key, enables all log keys

    curl --location --request PUT 'http://localhost:4985/_logging?logLevel=trace' \
    --header 'Content-Type: application/json' \
    --data-raw '{"*": true}'

    none

    Disable all log keys; no logging output

    curl --location --request PUT 'http://localhost:4985/_logging?logLevel=trace' \
    --header 'Content-Type: application/json' \
    --data-raw '{"none": true}'

    Admin

    Admin processes in Sync Gateway.

    Access

    Anytime an access() call is made in the sync function.

    Auth

    Authentication.

    Bucket

    Sync Gateway interactions with the bucket (trace level only).

    Cache

    Interactions with Sync Gateway’s in-memory channel cache.

    Changes

    Processing of /{db}/_changes requests.

    CRUD

    Updates made by Sync Gateway to documents.

    DCP

    DCP-feed processing.

    Events

    Event processing (webhooks).

    gocb

    All logging emitted by the GoCB SDK

    HTTP

    All requests made to the Sync Gateway REST APIs.

    HTTP+

    Additional information about HTTP requests (response times, status codes).

    Import

    Introduced in Sync Gateway 1.5 to help troubleshoot the import process of a document (this is the Sync Gateway process to make a document that was added through N1QL or the Server SDKs mobile-aware). This log key can be useful to troubleshoot why a given document was not successfully imported.

    Javascript

    All logging from Javascript. This includes: sync function, import filters, webhook filter function, and the custom ISGR conflict resolvers

    Migrate

    Logs messages that show when old inline document metadata is upgraded to xattrs

    Query

    Query is used for Sync Gateway code related to N1QL queries

    Replicate

    Log messages related to replications between Sync Gateways (using sg-replicate). This tag cannot be used for replications initiated by Couchbase Lite.

    SGCluster

    Log messages related to the sharded import and HA sg-replicate

    Sync

    Activity which relates to synchronization between Couchbase Lite and Sync Gateway

    SyncMsg

    Can be used for additional Sync logging output

    WS

    Websocket replication log messages

    WSFrame

    Can be used for additional WS logging output

    Set Log Color

    You may set a color for log output based on log level by using `logging.console.color_enabled` set to true

    This setting is always disabled on Windows for compatibility reasons.

    Redirect Console Log

    You can easily redirect the console log output to a file. This can be useful not only for diagnostic sessions, but also when you have specialized logging requirements, such as centralized logging. Just redirect the output and then apply your own log collection mechanism to feed that data elsewhere — see Example 3.

    Example 3. Console Log Redirection
    # Start Sync Gateway and redirect console output to a file
    ./sync-gateway > my_sg_logs.txt 2>&1
    
    # Start log collection to send to a centralized log aggregator.
    logcollector my_sg_logs.txt

    Continuous Logging

    In this section: Log File Outputs | Log File Rotation

    Continuous logging produces a set of log files aimed primarily at providing appropriate diagnostic information for the Couchbase Support team should their intervention be required. You define continuous logging settings in the configuration file — see: Example 1.

    With continuous logging the logs for each level are written to separate log files — see: Table 3. You can set individual retention policies for each log-level.

    Log File Outputs

    The log files output from continuous logging are intended solely for the use of Couchbase Support.

    If you require special log handling, for example for centralized logging, then use the Redirect Console Log feature to create a log file for this purpose from the console output stream.

    Sync Gateway produces four separate log files, split by log level. Each log file has its own guaranteed retention period - as shown in Table 3

    You can collect the log files, for analysis by Couchbase Support when diagnosing Sync Gateway issues, using SG Collect.

    Table 3. Continous Logging - Log File Outputs

    Log File

    Level

    Description

    Default enabled

    Default max_age

    Minimum max_age

    ERROR

    Critical error messages.

    sg_error.log

    true

    360 Days

    180 Days

    WARN

    Something is wrong but SG can still service requests

    sg_warn.log

    true

    180 Days

    90 Days

    INFO

    Important diagnostics for support and customers

    sg_info.log

    true

    6 Days

    3 Days

    DEBUG

    Lower level development analysis

    sg_debug.log

    false

    2 Days

    1 Day

    Each log level and its parameters are defined using the (xref-sgw-bmk-config-properties-log-level) property.

    Log File Rotation

    Log files are rotated when they exceed a threshold max_size (megabytes). Once rotated, they are compressed (gzip) to reduce the disk usage.

    Aged logs are cleaned up once their age exceeds max_age days — see: Table 3

    Configure log rotation using the logging-$level-rotation property.

    For pre-2.1 log rotation — see: Log Rotation pre-2.1


    1. Introduced in Sync Gateway version 2.1