Inter-Sync Gateway Replication

    +

    Description — Use inter-Sync Gateway replication to keep clusters in different mobile data centers in sync.
    Abstract — Inter-Sync Gateway replication supports resilient, secure, scalable bidirectional synchronization of data between cloud and edge data centers. It incorporates content presented under Manage>Replicate>Inter-Sync Gateway to give an integrated view.
    Related Content — Configuration Details | Admin REST API | Initialize Inter-Sync Gateway Replications

    Context Clarification

    This content relates only to inter-Sync Gateway replication in Sync Gateway 2.8+. For documentation on pre-2.8 inter-Sync Gateway replication (also known as SG Replicate) — see Inter-Sync Gateway Replication pre-2.8

    Introduction

    Couchbase Sync Gateway’s Inter-Sync Gateway Replicationglossary icon feature supports cloud-to-edgeglossary icon synchronization use cases, where data changes must be synchronized between a centralized cloud cluster and a large number of edge clusters whilst still enforcing fine grained access control. This is an increasingly important enterprise-level requirement.

    In the architecture diagram (Figure 1), the replicator on the active Sync Gateway node ensures that any database changes made to documents in either Sync Gateway databaseglossary icon instance are replicated to the other Sync Gateway instance, in accordance with the replication’s configuration — see replications for configuration details.

    icr replication overview
    Figure 1. Inter-Sync Gateway Replication architecture

    Use Cases

    Cloud-to-edge synchronization

    In this multi-cloud deployment mode large numbers of multiple edge clusters sync with one or more clusters in cloud data centers. Each edge can operate autonomously without network connectivity to the cloud data centers [1].

    A typical architecture for this use cases is shown in Figure 2

    icr cloud to edge200712
    Figure 2. Cloud-to-edge synchronization

    Active-to-active Mobile Synchronization

    Here edge clusters containing active Sync Gateway nodes are replicated between geographically separate cloud-based Sync Gateway deployments. An ideal use-case for inter-Sync Gateway replication, which was designed to keep clusters in different data centers in sync.

    The Couchbase Server Cross Data Center Replication API (XDCR) is similarly used to replicate between Couchbase Server clusters. However, the inter-Sync Gateway replication feature is specifically designed for Sync Gateway deployments and it must be used for replication between clusters containing an active Sync Gateway nodeglossary icon.

    A typical architecture for this use case is shown in Figure 3

    icr active mobile sync200713
    Figure 3. Active-to-active mobile synchronization

    Replication Characteristics

    Context

    Sync Gateway supports the ability to run replications between Sync Gateway clusters using a new websockets based protocol, with the active replicatorglossary icon synchronizingglossary icon changes between two Sync Gateway databasesglossary icon.

    All replications are based on a replication definitionglossary icon provided either to the Admin REST API or in Sync Gateway’s configuration file (JSON).

    Replications always involve at least one local database. Sync Gateway does not enable replication between two remote nodes, because replications are defined at database level and so at least one database will be local.

    All replications take place at the document level (but see also, Delta Sync).

    Sync Gateway nodes can opt-out of participating in the replication process using the database-level parameter sgreplicate_enabled.

    Related configuration elements: databases | replications | remote | sgreplicate_enabled

    Protocol

    Inter-Sync Gateway replications are based on websockets. This is the exact same protocol that is used for replication with Couchbase Lite 2.x clients.

    For users on releases prior to 2.8, SG Replicate provides a HTTP-based replication — see the appropriate documentation here — Inter-Sync Gateway Replication pre-2.8 (deprecated).

    The bi-directional, persistent, nature of websocket connections is ideal for applications such as a continuous Sync Gateway replication, which is constantly waiting-for and synchronizing change events.

    Types of Replication

    Replications are either: adhoc replicationglossary icon (REST API only) or persistent replicationglossary icon. They can also be configured to run in one of two-modes: continuousglossary icon or one-shotglossary icon.

    • Persistent

      Persistent replications survive Sync Gateway node restarts and continue running automatically unless configured not to. They can be configured to run in either continuousglossary icon or one-shotglossary icon mode.

    • Ad hoc

      Ad hoc replications are transient, existing only for the period of the replication. They provide a convenient way to:

      • Run one off replications (for example, when troubleshooting)

      • Run on-demand replications after Sync Gateway is started. For instance a replication that needs to be to be run only periodically can be configured as an ad hoc replication by an automated script scheduled to run when needed.

    Related configuration elements: replications | continuous | adhoc

    Delta Sync

    This content relates only to ENTERPRISE EDITION

    With delta-sync enabled on the replication and both databases involved, only the changed data items are transferred.

    You can configure replications to use delta-sync by:

    • Setting "enable_delta_sync": true in the replication definition

    • Setting "delta-sync": { "enabled": true} on both databases in their respective database definitions.

    Push replications to pre-2.8 targets do not use Delta Sync

    Related configuration elements: databases | replications | delta_sync | enable_delta_sync

    Directionality

    Replications are bi-directional. You can push, pull or push and pull between the two database endpoints.

    Related configuration elements: replications | direction

    Security

    Transport level security is provided for. Use the appropriate prefix in URL (WSS for websockets).

    Authentication

    Support for Basic Authentication using username and password credentials is provided

    Access Control

    Data access control is provided by Sync Gateway’s sync functionglossary icon and the username/password credentials. All replicated documents pass through this function ensuring that access permissions are adhered to.

    Related configuration elements: databases | sync
    Related how-to: Defining Sync Functions | Use Sync functions?

    Network Resilience

    Inter-Sync Gateway replications will automatically attempt to restart whenever the node restarts.

    Network resiliency is built-in for continuous replications. They respond to network issues such as lost connections, by applying a persistent exponential backoffglossary icon policy to attempt reconnection.

    The max_backoff_time determines the maximum wait time between retries. When the limit is reached retries are made every max_backoff_time minutes. Set "max_backoff_time": 0 to prevent indefinite retries. Exponential backoff retries will be attempted for up to 5 minutes and then stop if the connection has not been re-established


    Related replication definition elements: max_backoff_time

    High Availability

    Overview

    This content relates only to ENTERPRISE EDITION
    • Enterprise

    • Community

    Inter-Sync Gateway Replication provides built-in High Availability (HA) support. It uses node distribution to ensure all running replications are uniformly distributed across all available nodes, regardless of their originating node.

    A replication runs on only one node at any given time. When a node fails, the system automatically distributes that node’s replications across any available alternative nodes (providing the replication has been configured on multiple nodes).

    To use high-availability, configure the same replication on at least two Sync Gateway nodes.

    Even though automatic node-distribution is not available in COMMUNITY EDITION, you can make your replications more highly-available.

    Simply define the same replication on multiple nodes. They will then run on each of those nodes.

    This redundancy provides some resiliency if a node fails. As, although no automatic distribution of replications is done,if the replication is running on multiple nodes then it will continue running on any surviving nodes.

    Node Distribution

    The goal of node distribution is to maintain an optimal balance of replications across the cluster.,with any given replication runs on only one node at any give time.

    To achieve this Sync Gateway automatically balances, as equally as possible, the number of replications running on each node.

    Where multiple replications are configured on multiple nodes, Sync Gateway automatically distributes these replications across all the available nodes for which the replications are configured. It continually monitors and redistributes replications as the number of available nodes and the number of running replications in a cluster changes.

    The nodes' processing load and bandwidth usage is minimized by ensuring that a replicator runs on only one node at any given time — even where it has been configured to run on multiple nodes. This avoids the redundant exchange of data arising from duplicate replication.

    Configuration Requirements

    To configure a replication to be highly available, include its database and replication definition in the sync gateway configuration on each node in the cluster that you want to be able to run it. At least two nodes are required.

    Node distribution will automatically elect an appropriate node to run them on and take care of redistributing them if a node fails.

    Related configuration elements`: Configuration Details | Admin REST API

    Expected Failure Behavior

    If a node fails, then Sync Gateway will take any replications configured on multiple nodes, redistribute them across all available remaining nodes and restart them. Node distribution will continually seek to maintain an optimal distribution of replications across available nodes.

    Examples of Expected behavior

    This section provides examples of expected behavior in differing scenarios. It provides a comparison of how behavior differs between ENTERPRISE EDITION and COMMUNITY EDITION.

    The following scenarios are covered, each involves a sync gateway cluster with multiple nodes:

    • Homogenous configuration — see Example 1

    • Homogenous configuration with non-replicating node — see Example 2

    • Heterogenous configuration — see Example 3

    • Adding more nodes — see Example 4

    • Failing node — see Example 5

    Example 1. Homogenous configuration
    Scenario
    • The cluster comprises three Sync Gateway nodes

    • The same sync gateway configuration is applied across all nodes

    • All nodes are configured to run Replication Id 1

    • Sync Gateway automatically designates one of the three nodes to run Replication Id 1.

    • If a node goes down, Sync Gateway elects one of the remaining nodes to continue Replication Id 1.

    Sync Gateway runs Replication Id 1 on all nodes in the cluster.

    Example 2. Homogenous configuration with non-replicating node
    Scenario
    • The cluster comprises three Sync Gateway nodes.

    • Each node has the same sync gateway configuration, with one exception. The configuration on Node 3 has opted out of replication (sgreplicate_enabled=false)

    • All Three nodes are configured to run Replication Id 1.

    • Sync Gateway automatically designates either Node 1 or Node 2 to run the Replication Id 1.

    • If either Node 1 or Node 2 fails, Sync Gateway elects the non-failing node.

    • Sync Gateway runs Replication Id 1 on all nodes in cluster.

    • The system ignores the opt-out flag (sgreplicate_enabled).

    Example 3. Heterogenous configuration
    Scenario
    • The cluster comprises three Sync Gateway nodes

    • Both Node 1 and Node 2 are configured to run Replication Id 1

    • Node 3 is configured to run Replication Id 2 but not Replication Id 1

    • Sync Gateway automatically distributes Replication Id 1 and Replication Id 2 so that each runs on one of Node 1, Node 2 or Node 3, with no node running both replications simultaneously.

    • If any node fails whilst running either replication, Sync Gateway elects a non-failing node to continue that replication on. Where two nodes remain the node not running a replication will be chosen.

    • Sync Gateway runs Replication Id1 on Node 1 and Node 2 in the cluster

    • Sync Gateway runs Replication Id 2 on Node 3

    Note:

    • If Node 3 fails, then Replication Id 2 will not be continued on either of the remaining nodes as it is not configured on them

    • Similarly, if either or both of the other nodes (Node 1 and Node 2) fails, Node 3 will not be a candidate to run the corresponding replication.

    Example 4. Adding more nodes
    Scenario
    • The cluster comprises a single Sync Gateway node

    • Node 1 is configured to run Replication Id 1 and Replication Id 2

    • LATER . . . Node 2 is added to the cluster to run Replication Id 1 and Replication Id 2.

    • Sync Gateway designates Node 1 run both Replication Id 1 and Replication Id 2

    • LATER . . . when Node 2 is added . . .

      • Sync Gateway select one of the Node 1 replications to run on Node 2; let’s say it chooses Replication Id 2

      • Sync Gateway stops Replication Id 2 on Node 1

      • Sync Gateway starts Replication Id 2 on Node 2.

    • Sync Gateway designates Node 1 to run both Replication Id 1 and Replication Id 2

    • WHEN . . . Node 2 is added . . . Sync Gateway designates it to run both Replication Id 1 and Replication Id 2

    Example 5. Failing node
    Scenario
    • The cluster comprises three Sync Gateway nodes with a homogeneous configuration

    • All three nodes are configured to run Replication Id 1, Replication Id 2 and Replication Id 3

    • LATER . . . Node 3 goes down

    Sync Gateway automatically distributes the replications, one to each of the nodes

    • Lets assume the following distribution:

      • Node 1 runs Replication Id 1

      • Node 2 runs Replication Id 2

      • Node 3 runs Replication Id 3

    WHEN . . . Node 3 goes down . . . Sync Gateway elects either Node 1 or Node 2 to continue running Replication Id 3

    Sync Gateway runs all three replications (Replication Id 1 , Replication Id 2 and Replication Id 3) on all three nodes in the cluster (Node 1, Node 2 and Node 3)

    WHEN . . . Node 3 goes down . . . Node 1 and Node 2 continue to run Replication Id 1, Replication Id 2 and Replication Id 3

    Monitoring Node Distribution

    Use the _replicationStatus endpoint to access information about which replications are running on which nodes — see: _replicationStatus(replicationID) | _replicationStatus(replicationID)?action={action}

    This information is also collected and available in the log files.

    Conflict Resolution

    Automatic Conflict Resolution

    Inter-Sync Gateway replication supports automatic conflict resolution to resolve conflicting document changes.

    It delivers this by applying one of its built-in conflict resolver policiesglossary icon, which can be easily included in your own replications.

    The goal of automatic conflict resoluton is to return a winning revision based on the consistent application of the configured conflict resolver policyglossary icon.

    The default conflict resolver policy is to always returns a winner determined by the automatic conflict resolution policyglossary icon.

    For ENTERPRISE EDITION, a Custom Conflict Resolver policy is available, providing additional flexibility by allowing users to provide their own conflict resolution logic.

    Conflict Response on Active Replicator

    As soon as the active Sync Gateway database detects a conflict in a replicated document revision, it initiates its configured conflict resolver policy to determine a winning revision. This policy assesses the conflicting revisions and either determines the winning revision or returns an error if it fails while doing so.

    Conflict Response by Passive Replicator

    When a passive Sync Gateway database detects a conflict it responds to the active with a 409 response and rejects the revision. It is expected that the active Sync Gateway will pull the conflicting revision from the passive, resolve it on the active, and then subsequently push the resolved conflict back up.

    How Resolution Works

    Pull Replications

    For Pull replications the active Sync Gateway is responsible for detecting and resolving conflicts based on the configured conflict_resolution_type  — see configuration item: conflict_resolution_type.

    This is also how conflicts are handled when Couchbase Lite clients pull down documents to Sync Gateway.

    Note: Resolved conflicts are only transferred from active to passive Sync Gateways if a replication is setup between them.

    Push Replications
    Passive Sync Gateway

    The passive Sync Gateway will automatically detect and reject conflicting revisions being pushed to it.

    Note that conflicts are not resolved. The revision is rejected and the document returned — with a 409 Conflict — response to the active Sync Gateway.

    Active Sync Gateway

    It is the responsibility of the active sync Gateway to address rejected revisions in accordance with its specified conflict_resolution_type.

    This approach is the same as that adopted when Couchbase Lite clients push documents to Sync Gateway.

    Configure Conflict Resolution

    Invoke automatic conflict resolution by specifying the required conflict resolver policy in the replication definitionglossary icon. The specified policy is applied whenever a conflict is detected.

    Example 6. Using automatic conflict resolution
    • default

    • localWins

    • remoteWins

    "databases:"
      // other config as necessary
      "this_db:"
        // other config as necessary
        "sgreplicate_enabled": "true",
        "replications": [
            {
              "replication_id": "replication1",
              "direction": "push_and_pull",
              "continuous": true,
              "filter": "sync_gateway/bychannel",
              "query_params": [
                  "channel1",
                  "channel2"
              ],
              "conflict_resolution_type": "default",
              // other config as necessary
            }
        ]
    // other config as necessary
    "databases:"
      // other config as necessary
      "this_db:"
        // other config as necessary
        "sgreplicate_enabled": "true",
        "replications": [
            {
              "replication_id": "replication1",
              "direction": "push_and_pull",
              "continuous": true,
              "filter": "sync_gateway/bychannel",
              "query_params": [
                  "channel1",
                  "channel2"
              ],
              "conflict_resolution_type": "localWins",
              // other config as necessary
            }
        ]
    // other config as necessary
    "databases:"
      // other config as necessary
      "this_db:"
        // other config as necessary
        "sgreplicate_enabled": "true",
        "replications": [
            {
              "replication_id": "replication1",
              "direction": "push_and_pull",
              "continuous": true,
              "filter": "sync_gateway/bychannel",
              "query_params": [
                  "channel1",
                  "channel2"
              ],
              "conflict_resolution_type": "remoteWins",
              // other config as necessary
            }
        ]
    // other config as necessary

    Build a Conflict Resolution Policy [EE]

    This content relates only to ENTERPRISE EDITION

    Overview

    Custom conflict resolution is handled by the active Sync Gateway using a user-provided custom conflict resolverglossary icon. This Javascript function is embedded in the replication configuration.

    The predefined conflict resolver policies are also available as Javascript functions that you can call from within that custom_conflict_resolver function This is useful when you want to apply greater selectivity to the automatic conflict resolution process. For example, you want to apply a 'remote wins' policy only for a specific type of document - see the 'Use Policies' tab in Example 9.

    Conflict Resolution Approaches

    There are two ways to handle conflicts in your custom_conflict_resolver, you can either:

    • Choose a winning revision from among the conflicting revisions (see Example 9), or

    • Merge conflicting revision to create a new winning revision; losing revisions are tomb-stoned.

      However, users should avoid overly-complex resolver logic that may impact performance.

    Approaches to Error Handling

    Your custom conflict resolver function should not terminate the replication when it encounters exceptions or errors. Instead, you should log sufficient information to aid troubleshooting and recovery.

    For example, your custom conflict resolver function should:

    • Skip the document causing the issue

    • Log a suitable warning level message. Include at least the skipped document’s Id and the sequence Id of the revision in error.

    Refer to log files when troubleshooting conflict resolution errors, to identify the document id and revision sequence in error.

    Example 7. Some Error Scenarios and Recommended Resolutions
    Unexpected data in the remote document

    You should update the remote document to fix the issue. Doing so will cause replication of the update.

    Unexpected data in the local document

    You should update the local document to fix the issue. This will not trigger a pull-replication. Do a no-op-update [2] of the remote document, which will trigger replication and conflict resolution.

    Fault in conflict resolution javascript function

    Fix the Javascript logic and then either:

    • Do a no op update [2] of the remote document. This triggers a pull replication and subsequent conflict resolution.

    • Reset the replication (using _replicationstatus/reset endpoint). Not recommended as it introduces significant duplicate processing in re-syncing previously synced documents.

    Conflict Resolver Structure

    This example shows the basic structure of the conflict resolver function as it would be defined in the configuration file.

    Example 8. Conflict resolver structure
    
    "custom_conflict_resolver": "`function(conflict) { (1)
      //  . . .
      //  . . . application logic to determine winner
      //  . . .
      return conflict.LocalDocument;  (2)
    }`" (3)
    1 The conflict structure comprises both conflicting documents.
    type Conflict struct {
    	LocalDocument  Body `json:"LocalDocument"`
    	RemoteDocument Body `json:"RemoteDocument"`
    }
    LocalDocument

    This LocalDocument object encapsulates the body and metadata of the local conflicting document revision being replicated. Its content matches the JSON stored at the local Sync Gateway.

    RemoteDocument

    The RemoteDocument object, encapsulates the body and metadata of the remote conflicting document revision being replicated. Its content matches the JSON stored at the remote Sync Gateway.

    2 You should return one of:
    • conflict.LocalDocument

    • conflict.RemoteDocument

    • a new document body comprising the merged local and remote documents

    • a nil body, which will be resolved as a delete

    3 The conflict resolver function is enclosed by backticks (``)

    Sample Conflict Resolvers

    Example 9. Simple conflict resolvers
    • Use Built-in Policies

    • Nominate a Winner

    • Merge a Winner

    This example uses the built-in resolver functions to resolve the conflict based-on the document type.

    So, documents of type a-doc-type-1 are always resolved in favor of the remote revision. All other document types are resolved in accordance with the default resolver policy.

    "replications": [
      {
      "replication_id": "replication1",
      // other config as required
      "conflict_resolution_type": "custom",
      "custom_conflict_resolver": `
        function(conflict) {
          if  (conflict.LocalDocument.type == "a-doctype-1") &&
              (conflict.RemoteDocument.type == "a-doctype-1")
           {
             // Invoke the built in default resolver logic
             return defaultPolicy(conflict);
           }
          else {
            // Otherwise resolve in favor of remote document
              return conflict.RemoteDocument;
            }
        }
        `
      // other config as required
      }
    ]

    This example selects a winner based on relative priorities and builds a return response of its own rather than using either the localWins or remoteWins policy, although it does rely on the default resolver policy as a backstop.

    "replications": [
      {
        // . . . preceding replication details as required
      },
      {
        "replication_id": "replication2",
        // . . .   other config as required
        "conflict_resolution_type": "custom",
        "custom_conflict_resolver": `
          function(conflict) {
            // Custom conflict resolution policy based on priority
            if (conflict.LocalDocument.body.priority > conflict.RemoteDocument.body.priority) {
              // Choose a local winner
              // Optionally apply application logic to manipulate
              // the local object before returning it as the winner
              return conflict.LocalDocument;
            } else if (local.body.priority < remote.body.priority) {
                // Choose a remote winner
                // Optionally apply application logic to manipulate
                // the remote object before returning it as the winner
              return conflict.RemoteDocument;
              }; //end if
            } //end func()
            // Apply the default policy as a catch all
            return defaultPolicy(conflict);
        }` // end resolver property
      }, // end replication2
      {
        // . . . further replication details as required
      }
    ]
    // . . .   other config as required

    This example creates a winner by merging changes from the local and remote documents to create a new document object, which is returned as the winner.

    If both document.types are non-null and the local document.type is usedefault, the merge path is overridden and the default resolver policy is applied.

    "custom_conflict_resolver":`
      function(conflict) {
          if (  (conflict.LocalDocument.type != null) &&
                (conflict.RemoteDocument.type != null) &&
                (conflict.LocalDocument.type == "usedefault"))
          {
              console.log("Will use default policy");
              // Resolve using built-in policy
              return defaultPolicy(conflict);
          }
          else
          {
            // Merge local and remote docs
            var remoteDoc = conflict.RemoteDocument;
            console.log("full remoteDoc doc: "+JSON.stringify(remoteDoc));
            var localDoc = conflict.LocalDocument;
            console.log("full localDoc doc: "+JSON.stringify(localDoc));
            var mergedDoc = extend({}, localDoc, remoteDoc);
    
            console.log("full mergedDoc doc: "+JSON.stringify(mergedDoc));
            // Resolve using this merged doc as the winner
            return mergedDoc;
    
            function extend(target) {
                var sources = [].slice.call(arguments, 1);
                sources.forEach(function (source) {
                    for (var prop in source) {
                        target[prop] = source[prop];
                    }
                });
                return target;
            } // end function extend()
          } // end if
      }` // end function()

    Replication Initialization

    Introduction

    Replications are initialized by submitting a replication definitionglossary icon using either:

    • A 'JSON' configuration file (sync-gateway-config.json)

    • The Admin REST API, using a utility such as curl, or an application such as Postman.

    Wherever they are defined, the elements of a replication definition are the same, with the exception of these Admin REST API only elements:

    • adhoc — Use this to specify that the replication is ad hoc [3].

    • cancel — Use this to cancel on-going replications [3].

    Example 10. Replication Characteristics Highlights
    • Replication highlights

    • Running highlights

    • There are two types of replication: persistent and ad hoc (REST API only).

    • Replications of both types can run in one-shot or continuous replications modes.

    • All replications involve at least one local database.

    • Replications can be configured to purge documents when channel access is revoked (a removal notification is received).

    • Persistent continuous replications can be:

      • Reset — checkpoints can be reset to zero

      • Updated — only the parameter values provided in the PUT request body will be updated

    • Persistent and ad hoc replications can be:

      • Removed — only the replication_id is needed to delete ongoing continuous or one-shot replications.

    • ENTERPRISE EDITION only:

      • Replications can use delta-sync mode, whereby only the changed data-items are replicated.

    • Multiple identical replicators can be initiated on a Sync Gateway node provided each has a unique replication_Id.

    • inter-Sync Gateway replications introduced in Sync Gateway 2.8 as well as SG-Replicate can run on the same node, but you must ensure that they each have a different replication_id.

    • The user under which replication is being run must have read and write access to the data being replicated.

    • Exponential backoff when connection lost; this can be customized using the max_backoff_time configuration setting.

    • replications will continue trying to connect for 30 minutes following authentication failure (including user-invalid/doesn’t exist).

    • Running replications can be stopped. Stopped replications can be (re)Started.

    • If ALL the Sync Gateway nodes in a source or target Sync Gateway cluster go down in the middle of continuous replication, by default, the system should pick up from the last document that was successfully processed by both sides when the replication/cluster is restarted

    • REST ONLY

      • POST databases/{db}/_replication creates a replication using the {rep-id} specified in the body or if none specified, a unique UUID.

      • PUT databases/{db}/_replication/{rep-id} upserts replication {rep-id}.

    • ENTERPRISE EDITION only:

      • Replications are distributed even across all available Sync Gateway nodes and so are not guaranteed to run on their originating node.

      • If a multi-node Sync Gateway cluster loses a subset of sync gateway nodes, the remaining nodes continue replication uninterrupted IF they have been configured to handle the replication (continuous and one-shot replications).

    Replication Definition

    All replications are 'initialized' by a replication definitionglossary icon in the configuration file or Admin REST API and operate within the context of a local database.

    • Configured replications use the database.{db-name}.replications property to add a replication definition to a local database.

    • REST API replications specify the local database and replication identity in the API POST/PUT request. Providing the replication definition parameters in the request body as a JSON string.

    Both scenarios are covered in Example 11. It summarizes the replication definitionglossary icon elements[4], which are covered in more detail in Configuration Details.

    Database-level Settings

    A number of database-level options are also especially relevant to Inter-Sync Gateway Replication, including:

    • sgreplicate_enabled — use this ENTERPRISE EDITION setting to allow the database to participate in Inter-Sync Gateway Replications.

    • delta_sync — use this setting to enable delta-sync replication on the database, it must be set if you want to use delta-sync in your replication definition.

    • sgreplicate_websocket_heartbeat_secs — use this setting to override the default (5 minute) heartbeat interval for websocket ping frames for this database.

    • sync — use this setting to specify the sync function logic — this is an essential part of access-control.

    • unsupported.sgr_tls_skip_verify — use this unsupported option to make development an testing easier by skipping verification of TLS certificates.

    Replication-level Settings

    Example 11. Replication Definition
    • Summary of Parameters

    • Configured Example

    • REST API Example

    This table summarize all the available configurable items. It includes a link to a detailed description of each.

    Name and Link Summary

    REST API ONLY
    Use the Admin REST API’s adhoc parameter to specify that a replication is ad hoc rather than persistent.

    Use batch_size to specify the number of changes to be included in a single batch during replication.

    REST API ONLY
    Use the Admin REST API’s cancel parameter only when you want to want to cancel an existing active replication.

    Use conflict_resolution_type to specify how Sync Gateway should resolve conflicts. By default the automatic conflict resolution policy is applied.

    Use continuous to specify whether this replication will run continuously, or be one-shot.

    Use custom_conflict_resolver to provide the Javascript function used to resolve conflicts if "conflict_resolution_type": "custom".

    Use direction to specify the replication is push, pull or pushAndPull relative to this node.

    Use enable_delta_sync to specify use of delta sync for a replication.

    Use filter to specify the name of the function to be used to filter documents.

    Use max_backoff_time to specify the number of minutes Sync Gateway will spend trying to reconnect lost or unreachable remote targets.

    Use purge_on_removal to specify (per replication) whether removing a channel should trigger a purge.

    Use query_params to specify the key/value pairs to be passed to the filter named in filter.

    Use remote to specify the database endpoint on the remote Sync Gateway custer.

    Use replication_id to specify an identifying name for the replication.

    Use initial_state to specify the state in which to launch the replication.

    This is an example of a replication definition. Its purpose is to illustrate all configurable items in use and so should not be considered a working example.

    It creates a replication with the replication_ID of db1-rep-id1-pull-oneshot on a local database db1-local_, pulling data from a remote database db1-remote.

    "databases": {
     " db1": {                                                (1)
        "bucket":"db1",
        "server": "couchbase://cb-server",
        // ... other DB config ..
        "sgreplicate_enabled": true,                          (2)
        "replications":
          "db1-rep-id1-pull-oneshot":                         (3)
            "direction": "pull",                              (4)
            "remote": "https://example.com:4984/remote_db1",
            "user": "user1",                                  (5)
            "password": "password",
            "batch_size": 1000,                               (6)
            "conflict_resolution_type": "custom",             (7)
            "custom_conflict_resolver": "",                   (8)
            "continuous": false,                              (9)
            "enable_delta_sync": false,                       (10)
            "filter": "sync_gateway/bychannel",               (11)
            "query_params": ["channel.user1"]                 (12)
            "max_backoff_time": 5,                            (13)
            "purge_on_removal": false                         (14)
            "state": "running"                                (15)
        }
      }
    1 All replications are defined at database level within the context of a local database (e.g. DB1)
    2 Opt in to replication
    3 Define the replication_id
    4 Pull changes from the remote database at the specified url.
    5 Authenticate with the provided credentials. This user must have read and write access to both the local and remote databases.
    6 Batch together up to 1000 revisions at a time. This improve replication performance but consumes more memory resources.
    7 Apply a custom conflict resolution policy.
    8 Provide a working Javascript function to apply the required resolution policy.
    9 By setting continuous=false, we are creating a one-shot replication. We could also have omitted this parameter as it defaults to false.
    10 Don’t use delta-sync; the default behavior.
    11 Filter documents by channel.
    12 Replicate only those documents tagged with the channel names "user1".
    13 Wait no more than 5 minutes between retries after network failure; default behavior.
    14 Don’t purge following removal of a channel; the default behavior.
    15 Start the replicator immediately and on Sync Gateway node re(start);. We could also have omitted this parameter as this is the default behavior.

    This is an example of a replication definition as you might submit it to the Admin REST API.using curl. Its purpose is to illustrate all configurable items in use and so should not be considered a working example.

    It creates a replication with the replication_ID of db1-rep-id1-pull-oneshot on a local database db1-local_, pulling data from a remote database db1-remote.

    curl --location --request POST 'http://localhost:4985/db1-local/_replication/db1-rep-id1-pull-oneshot' \ (1)
    --header 'Content-Type: application/json' \
    --dataraw '{
    "replication_id": "db1-rep-id1-pull-oneshot" (2)
            "direction": "pull",                              (3)
            "remote": "https://example.com:4984/remote_db1",
            "user": "user1",                                  (4)
            "password": "password",
            "batch_size": 1000,                               (5)
            "conflict_resolution_type": "custom",             (6)
            "custom_conflict_resolver": "",                   (7)
            "continuous": false,                              (8)
            "enable_delta_sync": false,                       (9)
            "filter": "sync_gateway/bychannel",               (10)
            "query_params": ["channel.user1"]                 (11)
            "max_backoff_time": 5,                            (12)
            "purge_on_removal": false                         (13)
            "state": "running"                                (14)
      "adhoc": false, (15)
      "cancel": false (16)
    }'
    1 All replications take place at database level and in the context of a local database. Here we are setting the replication in the context of db1-local
    2 Define the replication_id
    3 Pull changes from the remote database at the specified url.
    4 Authenticate with the provided credentials. This user must have read and write access to both the local and remote databases.
    5 Batch together up to 1000 revisions at a time. This improve replication performance but consumes more memory resources.
    6 Apply a custom conflict resolution policy.
    7 Provide a working Javascript function to apply the required resolution policy.
    8 By setting continuous=false, we are creating a one-shot replication. We could also have omitted this parameter as it defaults to false.
    9 Don’t use delta-sync; the default behavior.
    10 Filter documents by channel.
    11 Replicate only those documents tagged with the channel names "user1".
    12 Wait no more than 5 minutes between retries after network failure; default behavior.
    13 Don’t purge following removal of a channel; the default behavior.
    14 Start the replicator immediately and on Sync Gateway node re(start);. We could also have omitted this parameter as this is the default behavior.
    15 Setting adhoc=false marks this as a persistent replication. The definition will survive Sync Gateway node restarts. This the default behaviour if this parameter is omitted.+
    16 Set cancel=true to cancel an initialized replication; otherwise you can omit this parameter.

    Generic Constraints

    Replication

    All active nodes in an active cluster must be running Sync Gateway version 2.8+.

    ENTERPRISE EDITION

    All replications are distributed evenly across available nodes. This means they cannot be guaranteed to run on the node from which they originate.

    Access rights

    The user running the replication must have read and write access to the data being replicated. This is not enforced by the system. Use your sync function to ensure a consistent approach is applied across all clusters.

    Mixing Inter-Sync Gateway Replication Versions

    Versions of inter-Sync Gateway replications pre- and post-2.8 can legitimately be in use at the same time, especially during transition. However, you should avoid initializing identical pre-2.8 (SG Replicate) and 2.8+ replications.

    Running Configured Replications

    Replications in the configuration file start automatically whenever Sync Gateway is (re)started. Unless you inhibit this by adding an "initial_state": "stopped" parameter to the replication definition — see: initial_state. You can manually start 'stopped' replication using Starting a replication.

    Example 12. Configured Replications — Continuous and One-shot
    • Continuous

    • One-shot

    //  . . . other configuration entries
    "db1-rep-id1-pull-cont":
      "replication_id": "db1-rep-id1-pull-cont",
      "direction": "pull",
      "continuous": true (1)
      "purge-on-removal": true,
      "remote": "http://user:password@example.com:4985/db1-remote",
      "filter":"sync_gateway/bychannel",
      "query_params": {
        "channels": ["channel1.user1"]
      }
    //  . . . other configuration entries
    1 Make this a continuous replication that remains running, listening for changes to process. Because it is also persistent, it will start automatically following Sync Gateway node restarts (state defaults to running).
    //. . . other configuration entries
    "db1-rep-id3-pull-oneshot":
    "replication_id": "db1-rep-id3-pull-oneshot", (1)
    "direction": "pull",
    "remote": "http://user:password@example.com:4985/db1-remote",
    "filter": "sync_gateway/bychannel",
    "query_params": { "channels": ["channel1.user1"] }
    //  . . . other configuration entries
    1 This a a one-shot replication because the continuous parameter defaults to false.

    Running Admin REST API Replications

    Replications initialized by sending a POST, or PUT, request to the _replication endpoint will start running automatically, unless the "initial_state": "stopped" parameter is specified. with a JSON object defining the replication parameters — as shown in Example 13.

    • You can run multiple replications simultaneously with different replication topologies, provided both databases being synchronized have the same sync function.

    You can submit requests using the curl utility (as in these examples) or an application such as Postman.

    Example 13. Submitting API Requests
    • Continuous Pull Replication

    • One-shot

    • Ad-hoc

    This example initializes a persistent, continuous, replication between a local database and one on a remote Sync Gateway instance.

    curl --location --request POST 'http://localhost:4985/db1-local/_replication/' \
    --header 'Content-Type: application/json' \
    --dataraw '{
      "replication_id": "db1-rep-id1-pull-cont",
      "direction": "pull",
      "continuous": true (1)
      "purge-on-removal": true,
      "remote": "http://user:password@example.com:4985/db1-remote",
      "filter":"sync_gateway/bychannel",
      "query_params": {
        "channels": ["channel1.user1"]
      }
    }'

    This example initializes a persistent, one-shot, replication between a local database and one on a remote Sync Gateway instance.

    The replication will run once, after a short delay to allow the Rest API to start. It will then run once after each Sync Gateway restart and-or when manually initiated using the _replicationStatus endpoint — see Manage Inter-Sync Gateway Replications.

    curl --location --request POST 'http://localhost:4985/db1-local/_replication/' \
    --header 'Content-Type: application/json' \
    --dataraw '{
    "replication_id": "db1-rep-id3-pull-oneshot", (1)
    "direction": "pull",
    "remote": "http://user:password@example.com:4985/db1-remote",
    "filter": "sync_gateway/bychannel",
    "query_params": { "channels": ["channel1.user1"] }
    }'
    1 This a a one-shot replication because the continuous parameter defaults to false.
    curl --location --request POST 'http://localhost:4985/db1-local/_replication/' \
    --header 'Content-Type: application/json' \
    --dataraw '{
      "replication_id": "db1-rep-id1-pull-adhoc",
      "adhoc": true, (1)
      "direction": "pull",
      "purge-on-removal": true,
      "remote": "http://user:password@example.com:4985/db1-remote",
      "filter":"sync_gateway/bychannel",
      "query_params": {
        "channels": ["channel1.user1"]
      }
    }'
    1 Run this replication as an ad hoc one. It will run once only, process all changes but not survive Sync Gateway restarts

    Replication Management

    Admin capabilities

    The Admin REST API provides two endpoints to assist in the monitoring, administration and management of replications. These enable users to examine, update, start and stop replications:

    You can use the endpoints manually or by using automation (for example, scripts or a container orchestration system such as Kubernetes).

    The available endpoints used for admin tasks are:

    • _replication — Retrieve, Update or Remove a replication definition

    • _replicationStatus — Stop, Start or Reset a replication

    This content relates only to ENTERPRISE EDITION

    Replications always run on the node on which they are configured. Users can only access replications on the node from which they make the request.

    Getting Replication Details

    You can view the current definition details of a replication. This includes replications configured in the JSON file and those initialized using the Admin REST API. You can do this for:

    • Individual replications (Example 14)

    • All replications defined for a specified database (Example 15)

    Replication information is returned a JSON object.

    Example 14. Get a replication definition
    • Request

    • Response

    curl --location --request GET 'http://localhost:4985/db1-local/_replication/db1-rep-id1' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      Payload in body
    {
      "db1-rep-id1": {
        "replication_id": "db1-rep-id1",
        "remote": "http://user1:****@example.com:4984/db1-remote",
        "direction": "pull",
        "purge_on_removal": true,
        "continuous": true,
        "filter": "sync_gateway/bychannel",
        "query_params": {
          "channels": [
            "channel.user1"
          ]
        },
        "assigned_node": "1de4994d136b982e"
      }
    }

    The following example retrieves definitions for all replications on a specified database, regardless of the node on which it was configured. The results are returned in an array; one entry per replication.

    Example 15. Get all replication definitions (for a database)
    • Request

    • Response

    curl --location --request GET 'http://localhost:4985/db1-local/_replication' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      Payload in body
    {
      "db1-rep-id1": {
          "replication_id": "db1-rep-id1",
          "remote": "http://user1:****@example1.com:4984/db1-remote",
          "direction": "pushAndPull",
          "conflict_resolution_type": "remoteWins",
          "purge_on_removal": true,
          "enable_delta_sync": true,
          "initial_state": "stopped",
          "continuous": true,
          "filter": "sync_gateway/bychannel",
          "query_params": {
              "channels": [
                  "channel.user1"
              ]
          },
          "batch_size": 1000,
          "assigned_node": "2c9b0d00a4e7c65a"
      },
      "db1-rep-id2": {
          "replication_id": "db1-rep-id2",
          "remote": "http://user1:****@example2.com:4984/db1-remote",
          "direction": "pushAndPull",
          "conflict_resolution_type": "remoteWins",
          "purge_on_removal": true,
          "enable_delta_sync": true,
          "max_backoff_time": 5,
          "initial_state": "running",
          "continuous": true,
          "filter": "sync_gateway/bychannel",
          "query_params": {
              "channels": [
                  "channel.user1"
              ]
          },
          "adhoc": true,
          "batch_size": 1000,
          "assigned_node": "2c9b0d00a4e7c65a"
      }
    }

    Updating a Replication

    You can update an existing replication’s definition, whether configured or initialized by Admin Rest API, by providing the details you want to change in an API call (Example 16). Changes will only be made to those parameters provided in the call.

    If you change the remote URI it must be to a valid URI.

    How do I change an existing replication’s definition details?

    Send a PUT request to the _replication endpoint. Specify just the changed items in the JSON body.

    Example 16. Update a replication’s details
    • Request

    • Response

    curl --location --request PUT 'http://localhost:4985/db1-local/_replication/db1-rep-id1 \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "direction": "push",
      "purge_on_removal":false, // set back to default
      "remote": "http://user1:password1@example.com:4984/db1-remote",
      "filter":"sync_gateway/bychannel",
      "query_params": {
        "channels":["channel.user1"]
      },
      "continuous": false
      }'

    A successful update will return a 200 response, with the following body:

    Success Response::
      Response Status 200 OK
      No payload

    If the replication_id in the body does not match that quoted in the URI you will see a 400 response as below.

    Bad Request Response::
      Response Status 400 Bad request
    
    {
        "error": "Bad Request",
        "reason": "Replication ID in body \"db1-rep-id1\" does not match request URI"
    }

    Removing a Replication

    Removing a replication will delete:

    • The persisted replication definition

    • All checkpoints associated with the replication

    • All replication status information associated with the replication

    To find the replication_id of an existing replication see Getting Replication Status Data.

    Action: Send a DELETE request to the replication endpoint specifying the replication_id to remove

    Example 17. Removing a replication
    • Request

    • Response

    curl --location --request DELETE 'http://localhost:4985/db2-local/_replication/db2-rep-id3' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      No payload

    Getting Replication Status Data

    Sync Gateway provides easy access to replication status data through the Admin REST API.

    You can obtain the replication status details for a specific replication, or for all replications across all nodes. This option can be useful, for example, to find any auto-generated replication_id details needed to enable further replication management activities.

    Replications always run on the node on which they are configured. Users can only access replications on the node from which they make the request.

    For more information on monitoring see: Monitoring Inter-Sync Gateway Replications

    Example 18. For a Single Replication

    This example targets a known replication-id and returns its status data.

    • Request

    • Response

    curl --location --request GET 'http://localhost:4985/db1/_replicationStatus/db1-rep-id2' \
    --header 'Content-Type: application/json' \
    [
      {
        "replication_id": "db1-rep-id2",
        "docs_read": 0,
        "docs_written": 10,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "running",
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "last_seq_pull": "8851",
        "last_seq_push": "10402"
    }
    ]
    Example 19. For All Replications

    This example targets all replications across all nodes. It filters the results using a query string — see: Monitoring Inter-Sync Gateway Replications for more on using this option.

    • Request

    • Response

    curl --location --request GET "http://localhost:4985/db1-local/_replicationStatus?activeOnly=false&includeConfig=true&localOnly=false&includeError=true" \ (1)
    --header 'Content-Type: application/json' \
    1 This example’s criteria selects replications with any status (including errors), on local and remote nodes. The returned status details also include replication definition details.
    [
      {
        "replication_id": "db1-rep-id1-pull",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "running",
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": { (1)
            "replication_id": "db1-rep-id1-pull",
            "cancel": true,
            "direction": "pull",
            "purge-on-removal": true,
            "remote": "http://user:****@example.com:4985/db1-remote",
            "filter":"sync_gateway/bychannel",
            "query_params": {
              "channels": ["channel1.user1"]
            },
            "continuous": true
        }
      },
      {
        "replication_id": "db1-rep-id2",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "stopped",  (2)
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": {
            "replication_id": "db1-rep-id2",
            "direction": "pull",
            "remote": "http://user:****@example.com:4985/db1-remote",
            "continuous": true
          }
      },
      {
        "replication_id": "db2-rep-id1",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "error", (3)
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": {
          "replication_id": "db2-rep-id1",
          "direction": "pull",
          "remote": "http://user:****@example2.com:4985/db2-remote",
          "continuous": true
        }
      }
    ]
    1 The configuration details included because includeConfig=true
    2 "Stopped" replications included because activeOnly=false
    3 "error" replications included because includeError=true

    Starting a Replication

    You can start a persistent or ad hoc replication not already in the running state. You need to specify the replication_id.

    If the replication is resetting it cannot be started until the reset is complete.

    Action: Send a POST request to the _replicationStatus endpoint with action=start

    Example 20. Start a replication
    • Request

    • Response

    curl --location --request PUT 'http://localhost:4985/db1-local/_replicationStatus/{{db1-rep-id}}?action=start' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      Payload in body
    {
      "replication_id": "db1-rep-id1",
      "docs_read": 0,
      "docs_written": 10,
      "doc_write_failures": 0,
      "doc_write_conflict": 0,
      "status": "running",
      "rejected_by_remote": 0,
      "rejected_by_local": 0,
      "last_seq_pull": "8851",
      "last_seq_push": "10402"
    }

    Stopping a Replication

    You can stop a persistent or ad hoc replication not already in the stopped state. You can use this, for example, to offline an edge cluster without waiting for a long replication to complete.

    Action: Send a POST request to the _replicationStatus endpoint with action=stop

    Example 21. Stopping replications
    • Request

    • Response

    curl --location --request PUT 'http://localhost:4985/db1-local/_replicationStatus/{{db1-rep-id1}}?action=stop' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      Payload in body
    {
      "replication_id": "db1-rep-id1",
      "docs_read": 0,
      "docs_written": 0,
      "doc_write_failures": 0,
      "doc_write_conflict": 0,
      "status": "stopped",
      "rejected_by_remote": 0,
      "rejected_by_local": 0
    }

    Resetting a Replication

    You can reset a persistent replication not in the running state. This can be useful to escape a system state where one or more documents have failed to sync but where resuming from previous synced checkpoint would skip over those documents. You need to specify the replication_id.

    If the replication is resetting it cannot be started until the reset is complete. The replication must be stopped before it can be reset.

    Action: Send a POST request to the _replicationStatus endpoint with action=reset

    Example 22. Reset a replication
    • Request

    • Response

    curl --location --request PUT 'http://localhost:4985/db1-local/_replicationStatus/{{db1-rep-id2}}?action=reset' \
    --header 'Content-Type: application/json' \
    Success Response::
      Response Status 200 OK
      Payload in body
    {
      "replication_id": "db1-rep-id2",
      "docs_read": 0,
      "docs_written": 0,
      "doc_write_failures": 0,
      "doc_write_conflict": 0,
      "status": "stopped",
      "rejected_by_remote": 0,
      "rejected_by_local": 0
    }

    Skipping TLS Certificate Verification

    Development and Testing Option ONLY

    This is an unsupported configuration option. It must not be used in a production environment. Its ongoing availability is not guaranteed.

    The configuration setting. database.this_db.unsupported.sgr_tls_skip_verify, can be used to skip the validation of TLS certificates, simplifying development and testing — see: Example 23 and the configuration item unsupported.sgr_tls_skip_verify.

    Example 23. Using sgr_tls_skip_verify
    {
      "databases": {
        "db1": {
          "server": "couchbase://localhost",
          "bucket": "db1",
          "username": "Administrator",
          "password": "password",
          "unsupported": {
            "sgr_tls_skip_verify": true
          },
          "replications": {
            "repl1": {
              "direction": "pushAndPull",
              "remote": "https://remotehost:4985/db1",
              "continuous": true
            }
          }
        }
      }
    }

    Replication Monitoring

    Retrieving Replication Status Data

    Sync Gateway provides easy access to replication status data through the Admin REST API.

    You can obtain the replication status details for a specific replication, or for all replications across all nodes. This option can be useful, for example, to find any auto-generated replication_id details needed to enable further replication management activities.

    Replications always run on the node on which they are configured. Users can only access replications on the node from which they make the request.

    Retrieving Status Data for a Specific Replication

    Use the Admin REST API endpoint replicationStatus to easily access replication status data for a specific replication id. Status data is returned regardless of the node the replication is running (or ran) on.

    Action: Send a GET request to the _replicationStatus endpoint with the required replication_id

    Example 24. Get status data for a specified replication

    This example returns status information for replication id 'db1-rep-id2'.

    • Request

    • Response

    curl --location --request GET 'http://localhost:4985/db1/_replicationStatus/db1-rep-id2' \
    --header 'Content-Type: application/json' \
    [
      {
        "replication_id": "db1-rep-id2",
        "docs_read": 0,
        "docs_written": 10,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "running",
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "last_seq_pull": "8851",
        "last_seq_push": "10402"
    }
    ]

    Retrieving Status Data for All Replications

    Use the Admin REST API’s _replicationStatus endpoint to access replication status data for all replications run, or running, on any node within the cluster. The JSON response comprises an array of results, one per replication.

    You can easily filter the results using the query string: ?activeOnly=false&includeConfig=true&localOnly=false&includeError=true`

    Available query string filters comprise:

    • activeOnly - return only active replications (default=false)

    • localOnly - return replications from the local node only (default=false)

    • includeError - return replications even if their status is "error" (default=true)

    • includeConfig - return the replication definition details (configuration) as well as the status data. This will include remote configuration definitions if localOnly=false (default=false)

    Action: Send a GET request to the _replicationStatus endpoint with an optional query string

    Example 25. Get status data for all replications meeting criteria

    This example retrieves status data, from across all nodes, for all replications that meet the specified criteria. The results are returned in an array; one entry per replication.

    • Request

    • Response

    curl --location --request GET "http://localhost:4985/db1-local/_replicationStatus?activeOnly=false&includeConfig=true&localOnly=false&includeError=true" \ (1)
    --header 'Content-Type: application/json' \
    1 This example’s criteria selects replications with any status (including errors), on local and remote nodes. The returned status details also include replication definition details.
    [
      {
        "replication_id": "db1-rep-id1-pull",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "running",
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": { (1)
            "replication_id": "db1-rep-id1-pull",
            "cancel": true,
            "direction": "pull",
            "purge-on-removal": true,
            "remote": "http://user:****@example.com:4985/db1-remote",
            "filter":"sync_gateway/bychannel",
            "query_params": {
              "channels": ["channel1.user1"]
            },
            "continuous": true
        }
      },
      {
        "replication_id": "db1-rep-id2",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "stopped",  (2)
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": {
            "replication_id": "db1-rep-id2",
            "direction": "pull",
            "remote": "http://user:****@example.com:4985/db1-remote",
            "continuous": true
          }
      },
      {
        "replication_id": "db2-rep-id1",
        "docs_read": 0,
        "docs_written": 0,
        "doc_write_failures": 0,
        "doc_write_conflict": 0,
        "status": "error", (3)
        "rejected_by_remote": 0,
        "rejected_by_local": 0,
        "config": {
          "replication_id": "db2-rep-id1",
          "direction": "pull",
          "remote": "http://user:****@example2.com:4985/db2-remote",
          "continuous": true
        }
      }
    ]
    1 The configuration details included because includeConfig=true
    2 "Stopped" replications included because activeOnly=false
    3 "error" replications included because includeError=true

    Retrieving Sync Gateway Statistics

    Sync Gateway maintains a comprehensive set of metrics covering performance and resource utilization.

    The statistics schema includes replication metrics collected on a per-replication basis. These can be especially useful in monitoring the health of Sync Gateway nodes. An increasingly important activity as deployments scale to support cloud-to-edge use cases.

    Access to this data is provided through the Admin REST API endpoint /_expvars.

    See: View Statistics and Metrics for a full description of the available metrics.


    1. This architecture is also known as ship-to-shore or hub-and-spoke.
    2. No-op update — refers to a change to the document body that has no impact on the app logic but will trigger an import by the Sync Gateway. One option could be to include a property used specifically for this purpose (i.e. a counter that can be incremented in response to conflict resolver errors).
    3. This parameter is not available in the configuration file.
    4. The definitions apply to configured and API replications).