March 23, 2025
+ 12

Initializing and running inter-Sync Gateway replication

Related topics: Overview | Run | Manage | Monitor | Conflict

Context Clarification

This content relates only to inter-Sync Gateway replication in Sync Gateway 2.8+. For documentation on pre-2.8 inter-Sync Gateway replication (also known as SG Replicate) — see the documentation for the appropriate release.

Introduction

Replications are initialized by submitting a replication definitionglossary icon using either:

  • A 'JSON' configuration file (sync-gateway-config.json)

  • The Admin REST API, using a utility such as curl, or an application such as Postman.

Wherever they are defined, the elements of a replication definition are the same, with the exception of the adhoc Admin REST API endpoint used to specify that the replication is ad hoc [1].

Example 1. Replication Characteristics Highlights
  • There are two types of replication: persistent and ad hoc (REST API only).

  • Replications of both types can run in one-shot or continuous replications modes.

  • All replications involve at least one local database.

  • Replications can be configured to purge documents when channel access is revoked (a removal notification is received).

  • Persistent continuous replications can be:

    • Reset — a checkpointglossary icon can be reset to zero

    • Updated — only the parameter values provided in the PUT request body will be updated

  • Persistent and ad hoc replications can be:

    • Removed — only the replication_id is needed to delete ongoing continuous or one-shot replications.

  • ENTERPRISE EDITION only:

    • Replications can use delta-sync mode, whereby only the changed data-items are replicated.

Replication Definition

All replications are 'initialized' by a replication definitionglossary icon in the configuration file or Admin REST API and operate within the context of a local database.

  • Configured replications use the database.{db-name}.replications property to add a replication definition to a local database.

  • REST API replications specify the local database and replication identity in the API POST/PUT request. Providing the replication definition parameters in the request body as a JSON string.

Both scenarios are covered in Example 2. It summarizes the replication definitionglossary icon elements[2], which are covered in more detail in Database Configuration.

Database-level Settings

A number of database-level options are also especially relevant to Inter-Sync Gateway Replication, including:

  • sgreplicate_enabled — use this ENTERPRISE EDITION setting to allow the database to participate in Inter-Sync Gateway Replications.

  • database.delta_sync — use this setting to enable delta-sync replication on the database, it must be set if you want to use delta-sync in your replication definition.

  • sgreplicate_websocket_heartbeat_secs — use this setting to override the default (5 minute) heartbeat interval for websocket ping frames for this database.

  • database.sync — use this setting to specify the sync function logic — this is an essential part of access-control.

  • unsupported.sgr_tls_skip_verify — use this unsupported option to make development an testing easier by skipping verification of TLS certificates.

Replication-level Settings

Example 2. Replication Definition

This table summarize all the available configurable items.

Data schema for the replication model

Name Description Schema

adhoc
optional

" About

Use the Admin REST API’s adhoc parameter to specify that a replication is ad hoc rather than persistent.

Behavior

Ad hoc replications behave the same as normal replications, but they are automatically removed when their status changes to stopped. This will usually be on completion, but may also be as a result of user action.

Constraints

This parameter is NOT available to configured replications; only those initialized using the Admin REST API."
Default : false

boolean

batch_size
optional

About

Use the optional batch_size property to specify the number of changes to be included in a single batch during replication.

integer

cancel
optional

About

Use this parameter on,y when you want to want to cancel an existing active replication.

Constraints

* This parameter is NOT available in configured replications; only those initialized using the Admin REST API.

* NOTE that the body of the request must be the same as the replication’s replication definition for the cancellation request to be honoured. For example, if you requested continuous replication, the cancellation request must also contain the continuous field.
Default : false

boolean

conflict_resolution_type
optional

About

The conflict_resolution_type property defines the conflict resolution policy that Sync Gateway applies when resolving conflicting revisions.

The default behavior is that automatic conflict resolution policy is applied.

Valid options - default - localWins - remoteWins - custom

Behavior

* default - Selecting default applies the following conflict resolution policy * Deletes always win (the delete with longest revision history wins if both revisions are deletes) * The revision with the longest revision history wins (so, the one with most changes and consequently the highest revision Id). * localWins - Selecting localWins will result in local revisions always being the winner in any conflict.

* remoteWins - Selecting remoteWins will result in remote revisions always being the winner in any conflict. * custom - Selecting custom specifies that you want to handle conflict resolution with your own application logic. You must provide this logic as a Javascript function by specifying it in using the custom-conflict-resolver parameter.

Example

---- "conflict_resolution_type":"remoteWins" ----

Constraints

* Replications created prior to version 2.8 will default to default.
Default : "default"

string

continuous
optional

About

The continuous property specifies whether this replication will run in continuous mode.

Behavior

* continuous=true– In continuous mode, changes are immediately synced in accordance with the replication definition. * continuous=false– Detected changes are synced in accordance with the replication definition. The replication ceases once all revisions are processed.

Constraints

* Optional for stops and removes
Default : false

boolean

custom_conflict_resolver
optional

About

The optional custom_conflict_resolver property specifies the Javascript function that will be used to resolve conflicts, if the custom conflict resolution type is specified in the conflict_resolution_type.

Options

The property is mandatory when conflict_resolution_type=custom and will be ignored in all other cases.

Using

Provide the required logic in a Javascript function, as a string within backticks (see also the description for the sync function`.

The function takes one parameter struct representing the conflict and comprising - the document id - the local document - the remote document

The function returns a document struct representing the winning revision.

Example

---- "custom_conflict_resolver":` function(conflict) { console.log("full remoteDoc doc: "+JSON.stringify(conflict.RemoteDocument)); return conflict.RemoteDocument; }` ----

Constraints

Using complex custom_conflict_resolver functions can noticeably degrade performance. Use a built-in resolver whenever possible.
Default : "none"

string

direction
optional

About

The mandatory direction property specifies whether the replication is push, pull or pushAndPull relative to this node.

The property value is referenced by the remote property.

Behavior

* pull - changes are pulled from the remote database * push - changes are pushed to the remote database * pushAndPull - changes are both pushed-to and pulled-from the remote database

Constraints

Replications created prior to version 2.8 derive their direction from the source/target url of the replication.

string

enable_delta_sync
optional

About

The optional enable_delta_sync parameter turns on delta sync for a replication. It works in conjunction with the database level setting delta_sync.enabled.

Options

* "enable_delta_sync": true, the replication can use delta sync (depending on delta_sync.enabled setting) * "enable_delta_sync": false, the replication cannot use delta sync

Behavior

The optional enable_delta_sync parameter works in conjunction with the database level delta_sync.enabled setting, to determine whether this replication uses delta sync.

* If "delta_sync.enabled": true for both databases involved in the replication, then this parameter enables or disables its use for this specific replication. * In all other cases it has no effect and the replication runs without delta-sync.

Constraints

* Applies ONLY to Enterprise Edition deployments. * Depends upon the setting of the database level parameter delta_sync.enabled * Replications created prior to version 2.8 must run with "enable_delta_sync": false * Push replications will not use Delta Sync when pushing to a pre-2.8 target
Default : false

boolean

filter
optional

About

Use the optional filter`property to defines the function to be used to filter documents.

Options

A common value used when replicating from Sync Gateway is `sync_gateway/bychannel. This option limits the pull replication to a specific set of channels. You can specify the required channels using query_params.

Behavior

Works in conjunction with query_params to control the documents processed by the replication.

Example

---- "filter":"sync_gateway/bychannel" ----

Constraints

OPTIONAL for stops and removes (even if defined during creation)

string

initial_state
optional

About

The optional initial_state property is used to specify that the replication must be launched in 'Stopped' mode

Behavior

All replications are configured to start on Sync Gateway launch. So, if omitted, the state defaults to 'Running'.

Constraints*

Replications created prior to version 2.8 will all default to a state of 'Running'.
Default** : "Running"

string

max_backoff_time
optional

The *max_backoff_time*property specifies the time-period (in minutes) during which Sync Gateway will attempt to reconnect lost or unreachable remote targets.

On disconnection, Sync Gateway will do an exponential backoff up to the specified value, after which it will attempt to reconnect indefinitely every max_backoff_time minutes.

If a zero value is specified, then Sync Gateway will do an exponential backoff up to an interval of five minutes before stopping the replication.

NOTE - this value defaults to five minutes for replications created prior to version 2.8.

integer

password
optional

About

Use password to provide the login password value for the accredited user running this replication.

Behavior

These details are used to authenticate credentials and approve access to data.

Once provided and recorded, the password data is redacted and will not be displayed in either the configuration file or Admin REST API. A string of * will be displayed in its place.
Default
* : "mandatory"

string

purge_on_removal
optional

About

The optional purge_on_removal property specifies, per replication, whether the removal of a channel triggers a purge.

Options - true or false - Default = false - Document removals are ignored by receiving end

Behavior

If purge_on_removal=false, then the removal of channels is ignored (not purged) by the receiving end.

Constraints

* Applies only to PULL replications, including the PULL portion of a PUSHANDPULL replication.

* Replications created prior to version 2.8 must be run with purge_on_removal=false.
Default : false

boolean

query_params
optional

About

The query_params property defines a set of key/value pairs used in the query string of the replication.

Behavior

This property works in conjunction with filters and channels to provide routing.

Using

You can use query_params’ channels function to pull from a specific set of `channels. To do so, you would also need to set the filter to sync_gateway/bychannels.

Example

[source,json] ---- "filter":"sync_gateway/bychannel", "query_params": { "channels":["channel.user1"] }, ----

Constraints

OPTIONAL for stops and removes (even if defined during creation)

< string > array

remote
optional

About

The remote property represents the endpoint of a database for the remote Sync Gateway. That is, it identifies the remote Sync Gateway database that is the subject of this replication’s push, pull or pushAndPull action.

Typically the endpoint will include URI, Port and Database name elements.

You can also include user credentials in the URL, in the form <username>:<password>. The credentials relate to an existing Sync Gateway user on the remote server.

Example `"remote": "http://user:password@example.com:4985/db1-remote"; `

Format

* a string containing a valid URL for a (remote) Sync Gateway database. * an object whose url property contains the Sync Gateway database URL.

Behavior

Dependent upon setting of direction.

If direction is : - pull, 'remote' defines the remote cluster from which data is pulled - push, 'remote' defines the remote cluster to which data is pushed - pushAndPull, 'remote' defines the push configuration.

Example

[source,json] ---- "remote": "http://www.example.com:4984/sample-database", ----

string

replication_id
optional

About

The replication_id property specifies either:

* For NEW replications, the ID to be assigned to the the replication. If no replication_id is specified, Sync Gateway will assign a random UUID to new replications.

* For existing replications, this is the ID of the required replication.

* If cancel=true, this is the id of the active replication task to be cancelled.

Constraints

If this is specified in the body of a POST or PUT request then it must be the same value as specified in the request URL.

string

username
optional

About

Use username to provide the name of the accredited user running this replication.

Behavior

These details are used to authenticate credentials and approve access to data

Once provided and recorded, the username data is redacted and will not be displayed in either the configuration file or Admin REST API. A string of * will be displayed in its place.
Default
* : "Mandatory"

string

Generic Constraints

Replication

All active nodes in an active cluster must be running Sync Gateway version 2.8+.

ENTERPRISE EDITION

All replications are distributed evenly across available nodes. This means they cannot be guaranteed to run on the node from which they originate.

Access rights

The user running the replication must have read and write access to the data being replicated. This is not enforced by the system. Use your sync function to ensure a consistent approach is applied across all clusters.

Mixing Inter-Sync Gateway Replication Versions

Versions of inter-Sync Gateway replications pre- and post-2.8 can legitimately be in use at the same time, especially during transition. However, you should avoid initializing identical pre-2.8 (SG Replicate) and 2.8+ replications.

Running Configured Replications

Replications in the configuration file start automatically whenever Sync Gateway is (re)started. Unless you inhibit this by adding an "initial_state": "stopped" parameter to the replication definition — see: initial_state. You can manually start 'stopped' replication using Starting a replication.

Example 3. Configured Replications — Continuous and One-shot
//  . . . other configuration entries
"db1-rep-id1-pull-cont":
"replication_id": "db1-rep-id1-pull-cont",
"direction": "pull",
"continuous": true (1)
"purge-on-removal": true,
"remote": "http://user:password@example.com:4985/db1-remote",
"filter":"sync_gateway/bychannel",
"query_params": {
  "channels": ["channel1.user1"]
}
//  . . . other configuration entries
1 Make this a continuous replication that remains running, listening for changes to process. Because it is also persistent, it will start automatically following Sync Gateway node restarts (state defaults to running).
2 The remote URL can also include the credentials for an existing Sync Gateway user on the remote server.

Running Admin REST API Replications

Replications initialized by sending a POST, or PUT, request to the _replication endpoint will start running automatically, unless the "initial_state": "stopped" parameter is specified. with a JSON object defining the replication parameters — as shown in Example 4.

You can run multiple replications simultaneously with different replication topologies, provided both databases being synchronized have the same sync function.

You can submit requests using the curl utility (as in these examples) or an application such as Postman.

Example 4. Submitting API Requests

This example initializes a persistent, continuous, replication between a local database and one on a remote Sync Gateway instance.

json
curl --location --request POST 'http://localhost:4985/db1-local/_replication/' \ --header 'Content-Type: application/json' \ --dataraw '{ "replication_id": "db1-rep-id1-pull-cont", "direction": "pull", "continuous": true (1) "purge-on-removal": true, "remote": "http://user:password@example.com:4985/db1-remote", "filter":"sync_gateway/bychannel", "query_params": { "channels": ["channel1.user1"] } }'


1. This parameter is not available in the configuration file.
2. The definitions apply to configured and API replications).