Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Data Sync using Sync Gateway

    +

    Description — Couchbase Lite Database Sync — Synchronizing data changes between local and remote databases using Sync Gateway
    Related Content — Handling Data Conflicts | Intra-device Data Sync |Certificate Pinning

    Replication Protocol

    Couchbase Mobile uses a replication protocol based on WebSockets.

    The replicator is designed to send documents from a source to a target database. The target can be one of the following:

    URLEndpoint

    To replicate data between a local Couchbase Lite database and remote Sync Gateway database or passive peer listener.

    DatabaseEndpoint

    To replicate data between two local Couchbase Lite databases to store data on secondary storage.

    MessageEndpoint

    To replicate with another Couchbase Lite database via a custom transportation protocol such iOS Multipeer Connectivity, Android WiFi Direct, Android NearByConnection, socket based transportation etc.

    Protocol Compatibility

    To use this protocol the replication URL should specify WebSockets as the URL scheme (see the Starting a Replication section below).

    Incompatibilities

    Couchbase Lite’s replication protocol is incompatible with CouchDB-based databases. And since Couchbase Lite 2.x+ only supports the new protocol, you will need to run a version of Sync Gateway that supports it — see: Compatibility.

    Legacy Compatibility

    Clients using Couchbase Lite 1.x can continue to use http as the URL scheme. Sync Gateway 2.0 will automatically use:

    • The 1.x replication protocol when a Couchbase Lite 1.x client connects through http://localhost:4984/db

    • The 2.0 replication protocol when a Couchbase Lite 2.0 client connects through ws://localhost:4984/db.

    Starting Sync Gateway

    Download Sync Gateway and start it from the command line with the configuration file created above.

    ~/Downloads/couchbase-sync-gateway/bin/sync_gateway

    For platform specific installation instructions, refer to the Sync Gateway installation guide.

    Starting a Replication

    Replication can be bidirectional, this means you can start a push/pull replication with a single instance. The replication’s parameters can be specified through the ReplicatorConfiguration object; for example, if you wish to start a push only or pull only replication.

    The following example creates a pull replication with Sync Gateway.

    class MyClass {
    var database: Database?
    var replicator: Replicator? (1)

    func startReplicator() {
        let url = URL(string: "ws://localhost:4984/db")! (2)
        let target = URLEndpoint(url: url)
        let config = ReplicatorConfiguration(database: database!, target: target)
        config.replicatorType = .pull

        self.replicator = Replicator(config: config)
        self.replicator?.start()
    }
}
    1 A replication is an asynchronous operation. To keep a reference to the replicator object, you can set it as an instance property.
    2 The URL scheme for remote database URLs has changed in Couchbase Lite 2.0. You should now use ws:, or wss: for SSL/TLS connections.

    To verify that documents have been replicated, you can:

    • Monitor the Sync Gateway sequence number returned by the database endpoint (GET /{tkn-db}/). The sequence number increments for every change that happens on the Sync Gateway database.

    • Query a document by ID on the Sync Gateway REST API (GET /{tkn-db}/{id}).

    • Query a document from the Query Workbench on the Couchbase Server Console.

    TLS

    The replicatorConfiguration class provides the acceptOnlySelfSignedServerCertificate method, which specifies whether the replicator ought to accept (only) self-signed certificates, rejecting all others.

    Two modes are supported:

    • False — The replicator verifies the server identity by using the trusted CA or using the pinned server certificate.

    • True — The replicator will accept all and only self-signed certificates. The replicator will reject non-self-signed certificates.

    public class ReplicatorConfiguration {
  public var acceptOnlySelfSignedServerCertificate: Bool { get set }
}

    WebSockets

    Couchbase Lite 2.0 uses WebSockets as the communication protocol to transmit data. Some load balancers are not configured for WebSocket connections by default (NGINX for example); so it might be necessary to explicitly enable them in the load balancer’s configuration (see Load Balancers).

    By default, the WebSocket protocol uses compression to optimize for speed and bandwidth utilization. The level of compression is set on Sync Gateway and can be tuned in the configuration file (replicator_compression).

    Replication Ordering

    To optimize for speed, the replication protocol doesn’t guarantee that documents will be received in a particular order. So we don’t recommend to rely on that when using the replication or database change listeners for example.

    Delta Sync

    Applies only to

    With Delta Sync only the changed parts of a Couchbase document are replicated. This can result in significant savings in bandwidth consumption as well as throughput improvements, especially when network bandwidth is typically constrained.

    Replications to a URLEndpoint (i.e Sync Gateway) automatically use delta sync if the databases.$db.delta_sync.enabled property is set to true in Sync Gateway’s configuration file.

    Replications to a DatabaseEndpoint automatically disable delta sync and replications to a MessageEndpoint automatically enable delta sync.

    Troubleshooting

    As always, when there is a problem with replication, logging is your friend. The following example increases the log output for activity related to replication with Sync Gateway.

    // Replicator
Database.setLogLevel(.verbose, domain: .replicator)
// Network
Database.setLogLevel(.verbose, domain: .network)

    Authentication

    By default, Sync Gateway does not enable authentication. This is to make it easier to get up and running with synchronization. You can enable authentication with the following properties in the configuration file:

    {
  "databases": {
    "mydatabase": {
      "users": {
        "GUEST": {"disabled": true}
      }
    }
  }
}

    To authenticate with Sync Gateway, an associated user must first be created. Sync Gateway users can be created through the POST /{tkn-db}/_user endpoint on the Admin REST API. Provided that the user exists on Sync Gateway, there are two ways to authenticate from a Couchbase Lite client: Basic Authentication or Session Authentication.

    Basic Authentication

    You can provide a user name and password to the basic authenticator class method. Under the hood, the replicator will send the credentials in the first request to retrieve a SyncGatewaySession cookie and use it for all subsequent requests during the replication. This is the recommended way of using basic authentication. The following example initiates a one-shot replication as the user username with the password password.

    let url = URL(string: "ws://localhost:4984/mydatabase")!
let target = URLEndpoint(url: url)
let config = ReplicatorConfiguration(database: database, target: target)
config.authenticator = BasicAuthenticator(username: "john", password: "pass")

self.replicator = Replicator(config: config)
self.replicator.start()

    Session Authentication

    Session authentication is another way to authenticate with Sync Gateway. A user session must first be created through the POST /{tkn-db}/_session endpoint on the Public REST API. The HTTP response contains a session ID which can then be used to authenticate as the user it was created for. The following example initiates a one-shot replication with the session ID that is returned from the POST /{tkn-db}/_session endpoint.

    let url = URL(string: "ws://localhost:4984/mydatabase")!
let target = URLEndpoint(url: url)
let config = ReplicatorConfiguration(database: database, target: target)
config.authenticator = SessionAuthenticator(sessionID: "904ac010862f37c8dd99015a33ab5a3565fd8447")

self.replicator = Replicator(config: config)
self.replicator.start()

    Replication Status

    The replication.status.activity property can be used to check the status of a replication. For example, when the replication is actively transferring data and when it has stopped.

    self.replicator.addChangeListener { (change) in
    if change.status.activity == .stopped {
        print("Replication stopped")
    }
}

    The following table lists the different activity levels in the API and the meaning of each one.

    State Meaning

    STOPPED

    The replication is finished or hit a fatal error.

    OFFLINE

    The replicator is offline as the remote host is unreachable.

    CONNECTING

    The replicator is connecting to the remote host.

    IDLE

    The replication caught up with all the changes available from the server. The IDLE state is only used in continuous replications.

    BUSY

    The replication is actively transferring data.
    The replication change object also has properties to track the progress (change.status.completed and change.status.total). But since the replication occurs in batches and the total count can vary through the course of a replication, those progress indicators are not very useful from the standpoint of an app user. Hence, these should not be used for tracking the actual progress of the replication.

    Replication Status and App Life Cycle

    The following diagram describes the status changes when the application starts a replication, and when the application is being backgrounded or foregrounded by the OS. It applies to iOS only.

    replicator states

    Additionally, on iOS, an app already in the background may be terminated. In this case, the Database and Replicator instances will be null when the app returns to the foreground. Therefore, as preventive measure, it is recommended to do a null check when the app enters the foreground, and to re-initialize the database and replicator if any of those is null.

    On other platforms, Couchbase Lite doesn’t react to OS backgrounding or foregrounding events and replication(s) will continue running as long as the remote system does not terminate the connection and the app does not terminate. It is generally recommended to stop replications before going into the background otherwise socket connections may be closed by the OS and this may interfere with the replication process.

    Handling Network Errors

    If an error occurs, the replication status will be updated with an Error which follows the standard HTTP error codes. The following example monitors the replication for errors and logs the error code to the console.

    self.replicator.addChangeListener { (change) in
    if let error = change.status.error as NSError? {
        print("Error code :: \(error.code)")
    }
}

    When a permanent error occurs (i.e., 404: not found, 401: unauthorized), the replicator (continuous or one-shot) will stop permanently. If the error is temporary (i.e., waiting for the network to recover), a continuous replication will retry to connect indefinitely and if the replication is one-shot it will retry for a limited number of times. The following error codes are considered temporary by the Couchbase Lite replicator and thus will trigger a connection retry.

    • 408: Request Timeout

    • 429: Too Many Requests

    • 500: Internal Server Error

    • 502: Bad Gateway

    • 503: Service Unavailable

    • 504: Gateway Timeout

    • 1001: DNS resolution error

    Replication Events

    You can choose to register for document updates during a replication.

    For example, the code snippet below registers a listener to monitor document replication performed by the replicator referenced by the variable replicator. It prints the document ID of each document received and sent.

    let token = self.replicator.addDocumentReplicationListener { (replication) in
    print("Replication type :: \(replication.isPush ? "Push" : "Pull")")
    for document in replication.documents {
        if (document.error == nil) {
            print("Doc ID :: \(document.id)")
            if (document.flags.contains(.deleted)) {
                print("Successfully replicated a deleted document")
            }
        } else {
            // There was an error
        }
    }
}

self.replicator.start()

    The following example stops the change listener with the token from the previous example.

    self.replicator.removeChangeListener(withToken: token)

    Document Access Removal Behavior

    When access to a document is removed on Sync Gateway, the document replication listener sends a notification with the AccessRemoved flag set to true and subsequently purges the document from the database.

    Custom Headers

    Custom headers can be set on the configuration object. And the replicator will send those header(s) in every request. As an example, this feature can be useful to pass additional credentials when there is an authentication or authorization step being done by a proxy server (between Couchbase Lite and Sync Gateway).

    let config = ReplicatorConfiguration(database: database, target: target)
config.headers = ["CustomHeaderName": "Value"]

    Channels

    By default, Couchbase Lite gets all the channels to which the configured user account has access. This behavior is suitable for most apps that rely on user authentication and the sync function to specify which data to pull for each user.

    Optionally, it’s also possible to specify a comma-separated list of channel names on Couchbase Lite’s replicator configuration object. In this case, the replication from Sync Gateway will only pull documents tagged with those channels.

    Replication Checkpoint Reset

    Replicators use checkpoints to keep track of documents sent to the target database. Without checkpoints , Couchbase Lite would replicate the entire database content to the target database on each connection, even though previous replications may already have replicated some or all of that content.

    This functionality is generally not a concern to application developers. However, if you do want to force the replication to start again from zero, use the checkpoint reset method replicator.resetCheckpoint() before starting the replicator.

    self.replicator.resetCheckpoint()
self.replicator.start()

    Replication Filters

    Replication Filters allow you to have quick control over which documents are stored as the result of a push and/or pull replication.

    Push Filter

    A push filter allows an app to push a subset of a database to the server, which can be very useful in some circumstances. For instance, high-priority documents could be pushed first, or documents in a "draft" state could be skipped.

    The following example filters out documents whose type property is equal to draft.

    let url = URL(string: "ws://localhost:4984/mydatabase")!
let target = URLEndpoint(url: url)

let config = ReplicatorConfiguration(database: database, target: target)
config.pushFilter = { (document, flags) in (1)
    if (document.string(forKey: "type") == "draft") {
        return false
    }
    return true
}

self.replicator = Replicator(config: config)
self.replicator.start()
    1 The callback should follow the semantics of a pure function. Otherwise, long running functions would slow down the replicator considerably. Furthermore, your callback should not make assumptions about what thread it is being called on.

    Pull Filter

    A pull filter gives an app the ability to validate documents being pulled, and skip ones that fail. This is an important security mechanism in a peer-to-peer topology with peers that are not fully trusted.

    Pull replication filters are not a substitute for channels. Sync Gateway channels are designed to be scalable (documents are filtered on the server) whereas a pull replication filter is applied to a document once it has been downloaded. 
    let url = URL(string: "ws://localhost:4984/mydatabase")!
let target = URLEndpoint(url: url)

let config = ReplicatorConfiguration(database: database, target: target)
config.pullFilter = { (document, flags) in (1)
    if (flags.contains(.deleted)) {
        return false
    }
    return true
}

self.replicator = Replicator(config: config)
self.replicator.start()
    1 The callback should follow the semantics of a pure function. Otherwise, long running functions would slow down the replicator considerably. Furthermore, your callback should not make assumptions about what thread it is being called on.
    Losing access to a document (via the Sync Function) also triggers the pull replication filter. Filtering out such an event would retain the document locally. As a result, there would be a local copy of the document disjointed from the one that resides on Couchbase Server. Further updates to the document stored on Couchbase Server would not be received in pull replications and further local edits could be potentially pushed, which would result in 409 errors since access has been revoked.

    Related Content

    How to
    Concepts
    Community Resources ...