Working with Replications
Description — Couchbase database replication and synchronization concepts
Related Content — Conflicts | Inter-Database Replication | Certificate Pinning
Replication
Couchbase Mobile uses a replication protocol based on WebSockets.
The replicator is designed to send documents from a source to a target database. The target can be one of the following:
- URLEndpoint
-
To replicate data between a local Couchbase Lite database and remote Sync Gateway database or passive peer listener.
- DatabaseEndpoint
-
To replicate data between two local Couchbase Lite databases to store data on secondary storage.
- MessageEndpoint
-
To replicate with another Couchbase Lite database via a custom transportation protocol such iOS Multipeer Connectivity, Android WiFi Direct, Android NearByConnection, socket based transportation etc.
Compatibility
| The new protocol is incompatible with CouchDB-based databases. And since Couchbase Lite 2 only supports the new protocol, you will need to run a version of Sync Gateway that supports it. |
To use this protocol with Couchbase Lite 2.0, the replication URL should specify WebSockets as the URL scheme (see the "Starting a Replication" section below). Mobile clients using Couchbase Lite 1.x can continue to use http as the URL scheme. Sync Gateway 2.0 will automatically use the 1.x replication protocol when a Couchbase Lite 1.x client connects through "http://localhost:4984/db" and the 2.0 replication protocol when a Couchbase Lite 2.0 client connects through "ws://localhost:4984/db".
Starting Sync Gateway
Download Sync Gateway and start it from the command line with the configuration file created above.
powershell & 'C:\Program Files (x86)\Couchbase\sync_gateway.exe' sync-gateway-config.json
bash ~/Downloads/couchbase-sync-gateway/bin/sync_gateway
For platform specific installation instructions, refer to the Sync Gateway installation guide.
Starting a Replication
Replication objects are now bidirectional, this means you can start a push/pull replication with a single instance.
The replication’s parameters can be specified through the ReplicatorConfiguration object;
for example, if you wish to start a push only or pull only replication.
The following example creates a pull replication with Sync Gateway.
public class MyClass
{
public Database Database { get; set; }
public Replicator Replicator { get; set; } (1)
public void StartReplication()
{
var url = new Uri("ws://localhost:4984/db"); (2)
var target = new URLEndpoint(url);
var config = new ReplicatorConfiguration(Database, target)
{
ReplicatorType = ReplicatorType.Pull
};
Replicator = new Replicator(config);
Replicator.Start();
}
}| 1 | A replication is an asynchronous operation.
To keep a reference to the replicator object, you can set it as an instance property. |
| 2 | The URL scheme for remote database URLs has changed in Couchbase Lite 2.0.
You should now use ws:, or wss: for SSL/TLS connections. |
Unresolved include directive in modules/csharp/pages/learn/csharp-replication.adoc - include::partial$verify-replication.adoc[]
Couchbase Lite 2.0 uses WebSockets as the communication protocol to transmit data. Some load balancers are not configured for WebSocket connections by default (NGINX for example); so it might be necessary to explicitly enable them in the load balancer’s configuration (see Load Balancers).
By default, the WebSocket protocol uses compression to optimize for speed and bandwidth utilization.
The level of compression is set on Sync Gateway and can be tuned in the configuration file (replicator_compression).
Troubleshooting
As always, when there is a problem with replication, logging is your friend. The following example increases the log output for activity related to replication with Sync Gateway.
Database.SetLogLevel(LogDomain.Replicator, LogLevel.Verbose);
Database.SetLogLevel(LogDomain.Network, LogLevel.Verbose);Authentication
By default, Sync Gateway does not enable authentication. This is to make it easier to get up and running with synchronization. You can enable authentication with the following properties in the configuration file:
{
"databases": {
"mydatabase": {
"users": {
"GUEST": {"disabled": true}
}
}
}
}To authenticate with Sync Gateway, an associated user must first be created.
Sync Gateway users can be created through the POST /{tkn-db}/_user endpoint on the Admin REST API.
Provided that the user exists on Sync Gateway, there are two ways to authenticate from a Couchbase Lite client: Basic Authentication or Session Authentication.
Basic Authentication
You can provide a user name and password to the basic authenticator class method.
Under the hood, the replicator will send the credentials in the first request to retrieve a SyncGatewaySession cookie and use it for all subsequent requests during the replication.
This is the recommended way of using basic authentication.
The following example initiates a one-shot replication as the user john with the password pass.
var url = new Uri("ws://localhost:4984/mydatabase");
var target = new URLEndpoint(url);
var config = new ReplicatorConfiguration(database, target);
config.Authenticator = new BasicAuthenticator("john", "pass");
var replicator = new Replicator(config);
replicator.Start();Session Authentication
Session authentication is another way to authenticate with Sync Gateway.
A user session must first be created through the POST /{tkn-db}/_session endpoint on the Public REST API.
The HTTP response contains a session ID which can then be used to authenticate as the user it was created for.
The following example initiates a one-shot replication with the session ID that is returned from the POST /{tkn-db}/_session endpoint.
var url = new Uri("ws://localhost:4984/mydatabase");
var target = new URLEndpoint(url);
var config = new ReplicatorConfiguration(database, target);
config.Authenticator = new SessionAuthenticator("904ac010862f37c8dd99015a33ab5a3565fd8447");
var replicator = new Replicator(config);
replicator.Start();Replication Status
The replication.Status.Activity property can be used to check the status of a replication.
For example, when the replication is actively transferring data and when it has stopped.
replicator.AddChangeListener((sender, args) =>
{
if (args.Status.Activity == ReplicatorActivityLevel.Stopped) {
Console.WriteLine("Replication stopped");
}
});The following table lists the different activity levels in the API and the meaning of each one.
| State | Meaning |
|---|---|
|
The replication is finished or hit a fatal error. |
|
The replicator is offline as the remote host is unreachable. |
|
The replicator is connecting to the remote host. |
|
The replication caught up with all the changes available from the server.
The |
|
The replication is actively transferring data. |
The replication change object also has properties to track the progress (change.status.completed and change.status.total).
But since the replication occurs in batches and the total count can vary through the course of a replication, those progress indicators are not very useful from the standpoint of an app user.
Hence, these should not be used for tracking the actual progress of the replication.
|
Replication Status and App Life Cycle
Couchbase Lite doesn’t react to OS backgrounding or foregrounding events and replication(s) will continue running as long as the remote system does not terminate the connection and the app does not terminate. It is generally recommended to stop replications before going into the background otherwise socket connections may be closed by the OS and this may interfere with the replication process.
Handling Network Errors
If an error occurs, the replication status will be updated with an Error which follows the standard HTTP error codes.
The following example monitors the replication for errors and logs the error code to the console.
replicator.AddChangeListener((sender, args) =>
{
if (args.Status.Error != null) {
Console.WriteLine($"Error :: {args.Status.Error}");
}
});When a permanent error occurs (i.e., 404: not found, 401: unauthorized), the replicator (continuous or one-shot) will stop permanently.
If the error is temporary (i.e., waiting for the network to recover), a continuous replication will retry to connect indefinitely and if the replication is one-shot it will retry for a limited number of times.
The following error codes are considered temporary by the Couchbase Lite replicator and thus will trigger a connection retry.
-
408: Request Timeout -
429: Too Many Requests -
500: Internal Server Error -
502: Bad Gateway -
503: Service Unavailable -
504: Gateway Timeout -
1001: DNS resolution error
Replication Events
You can choose to register for document updates during a replication.
For example, the code snippet below registers a listener to monitor document replication performed by the replicator referenced by the variable replicator. It prints the document ID of each document received and sent.
var token = replicator.AddDocumentReplicationListener((sender, args) =>
{
var direction = args.IsPush ? "Push" : "Pull";
Console.WriteLine($"Replication type :: {direction}");
foreach (var document in args.Documents) {
if (document.Error == null) {
Console.WriteLine($"Doc ID :: {document.Id}");
if (document.Flags.HasFlag(DocumentFlags.Deleted)) {
Console.WriteLine("Successfully replicated a deleted document");
}
} else {
// There was an error
}
}
});
replicator.Start();The following example stops the change listener with the token from the previous example.
replicator.RemoveChangeListener(token);Document Access Removal Behavior
When access to a document is removed on Sync Gateway, the document replication listener sends a notification with the AccessRemoved flag set to true and subsequently purges the document from the database.
Custom Headers
Custom headers can be set on the configuration object. And the replicator will send those header(s) in every request. As an example, this feature can be useful to pass additional credentials when there is an authentication or authorization step being done by a proxy server (between Couchbase Lite and Sync Gateway).
var config = new ReplicatorConfiguration(database, target)
{
Headers = new Dictionary<string, string>
{
["CustomHeaderName"] = "Value"
}
};Replication Checkpoint Reset
Replicators use checkpoints to keep track of documents sent to the target database. Without checkpoints, Couchbase Lite would replicate the entire database content to the target database on each connection, even though previous replications may already have replicated some or all of that content.
This functionality is generally not a concern to application developers.
However, if you do want to force the replication to start again from zero, use the checkpoint reset method replicator.resetCheckpoint() before starting the replicator.
// replicator is a Replicator instance
replicator.ResetCheckpoint();
replicator.Start();Replication Filters
Replication Filters allow you to have quick control over which documents are stored as the result of a push and/or pull replication.
Push Filter
A push filter allows an app to push a subset of a database to the server, which can be very useful in some circumstances. For instance, high-priority documents could be pushed first, or documents in a "draft" state could be skipped.
The following example filters out documents whose type property is equal to draft.
var url = new Uri("ws://localhost:4984/mydatabase");
var target = new URLEndpoint(url);
var config = new ReplicatorConfiguration(database, target);
config.PushFilter = (document, flags) => (1)
{
if (flags.HasFlag(DocumentFlags.Deleted)) {
return false;
}
return true;
};
// Dispose() later
var replicator = new Replicator(config);
replicator.Start();| 1 | The callback should follow the semantics of a pure function. Otherwise, long running functions would slow down the replicator considerably. Furthermore, your callback should not make assumptions about what thread it is being called on. |
Pull Filter
A pull filter gives an app the ability to validate documents being pulled, and skip ones that fail. This is an important security mechanism in a peer-to-peer topology with peers that are not fully trusted.
| Pull replication filters are not a substitute for channels. Sync Gateway channels are designed to be scalable (documents are filtered on the server) whereas a pull replication filter is applied to a document once it has been downloaded. |
var url = new Uri("ws://localhost:4984/mydatabase");
var target = new URLEndpoint(url);
var config = new ReplicatorConfiguration(database, target);
config.PullFilter = (document, flags) => (1)
{
if (document.GetString("type") == "draft") {
return false;
}
return true;
};
// Dispose() later
var replicator = new Replicator(config);
replicator.Start();| 1 | The callback should follow the semantics of a pure function. Otherwise, long running functions would slow down the replicator considerably. Furthermore, your callback should not make assumptions about what thread it is being called on. |
| Losing access to a document (via the Sync Function) also triggers the pull replication filter. Filtering out such an event would retain the document locally. As a result, there would be a local copy of the document disjointed from the one that resides on Couchbase Server. Further updates to the document stored on Couchbase Server would not be received in pull replications and further local edits could be potentially pushed, which would result in 409 errors since access has been revoked. |