Data Sync using Sync Gateway
Description — Couchbase Lite JavaScript — Synchronizing data changes between local and remote databases using Sync Gateway
Related Content — Handling Data Conflicts
|
Code Snippets
All code examples are indicative only.
They demonstrate the basic concepts and approaches to using a feature.
Use them as inspiration and adapt these examples to best practice when developing applications for your platform.
|
Introduction
Couchbase Lite JavaScript provides API support for secure, bi-directional, synchronization of data changes between browser applications and a central server database. It does so by using a replicator to interact with Sync Gateway.
The replicator is designed to manage replication of documents and document changes between a source and a target database. For example, between a local Couchbase Lite database in the browser and a remote Sync Gateway database, which is ultimately mapped to a bucket in a Couchbase Server instance in the cloud or on a server.
This page shows sample code and configuration examples covering the implementation of a replication using Sync Gateway.
Your application runs a replicator (also referred to here as a client), which will initiate connection with a Sync Gateway (also referred to here as a server) and participate in the replication of database changes to bring both local and remote databases into sync.
| Configuring CORS settings for Sync Gateway is a prerequisite for enabling data syncronization with the JavaScript SDK. See CORS Configuration for more details. |
Replication Concepts
Couchbase Lite allows for one database for each application running in the browser. This database can contain one or more scopes. Each scope can contain one or more collections.
To learn about Scopes and Collections, see Databases
You can set up a replication scheme across these data levels:
- Database
-
The
_defaultcollection is synced. - Collection
-
A specific collection or a set of collections is synced.
As part of the syncing setup, the Gateway has to map the Couchbase Lite database to the database being synced on Capella.
Replication Protocol
Scheme
Couchbase Lite JavaScript uses a WebSocket-based replication protocol.
The replication URL must specify WebSockets as the URL scheme using ws:// (non-TLS) or wss:// (SSL/TLS) prefixes.
Always use wss:// (WebSocket Secure) in production for encrypted communication with Sync Gateway.
|
- Incompatibilities
-
Couchbase Lite’s replication protocol is incompatible with CouchDB-based databases and PouchDB’s replication protocol.
Scopes and Collections
Scopes and Collections allow you to organize your documents in Couchbase Lite.
When syncing, you can configure the collections to be synced.
The collections specified in the Couchbase Lite replicator setup must exist (both scope and collection name must be identical) on the Sync Gateway side, otherwise starting the Couchbase Lite replicator will result in an error.
During replication:
-
If Sync Gateway config (or server) is updated to remove a collection that is being synced, the client replicator will be offline and will be stopped after the first retry. An error will be reported.
-
If Sync Gateway config is updated to add a collection to a scope that is being synchronized, the replication will ignore the collection. The added collection will not automatically sync until the Couchbase Lite replicator’s configuration is updated.
Configuration Summary
You should configure and initialize a replicator for each Couchbase Lite database instance you want to sync. The following example shows the configuration and initialization process.
// Open database
const database = await Database.open({
name: 'myapp',
version: 1,
collections: {
tasks: {},
users: {}
}
});
// Configure replicator
const replicatorConfig = {
database: database, (1)
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: { (2)
tasks: { pull: {}, push: {} },
users: { pull: {}, push: {} }
},
credentials: { (3)
username: 'user@example.com',
password: 'password'
}
};
// Create replicator
const replicator = new Replicator(replicatorConfig); (4)
// Add status change listener
replicator.onStatusChange = (status) => { (5)
console.log('Replication status:', status.status);
if (status.error) {
console.error('Replication error:', status.error);
}
};
// Start replication
await replicator.run(); (6)
Notes on Example
| 1 | Configure the replicator with database and target URL |
| 2 | Configure collections to replicate |
| 3 | Set up authentication credentials |
| 4 | Create the replicator instance |
| 5 | Add status change listener |
| 6 | Start the replicator |
Configure
Configure Target
Initialize and define the replication configuration with local and remote database locations using the ReplicatorConfig object.
The configuration provides:
-
The local database to be synced
-
The server’s URL (including the port number and the name of the remote database to sync with)
-
The URL scheme for WebSocket URLs uses
ws:(non-TLS) orwss:(SSL/TLS) prefixes
const replicatorConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp', (1)
collections: {
tasks: { pull: {}, push: {} }
}
};
| 1 | Use wss:// to ensure TLS encryption (strongly recommended in production) |
Sync Mode
Define the direction and type of replication you want to initiate.
Use the ReplicatorConfig object’s replicatorType and continuous parameters to specify:
-
The type (or direction) of the replication:
-
pushAndPull- Bi-directional replication (default) -
pull- Pull-only replication -
push- Push-only replication
-
-
The replication mode:
-
Continuous - remaining active indefinitely to replicate changed documents (
continuous: true) -
Ad-hoc - a one-shot replication of changed documents (
continuous: false)
-
// Configure bi-directional continuous replication
const bidirectionalConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: {
pull: {}, // Pull changes from server
push: {} // Push changes to server
}
}
};
// Configure pull-only replication
const pullOnlyConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: {
pull: {} // Only pull, no push
}
}
};
// Configure push-only replication
const pushOnlyConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: {
push: {} // Only push, no pull
}
}
};
|
Unless there is a solid use-case not to, always initiate a single This prevents the replications generating the same checkpoint resulting in conflicts. |
Authentication
Sync Gateway or App Services users should be placed in the credentials section of the replication configuration.
To create a Basic Auth user in Sync Gateway, you use the Admin REST API. The Admin API allows you to create and manage users, roles, and access settings for a specific database.
To create a new user, send a POST request to {db}/_user/{username} with a minimal JSON body:
{
"name": "alice",
"password": "secret123"
}
This creates a user with Basic Authentication enabled, meaning you can authenticate against Sync Gateway using username and password in your Couchbase Lite for JavaScript replicator configuration.
const replicator = new Replicator({
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: { pull: {}, push: {} }
},
credentials: {
username: 'alice',
password: 'secret123'
},
continuous: true
});
await replicator.start();
Replication Filters
Replication Filters allow you to have control over the documents stored as the result of a push and/or pull replication.
Push Filter
The push filter allows an app to push a subset of a database to the server.
const replicatorConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: {
push: {
filter: (doc, flags) => { (1)
// Only push documents that are not deleted
if (flags.deleted) {
return false;
}
// Only push tasks that are completed
return doc.completed === true;
}
}
}
}
};
const replicator = new Replicator(replicatorConfig);
| 1 | The callback should follow the semantics of a pure function. Long running functions would slow down the replicator considerably. |
Pull Filter
The pull filter gives an app the ability to validate documents being pulled, and skip ones that fail.
| Pull replication filters are not a substitute for channels. Sync Gateway channels are designed to be scalable (documents are filtered on the server) whereas a pull replication filter is applied to a document once it has been downloaded. |
const replicatorConfig = {
database: database,
url: 'wss://sync-gateway.example.com:4984/myapp',
collections: {
tasks: {
pull: {
filter: (doc, flags) => {
// Skip deleted documents
if (flags.deleted) {
return false;
}
// Only pull tasks assigned to current user
return doc.assignedTo === 'currentUser@example.com';
}
}
}
}
};
const replicator = new Replicator(replicatorConfig);
Initialize
Start Replicator
Use the Replicator class constructor to initialize the replicator with the configuration you have defined.
You can optionally add change listeners before starting the replicator using start().
// Create replicator
const replicator = new Replicator(replicatorConfig); (1)
// Start replication
await replicator.run(); (2)
console.log('Replication started');
| 1 | Create the replicator with the configuration |
| 2 | Start the replicator |
Monitor
You can monitor a replication’s status by using change listeners and the replicator.status property.
This enables you to know when the replication is actively transferring data and when it has stopped.
Change Listeners
Use change listeners to monitor replication progress. You can add a replicator change listener at any point; it will report changes from the point it is registered.
|
Best Practice
Remove listeners when they’re no longer needed to prevent memory leaks
|
// Monitor replication status
replicator.onStatusChange = (status) => {
console.log('Replication status:', status.status);
// Check progress
if (status.pulledRevisions !== undefined) {
console.log('Documents pulled:', status.pulledRevisions);
}
if (status.pushedRevisions !== undefined) {
console.log('Documents pushed:', status.pushedRevisions);
}
// Handle errors
if (status.error) {
console.error('Replication error:', status.error);
// Check if it's a recoverable error
if (status.status === 'offline') {
console.log('Replicator is offline, will retry...');
} else if (status.status === 'stopped') {
console.log('Replicator stopped due to error');
}
}
// Handle different states
switch (status.status) {
case 'connecting':
console.log('Connecting to server...');
break;
case 'busy':
console.log('Actively transferring data');
break;
case 'idle':
console.log('Caught up with server');
break;
case 'stopped':
console.log('Replication stopped');
break;
}
};
Replicator Status
You can check the replicator status using the status property.
The status indicates whether the replicator is actively transferring data or if it has stopped.
The returned status structure comprises:
-
activity- stopped, offline, connecting, idle or busy -
progress-
completed- the total number of changes completed -
total- the total number of changes to be processed
-
-
error- the current error, if any
Replication States
State |
Meaning |
|---|---|
|
The replication is finished or hit a fatal error. |
|
The replicator is offline as the remote host is unreachable. |
|
The replicator is connecting to the remote host. |
|
The replication caught up with all the changes available from the server.
The |
|
The replication is actively transferring data. |
Monitor Document Changes
You can register listeners to monitor document replication.
// Monitor individual document replication
replicator.onDocuments = (collection, direction, documents) => {
console.log(`${direction} - ${documents.length} documents in ${collection.name}`);
for (const doc of documents) {
if (doc.error) {
console.error(`Error ${direction}ing ${doc.id}:`, doc.error);
} else if (doc.flags?.deleted) {
console.log(`Document ${doc.id} was deleted`);
} else {
console.log(`Document ${doc.id} ${direction}ed successfully`);
}
}
};
Documents Pending Push
You can check whether documents are waiting to be pushed in any forthcoming sync.
// Check if there are documents waiting to be pushed
const tasksCollection = await database.collection('tasks');
// Get pending document IDs for the collection
const pendingDocs = await replicator.getPendingDocumentIDs(tasksCollection);
if (pendingDocs.length > 0) {
console.log(`${pendingDocs.length} documents pending push`);
console.log('Pending document IDs:', pendingDocs);
} else {
console.log('No documents pending push');
}
// Check if a specific document is pending
const docId = 'task-123';
const isPending = await replicator.isDocumentPending(tasksCollection, docId);
console.log(`Document ${docId} is ${isPending ? 'pending' : 'not pending'}`);
Stop
Stopping a replication is straightforward using stop().
This initiates an asynchronous operation and so is not necessarily immediate.
// Stop the replicator
replicator.stop();
console.log('Replication stopped');
Error Handling
When the replicator detects a network error it updates its status depending on the error type (permanent or temporary) and returns an appropriate error code.
// Monitor for network errors
replicator.onStatusChange = (status) => {
if (status.error) {
const error = status.error;
console.error('Replication error:', error.message);
// Check error code if available
if (error.code) {
switch (error.code) {
case 401:
console.error('Unauthorized - check credentials');
break;
case 404:
console.error('Database not found on server');
break;
case 408:
console.log('Request timeout - will retry');
break;
case 429:
console.log('Too many requests - will retry');
break;
case 500:
case 502:
case 503:
case 504:
console.log('Server error - will retry');
break;
case 1001:
console.log('DNS resolution error - will retry');
break;
default:
console.error('Unexpected error code:', error.code);
}
}
// Check replicator status after error
if (status.status === 'stopped') {
console.log('Replicator stopped - permanent error');
} else if (status.status === 'offline') {
console.log('Replicator offline - will retry connection');
}
}
};
For permanent network errors (for example, 404 not found, or 401 unauthorized):
The replicator will stop permanently. It sets its status to STOPPED.
For recoverable or temporary errors: The replicator sets its status to OFFLINE, then:
-
If
continuous=trueit retries the connection indefinitely -
If
continuous=false(one-shot) it retries the connection a limited number of times