Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

A newer version of this documentation is available.

View Latest

Sync with Couchbase Server

      +

      Use Sync Gateway to sync Couchbase Server changes securely from cloud to edge
      This content explains how Sync Gateway synchronizes document changes made through Couchbase SDKs and N1QL queries.

      Related Sync topics: Sync with Couchbase Lite | Inter-Sync Gateway Replication | Delta Sync | Resync

      Users of Couchbase Server 6.0 should ensure they have addressed the known issue (MB-41255) by upgrading to one of the recommended Couchbase Server versions (6.0.5, 6.5.2, or 6.6.1).

      The known issue can cause re-balance failures and/or failed replica writes of deleted or expired documents that use Xattrs.

      This impacts Sync Gateway deployments running with shared bucket access enabled, which use Xattrs for metadata storage.

      Introduction

      Sync Gateway uses the Shared Bucket Access and Import Processing features to synchronize document changes made through Couchbase Server with those made by Sync Gateway and Couchbase Lite clients, and vice versa [1].

      • Shared Bucket Access — this is mechanism that enables Couchbase Server’s SDK applications, N1QL Queries, Sync Gateway and Couchbase Lite applications to read and write to the same bucket simultaneously — see: Figure 1 and the configuration property $dbname.enable_shared_bucket_access

      • Import Processing — is the mechanism by which Sync Gateway becomes aware of non-Sync Gateway data changes and then obtains the mobile metadata it requires to replicate those changes — see: Import Processing.

      Shared Bucket Access

      shared bucket access
      Figure 1. Sync Gateway Data Access Paths

      Documents

      With bucket-sharing enabled, Couchbase Server documents can be inserted directly (using N1QL or SDKs) or by using Sync Gateway’s Public REST API.

      Sync Gateway [2] creates the metadata it needs by abstracting it from the SDK or N1QL applications reading and writing data directly to Couchbase Server buckets. It uses Couchbase Server XATTRs [3] to store that metadata into an external document fragment — see Extended Attributes (XATTR).

      The REST API will also include the following behavioral changes:

      • Purging — /{db}/_purge removes the document and its associated extended attributes

      • Updating — put /{db}/{docid} will tombstone the active revision

      Blobs and Attachments

      Couchbase Server SDK/N1QL

      Use Sync Gateway’s REST API’s /{db}/{docid}/{attachment} endpoints to manage attachments and blob data; you cannot use Couchbase Server SDKs to do this directly.

      Standard practice would be to create the document using the SDK and then associate its blobs/attachments using the Add/Update Attachment (/{db}/{docid}/{attachment}) endpoint. You can see a practical example in this blog post — https://blog.couchbase.com/store-sync-binary-data-attachments-blobs-couchbase-mobile

      Couchbase Lite Apps

      Couchbase Lite apps seamlessly handle blobs and attachments, see the appropriate platform examples here:

      Related Couchbase Lite content

      Android | C# | Java | Objective-C | Swift

      Using a WebApp

      Attachments can be accessed through Sync Gateway’s REST API using the /\{tkn-db}/\{doc}/\{attachment} endpoint.

      Tombstone Revisions

      Note that, with bucket-sharing enabled, tombstone revisionglossary icons are not retained indefinitely; they are purged based on the server’s metadata purge interval.

      To ensure tombstones are synced with clients, you should set the server’s metadata purge interval based on your expected replication frequency — see the $dbname.enable_shared_bucket_access reference.

      Accessing Sync Metadata

      Mobile metadata is not kept in the document, but in a system extended attribute (XATTR) in Couchbase Server.

      The N1QL query language [3] supports the ability to query these extended attributes (XATTRS) and hence the document’s sync metadata — see: Example 1.

      Example 1. Querying XATTRS-bsed sync metadata
      SELECT meta().xattrs._sync FROM `travel-sample` WHERE meta().id = "mydocId"
      The sync metadata is maintained internally by Sync Gateway and its structure can change at any time. It should not be used to drive business logic of applications. The direct use of the N1QL query is unsupported and must not be used in production environments.

      Enable Shared Bucket Access

      Shared bucket access is an opt-in feature. You can enable it without bringing down the entire Sync Gateway cluster  — see Example 2.

      Example 2. Enable Bucket-Sharing
      {
    "databases": {
        "db": {
            "bucket": "my-bucket",
            "username": "my-user",
            "password": "my-password",
            "server": "http://localhost:8091",
            "enable_shared_bucket_access": true, (1)
            "import_docs": true (2)
        }
    }
}
      1 The enable_shared_bucket_access property is used to enable bucket-sharing. It must be true on all nodes participating in such a configuration.
      2 The import_docs property is used to specify that a Sync Gateway node participates (exclusively) in Import Processing. The mechanism by which Sync Gateway incorporates changes to data buckets it shares with Couchbase Server — see: Import Processing.

      Reference

      Configuration file references:

      Import Processing

      The import process is a key part of mobile convergence. It is the means by which Sync Gateway becomes aware of non-Sync Gateway data changes and obtains the mobile metadata it requires to replicate changes.

      shared bucket access

      Any non-Sync Gateway change is eligible for import. The document is first run through the Sync Function to compute read security and routing, with the following differences:

      • The import is processed with an admin user context in the Sync Function, similar to writes made through the Sync Gateway Admin API. This means that requireAccess, requireUser and requireRole calls in the Sync Function are treated as no-ops.

      • During import, oldDoc is nil when the Sync Function is executed.

      You can specify a filter function using the import_filter property, which will only import specific documents.

      Use the Import+ log key to troubleshoot import processing issues in the logs.

      Configuration

      Note that import_docs only takes effect if the enabled_shared_bucket_access is set to true.

      ENTERPRISE EDITION

      The import_docs parameter defaults to true, implying that, by default, all nodes in a cluster participate in import processing. To exclude a node, set "import_docs": false.

      COMMUNITY EDITION

      The import_docs parameter defaults to false and must be explicitly set to true.

      The following table describes the key behavior differences between Community Edition and Enterprise Edition when import_docs is enabled, disabled or not set at all.

      enabled_shared_bucket_access import_docs Behavior (EE) Behavior (CE)

      true

      not set

      Assumes import: true by default

      Assumes import: false by default

      true

      false

      Node omitted from import processing (supported for workload isolation)

      Node omitted from import processing

      true

      true

      Node participates in import processing, and is assigned import partitions.

      Node performs import processing for all server mutations.

      false

      not set

      import docs is false by default

      import docs is false by default

      false

      true

      import docs property ignored, warning logged

      import docs property ignored, warning logged

      false

      false

      Import docs is false

      Import docs is false

      High Availability

      In Enterprise Edition, import processing work is sharded across all Sync Gateway nodes with import enabled. This implies that if one of the nodes fail, the failed shard is automatically picked up by the remaining nodes in the cluster. This way, you get High Availability of import processing.

      In Community Edition, there is no sharding of import across the nodes participating in the import processing. Each import node processes all server mutations.

      Workload Isolation

      As described in the table above, if import_docs is set to false, the node will not be participating in the import process. This configuration is specifically recommended for workload isolation: to isolate import nodes from the client-facing nodes. Workload isolation may be preferable in deployments with high write throughput.

      The following diagram shows an example architecture of two Sync Gateway nodes handling the incoming client connections (import_docs: false) and two nodes sharing the import processing (import_docs: true).

      workload isolation

      Reference

      The reference to the configuration properties can be found below.

      Related Content

      API Topics
      Reference
      Community
      1. Prior to Release 2.5, Server 5.0 all writes had to go through Sync Gateway, or had to use bucket shadowing to ensure that the security and replication metadata needed by mobile applications was preserved.
      2. As of Sync Gateway 1.5
      3. As of Couchbase Server 5.0