Mobile-Server Replication

    +

    Description — Sync Gateway replication keeps distributed database changes in sync
    Abstract — This content describes how Sync Gateway synchronizes document changes made through Couchbase SDKs and N1QL queries.
    Related Content — Configuration Details | Admin REST API

    Shared Bucket Access

    Sync Gateway uses the Shared Bucket Access feature to synchronize document changes made through Couchbase SDKs and N1QL queries with Couchbase Lite clients, and vice versa. The following diagram shows different client types (mobile, web, desktop) writing data to the same bucket in Couchbase Server.

    shared bucket access

    In previous releases, you either had to ensure all writes happened through Sync Gateway, or had to set up bucket shadowing to ensure that the security and replication metadata needed by mobile applications was preserved.

    As of Sync Gateway 1.5, the metadata created by the Sync Gateway is abstracted from applications reading and writing data directly to Couchbase Server. Sync Gateway 1.5 utilizes a new feature of Couchbase Server 5.0 called XATTRs (eXtended ATTRibutes) to store that metadata into an external document fragment.

    Enable Shared Bucket Access

    Shared bucket access is an opt-in feature, and can be enabled without bringing down the entire Sync Gateway cluster.

    The getting started page includes step-by-step instructions to:

    1. Install Couchbase Server 5.0 or above.

    2. Configure Couchbase Server with RBAC.

    3. Install Sync Gateway with a configuration file that enables Mobile-Web data synchronization.

      Enable mobile web data synchronization
      {
          "databases": {
              "db": {
                  "bucket": "my-bucket",
                  "username": "my-user",
                  "password": "my-password",
                  "server": "http://localhost:8091",
                  "enable_shared_bucket_access": true, (1)
                  "import_docs": true (2)
              }
          }
      }
      1 The enable_shared_bucket_access property is used to enable Web-Server data sync and must be true on all nodes participating in such a configuration.
      2 The import_docs property is used to specify that a Sync Gateway node should (only) import document changes using the [Import process].

    Once enabled, documents can be inserted on the Server directly (using N1QL or SDKs) or through the Sync Gateway REST API. The mobile metadata is no longer kept in the document, but in a system extended attribute in Couchbase Server.

    Import Process

    The import process is a key part of mobile convergence.

    It is the means by which Sync Gateway becomes aware of non-Sync Gateway data changes and obtains the mobile metadata it requires to replicate changes. Any non-Sync Gateway change is eligible for import.

    The document is first run through the Sync Function to compute read security and routing, with the following differences:

    • The import is processed with an admin user context in the Sync Function, similar to writes made through the Sync Gateway Admin API. This means that requireAccess, requireUser and requireRole calls in the Sync Function are treated as no-ops.

    • During import, oldDoc is nil when the Sync Function is executed.

    You can specify a filter function using the import_filter property, which will only import specific documents.

    Use the Import+ log key to troubleshoot import processing issues in the logs.

    Configuration

    Note that import_docs only takes effect if the enabled_shared_bucket_access is set to true.

    ENTERPRISE EDITION

    The import_docs parameter defaults to true, implying that, by default, all nodes in a cluster participate in import processing. To exclude a node, set "import_docs": false.

    COMMUNITY EDITION

    The import_docs parameter defaults to false and must be explicitly set to true.

    The following table describes the key behavior differences between Community Edition and Enterprise Edition when import_docs is enabled, disabled or not set at all.

    enabled_shared_bucket_access import_docs Behavior (EE) Behavior (CE)

    true

    not set

    Assumes import: true by default

    Assumes import: false by default

    true

    false

    Node omitted from import processing (supported for workload isolation)

    Node omitted from import processing

    true

    true

    Node participates in import processing, and is assigned import partitions.

    Node performs import processing for all server mutations.

    false

    not set

    import docs is false by default

    import docs is false by default

    false

    true

    import docs property ignored, warning logged

    import docs property ignored, warning logged

    false

    false

    Import docs is false

    Import docs is false

    High Availability of Import Processing

    In Enterprise Edition, import processing work is sharded across all Sync Gateway nodes with import enabled. This implies that if one of the nodes fail, the failed shard is automatically picked up by the remaining nodes in the cluster. This way, you get High Availability of import processing.

    In Community Edition, there is no sharding of import across the nodes participating in the import processing. Each import node processes all server mutations.

    Workload Isolation

    As described in the table above, if import_docs is set to false, the node will not be participating in the import process. This configuration is specifically recommended for workload isolation: to isolate import nodes from the client-facing nodes. Workload isolation may be preferable in deployments with high write throughput.

    The following diagram shows an example architecture of two Sync Gateway nodes handling the incoming client connections (import_docs: false) and two nodes sharing the import processing (import_docs: true).

    workload isolation

    Metadata Purge Interval

    When $dbname.enable_shared_bucket_access is enabled, tombstone revisionglossary icon are not retained indefinitely. They are purged based on the server’s metadata purge interval. To ensure tombstones are replicated to clients, you should set the server’s metadata purge interval based on your expected replication frequency (see the $dbname.enable_shared_bucket_access reference).

    Reference

    The reference to the configuration properties can be found below.

    FAQ

    How do I query the document’s sync metadata?

    Starting in Couchbase Server 5.5, the N1QL query language supports the ability to select extended attributes (xattrs) where the document’s sync metadata is stored. The following query shows an example of that feature.

    SELECT meta().xattrs._sync FROM `travel-sample` WHERE meta().id = "mydocId"

    Prior to 5.5, there is no way to query the mobile metadata with shared_bucket_access_enabled: true.

    The sync metadata is maintained internally by Sync Gateway and its structure can change at any time. It should not be used to drive business logic of applications. The direct use of the N1QL query is unsupported and must not be used in production environments.

    How do I access a blob stored in Couchbase Lite?

    See the Couchbase Lite examples here:

    How do I access an attachment from a WebApp?

    Attachments can be accessed through Sync Gateway’s REST API using the /{tkn-db}/{doc}/{attachment} endpoint.

    Migrating from Bucket Shadowing

    As of Sync Gateway 1.5, the Bucket Shadowing feature is deprecated and no longer supported. The following steps outline a recommended method for migrating from Bucket Shadowing to the latest version with interoperability between Couchbase Server SDKs and Couchbase Mobile.

    1. Follow the recommendations in the Couchbase Server documentation to upgrade all instances to 5.0.

    2. Create a new bucket on Couchbase Server (bucket 2).

    3. Install Sync Gateway 1.5 on a separate node with shared access enabled and connect it to the new bucket (bucket 2).

    4. Setup a push replication from the Sync Gateway instance used for Bucket Shadowing to the Sync Gateway 1.5 instance.

    5. Once the replication has completed, test your application is performing as expected.

    6. Update the load balancer to direct incoming traffic to the Sync Gateway 1.5 instance when you are ready to upgrade.

    7. Delete the first bucket (bucket 1).

    bucket shadowing migration