Introduction

Obsolete documentation

For the latest Couchbase Mobile documentation, visit the Couchbase Mobile developer portal.

Couchbase Sync Gateway is an add-on that enables Couchbase Server 2.0 and later to act as a replication endpoint for Couchbase Lite. Sync Gateway runs an HTTP listener process that provides a passive replication endpoint and uses a Couchbase Server bucket as persistent storage for all database documents.

Architecture

Sync Gateway provides an HTTP front-end for Couchbase Server that syncs with Couchbase Lite. The following figure shows how Sync Gateway interacts with mobile apps and Couchbase Server.

Getting Started With Sync Gateway

You can run Sync Gateway on the following operating systems:

  • Mac OS X 10.6 or later with a 64-bit CPU

  • Red Hat Linux

  • Ubuntu Linux

Installing Sync Gateway

You can download Sync Gateway for your platform from http://www.couchbase.com/communities/couchbase-sync-gateway.

The download contains an executable file called sync_gateway that you run as a command-line tool. For convenience, you can move it to a directory that is included in your $PATH environment variable.

Connecting Sync Gateway to Couchbase Server

You can connect Sync Gateway to Couchbase Server 2.0 or later.

To connect Sync Gateway to Couchbase Server:

  1. Open the Couchbase Server Admin Console and log on using your administrator credentials.
  2. In the toolbar, click Data Buckets.
  3. On the Data Buckets page, click Create New Data Bucket and create a bucket named sync_gateway in the default pool.

You can use any name you want for your bucket, but sync_gateway is the default name that Sync Gateway uses if you do not specify a bucket name when you start Sync Gateway. If you use a different name for your bucket, you need to specify the -bucket option when you start Sync Gateway.

Starting Sync Gateway

You start Sync Gateway by running sync_gateway with the -url option. The argument for the -url option is the HTTP URL of the Couchbase server to which you want Sync Gateway to connect. If you do not include any additional command-line options, the default values are used.

The following command starts Sync Gateway on port 4984, connects to the default bucket named sync_gateway in the Couchbase Serving running on localhost, and starts the admin server on port 4985.

$ ./sync_gateway -url http://localhost:8091

If you used a different name for the Couchbase Server bucket or want to listen on a different port, you need to include those parameters as command-line options. For information about the available command-line options, see Administering Sync Gateway.

Stopping Sync Gateway

You can stop Sync Gateway by typing Control-C. There is no specific shutdown procedure and it is safe to stop it at any time.

Administering Sync Gateway

This section describes how to administer Sync Gateway.

Command Line Tool

You can launch the sync_gateway with command-line options. However, in the long run, it’s better to use JSON configuration files, which are the only way to serve multiple databases. You can also combine command-line options with configuration files.

The format of the sync_gateway command is:

sync_gateway [Options] [ConfigurationFile...] 

Options

Option Default Description
-adminInterface 127.0.0.1:4985 Port the Admin REST API listens on
-bucket sync_gateway Name of the Couchbase bucket to use
-dbname bucket name Name of the database to serve via the Sync REST API
-help None Lists the available options and exits.
-interface 4984 Port the Sync REST API listens on
-log None Comma-separated list of logging keywords to enable. The HTTP keyword is always enabled, which means HTTP requests and error responses are always logged.
-personaOrigin None Base URL for Persona authentication. It should be the same URL that the client uses to reach the server.
-pool default Couchbase Server pool name in which to find buckets
-pretty false Pretty-print JSON responses. This is useful for debugging, but reduces performance.
-url walrus: URL of the database server. An HTTP URL implies Couchbase Server, a walrus: URL implies the built-in Walrus database.
-verbose false Logs more information about requests.

The command-line tool uses the regular Go flag parser, so you can prefix options with one or two hyphen (-) characters and give option values either as a following argument or in the same argument after an equal sign (=).

The following command does not include any parameters and just uses the default values. It connects to the bucket named sync_gateway in the pool named default of the built-in Walrus database. It is served from port 4984, with the admin interface on port 4985.

$ sync_gateway

The following command creates an ephemeral, in-memory Walrus database, served as db and specifies pretty-printed JSON responses.

$ sync_gateway -url=walrus: -bucket=db -pretty

The following command uses a Walrus database that is persisted to a file named /tmp/walrus/db.walrus.

$ sync_gateway -url=walrus:///tmp/walrus -bucket=db -pretty

Configuration Files

Instead of entering the settings on the command-line, you can store them in a JSON file and then just provide the path to that file as a command-line argument. As a bonus, the file lets you run multiple databases.

If you want to run multiple databases you can either add more entries to the databases property in the configuration file, or you can define each database in its own configuration file and list each of the configuration files on the command line.

Configuration files have one syntactic feature that’s not standard JSON: any text between backticks (`) is treated as a string, even if it spans multiple lines or contains double-quotes. This makes it easy to embed JavaScript code , such as the sync function.

The following sample configuration file starts a server with the default settings:

{
   "interface":":4984",
   "adminInterface":":4985",
   "log":["REST"],
   "databases":{
      "sync_gateway":{
         "server":"http://localhost:8091",
         "bucket":"sync_gateway",
         "sync":`function(doc) {channel(doc.channels);}`
      }
   }
}

You can see an example of a more complex configuration file in the CouchChat-iOS sample app.

The following command starts Sync Gateway with the parameters specified in a configuration file named config.json:

$ sync_gateway config.json

The following command starts Sync Gateway with the parameters specified in a configuration file named config.json and adds additional logging by including the -log option on the command line:

$ sync_gateway -log=HTTP+,CRUD config.json

Administering the REST APIs

Sync Gateway provides the following REST APIs:

  • The Sync REST API is used for client replication. The default port for the Sync REST API is 4984.

  • The Admin REST API is used to administer user accounts and roles. It can also be used to look at the contents of databases in superuser mode. The default port for the Admin REST API is 4985. By default, the Admin REST API is reachable only from localhost for safety reasons.

Managing API Access

The APIs are accessed on different TCP ports, which makes it easy to expose the Sync REST API on port 4984 to clients while keeping the Admin REST API on port 4985 secure behind your firewall.

If you want to change the ports, you can do that in the configuration file.

  • To change the Sync REST API port, set the interface property in the configuration file.

  • To change the Admin REST API port, set the adminInterface property in the configuration file.

The value of the property is a string consisting of a colon followed by a port number (for example, :4985). You can also prepend a host name or numeric IP address before the colon to bind only to the network interface with that address.

As a useful special case, the IP address 127.0.0.1 binds to the loopback interface, making the port unreachable from any other host. This is the default setting for the admin interface.

Managing Guest Access

Sync Gateway does not allow anonymous or guest access by default. A new server is accessible through the Sync REST API only after you enable guest access or create some user accounts. You can do this either by editing the configuration file before starting the server or by using the Admin REST API. For more information, see Anonymous Access.

Authorizing Users

You can authorize users and control their access to your database by creating user accounts and assigning roles to users.

Accounts

You manage accounts by using the Admin REST API.This interface is privileged and for administrator use only. To allow clients to manage accounts, you need to have some other server-side mechanism that calls through to this API.

The URL for a user account is /databasename/_user/name, where databasename is the configured name of the database and name is the user name. The content of the resource is a JSON document with the following properties:

  • admin_channels: Lists the channels that the user is granted access to by the administrator. The value is an array of channel name strings.

  • admin_roles: The roles that the user is explicitly granted access to through the Admin REST API. It contains an array of role name strings.

  • all_channels: Like the admin_channels property, but also includes channels the user is given access to by other documents via a sync function. This is a derived property and changes to it are ignored.

  • disabled: This property is usually not included. if the value is set to true, access for the account is disabled.

  • email: The user’s email address. This property is optional, but Persona login needs it.

  • name: The user name (the same name used in the URL path). The valid characters for a user name are alphanumeric ASCII characters and the underscore character. The name property is required in a POST request. You don’t need to include it in a PUT request because the user name is specified in the URL.

  • password: In a PUT or POST request, you can set the user’s password with this property. It is not returned by a GET request.

  • roles: Like the admin_roles property, but also includes roles the user is given access to by other documents via a sync function. This is a derived property and changes to it are ignored. It contains an array of role name strings.

You can create a new user by sending a PUT request to its URL or by sending a POST request to /$DB/_user/.

Anonymous Access

A special user account named GUEST applies to unauthenticated requests. Any request to the Sync REST API that does not have an Authorization header or a session cookie is treated as coming from the GUEST account. This account and all anonymous access is disabled by default.

To enable the GUEST account, set its disabled property to false. You might also want to give it access to some channels. If you don’t assign some channels to the GUEST account, anonymous requests won’t be able to access any documents. The following sample command enables the GUEST account and allows it access to a channel named public:

$ curl -X PUT localhost:4985/$DB/_user/GUEST --data \
   '{"disabled":false, "admin_channels":["public"]}'

Roles

Roles are named collections of channels. A user account can be assigned to zero or more roles. A user inherits the channel access of all roles it belongs to. This is very much like Unix groups, except that roles do not form a hierarchy.

You access roles through the Admin REST API much like users are accessed, through URLs of the form /dbname/_role/name. Role resources have a subset of the properties that users do: name, admin_channels, all_channels.

Roles have a separate namespace from users, so it’s legal to have a user and a role with the same name.

Authenticating Users

You can authenticate users by using the methods described in the following sections.

HTTP

Sync Gateway allows clients to authenticate by using either HTTP Basic Auth or cookie-based sessions. The session URL is /dbname/_session.

Facebook

Sync Gateway supports Facebook Login, which allows users to log in by using their Facebook account. To enable it, add a top-level facebook property to your server configuration file. For example:

"facebook" : {
   "register" : true
}

Clients log in by sending a POST request to /dbname/_facebook, with a JSON body that contains the following objects:

  • access_token—Access token returned by Facebook
  • email—Email address of the user
  • remote_url—Sync Gateway replication endpoint URL

Just as with a _session login, the response sets a session cookie.

Persona

Sync Gateway supports Mozilla Persona, a sign-in system for the web that allows clients to authenticate by using an email address. You can enable Persona either by modifying your server configuration file or by starting Sync Gateway with an additional command-line option.

To enable Persona by modifying the configuration file, add a top-level persona property to the config.json file. The value of the persona property is an object with an origin property that contains your server’s canonical root URL as seen by clients. For example:

"persona" : {
   "origin" : "http://example.com/",
   "register" : true
}

To enable Persona when you start Sync Gateway, add the -personaOrigin option to the command line and specify the server’s canonical root URL. For example:

$ sync_gateway -personaOrigin http://example.com

The origin URL must be specified explicitly because the Persona protocol requires both client and server to agree on the server’s identity, and there’s no reliable way to derive the URL on the server, especially if it’s behind a proxy.

After that’s set up, you need to set the email property of the user accounts, so that the server can look up the account based on the email address given in the Persona credentials.

Clients log in by sending a POST request to /dbname/_persona. The request body is a JSON document that contains an assertion property whose value is the signed assertion received from the identity provider. Just as with a _session login, the response sets a session cookie.

Facebook and Persona Account Registration

If the register property of the Facebook or Persona configuration is true, then clients can implicitly register new user accounts by authenticating through Facebook or Persona. If Sync Gateway verifies the client’s assertion, but no existing user account has that email address, it creates a new user account for that email and returns a session cookie.

The user name for the new account is the same as the authenticated email address and has a random password. There is no way to retrieve the password, so in the future the client must continue to log in by using Facebook or Persona, unless the app server replaces the password with one known to the client.

Custom (Indirect) Authentication

An app server can create a session for a user by sending a POST request to /dbname/_session. This works only on the admin port.

The request body is a JSON document with the following properties:

  • name: User name

  • ttl: Number of seconds until the session expires. This is an optional parameter. If ttl is not provided, the default value of 24 hours is used.

The response body is a JSON document that contains the following properties:

  • session_id: Session string

  • cookie_name: Name of the cookie the client should send

  • expires : Date and time that the session expires

This allows the app server to optionally do its own authentication using the following control flow:

  1. Client sends credentials to your app server.

  2. App server authenticates the credentials however it wants (LDAP, OAuth, and so on).

  3. App server sends a POST request with the user name to the Sync Gateway Admin REST API server /dbname/_session endpoint.

  4. If the request fails with a 404 status, there is no Sync Gateway user account with that name. The app server can then create one (also using the Admin REST API) and repeat the _session request.

  5. The app server adds a Set-Cookie: HTTP header to its response to the client, using the session cookie name and value received from the gateway.

Subsequent client requests to the gateway will now include the session in a cookie, which the gateway will recognize. For the cookie to be recognized, your site must be configured so that your app’s API and the gateway appear on the same public host name and port.

Session Expiration

By default, a session created on Sync Gateway lasts 24 hours. If you create sessions by sending a POST request to /db/_session, you can set a custom value that overrides the system default. However, if you are using Persona for authentication, the only way to customize the session length is by modifying the kDefaultSessionTTL constant in the rest_session.go file.

Developing

This section contains information and concepts you need to know when developing apps that interact with Sync Gateway.

Channels

Sync Gateway uses channels to make it easy to share a database between a large number of users and control access to the database. Channels are the intermediaries between documents and users. Every document in the database belongs to a set of channels, and every user is allowed to access a set of channels. You use channels to:

  • Partition the data set.
  • Authorize users to access documents.
  • Constrain the amount of data synced to mobile clients.

A replication from Sync Gateway specifies a set of channels to replicate. Documents that do not belong to any of the specified channels are ignored (even if the user has access to them).

You do not need to register or preassign channels. Channels come into existence as documents are assigned to them. Channels with no documents assigned to them are empty.

Valid channel names consist of text letters [A–Z, a–z], digits [0–9], and a few special characters [-+=/_.@] . The empty string is not allowed. The special channel name * denotes all channels. Channel names are compared literally—the comparison is case and diacritical sensitive.

Mapping documents to channels

You assign documents to channels either by adding a channels property to the document or by using a sync function. No matter which option you choose, the channel assignment is implicit—the content of the document determines what channels it belongs to.

Using a Channels Property

Adding a channels property to each document is the easiest way to map documents to channels. The channels property is an array of strings that contains the names of the channels to which the document belongs. If you do not include a channels property in a document, the document does not appear in any channels.

Using a Sync Function

Creating a sync function is a more flexible way to map documents to channels. A sync function is a JavaScript function that takes a document body as input and, based on the document content, decides what channels to assign the document to. The sync function cannot reference any external state and must return the same results every time it’s called on the same input.

You specify the sync function in the configuration file for the database. Each sync function applies to one database.

To add the current document to a channel, the sync function calls the special function channel, which takes one or more channel names (or arrays of channel names) as arguments. For convenience, channel ignores null or undefined argument values.

Defining a sync function overrides the default channel mapping mechanism (the document’s channels property is ignored). The default mechanism is equivalent to the following simple sync function:

function (doc) {
    channel (doc.channels);
}

Replicating Channels to Couchbase Lite

if a client doesn’t specify any channels to replicate, it gets all the channels to which its user account has access. Due to this behavior, most apps do not have to specify a channels filter—instead they can just do the default sync configuration on the client (that is, specify the Sync Gateway database URL with no filter) to replicate the channels of interest.

To replicate channels to Couchbase Lite, you configure the replication to use a filter named sync_gateway/bychannel with a filter parameter named channels. The value of the channels parameter is a comma-separated list of channels to fetch. The replication from Sync Gateway now pulls only documents tagged with those channels.

A document can be removed from a channel without being deleted. For example, this can happen when a new revision is not added to one or more channels that the previous revision was in. Subscribers (downstream databases pulling from this database) should know about this change, but it’s not exactly the same as a deletion.

Sync Gateway’s _changes feed includes one more revision of a document after it stops matching a channel. It adds a removed property to the entry where this happens. (No client yet recognizes this property, though.) The value of the removed property is an array of strings where each string names a channel in which this revision no longer appears. Also, the body of the document appears to be empty to the client.

The effect on the client is that after a replication it sees the next revision of the document (the one that causes it to no longer match the channel). It won’t get any further revisions until the next one that makes the document match again.

This algorithm ensures that any views running in the client do not include an obsolete revision. The app code should use views to filter the results rather than just assuming that all documents in its local database are relevant.

If a user’s access to a channel is revoked or a client stops syncing with a channel, documents that have already been synced are not removed from the user’s device.

Authorizing User Access

The all_channels property of a user account determines which channels the user can access. Its value is derived from the union of:

  • The user’s admin_channels property, which is settable via the admin REST API.
  • The channels that user has been given access to by access() calls from sync functions invoked for current revisions of documents (see Programmatic Authorization).
  • The all_channels properties of all roles the user belongs to, which are themselves computed according to the above two rules.

The only documents a user can access are those whose current revisions are assigned to one or more channels the user has access to:

  • A GET request to a document not assigned to one or more of the user’s available channels fails with a 403 error.
  • The _all_docs property is filtered to return only documents that are visible to the user.
  • The _changes property ignores requests (via the channels parameter) for channels not visible to the user.

Write protection—access control of document PUT or DELETE requests—is done by document validation. This is handled in the sync function rather than a separate validation function.

After a user is granted access to a new channel, the changes feed incorporates all existing documents in that channel, even those from earlier sequences than the client’s since parameter. That way the next client pull retrieves all documents to which the user now has access.

Programmatic Authorization

Documents can grant users access to channels. This is done by writing a sync function that recognizes such documents and calls a special access() function to grant access.

The access() function takes the following parameters: a user name or array of user names and a channel name or array of channel names. For convenience, null values are ignored (treated as empty arrays).

A typical example is a document that represents a shared resource (like a chat room or photo gallery). The document has a members property that lists the users who can access the resource. If the documents belonging to the resource are all tagged with a specific channel, then the following sync function can be used to detect the membership property and assign access to the users listed in it:

function(doc) {
  if (doc.type == "chatroom") {
    access (doc.members, doc.channel_id)
  }
}

In the example, a chat room is represented by a document with a type property set to chatroom. The channel_id property names the associated channel (with which the actual chat messages are tagged) and the members property lists the users who have access.

The access() function can also operate on roles. If a user name string begins with role: then the remainder of the string is interpreted as a role name. There’s no ambiguity here, because “:” is an illegal character in a user or role name.

Because anonymous requests are authenticated as the user “GUEST”, you can make a channel and its documents public by calling access with a username of GUEST.

Authorizing Document Updates

Sync functions can also authorize document updates. A sync function can reject the document by throwing an exception:

throw ({forbidden: "error message"})

A 403 Forbidden status and the given error string is returned to the client.

To validate a document you often need to know which user is changing it, and sometimes you need to compare the old and new revisions. To get access to the old revision, declare the sync function like this:

function(doc, oldDoc) { ... }

oldDoc is the old revision of the document (or empty if this is a new document).

You can validate user privileges by using the helper functions: requireUser, requireRole, or requireAccess. Here’s some examples of how you can use the helper functions:

// throw an error if username is not "snej"
requireUser("snej")
 
// throw if username is not in the list
requireUser(["snej", "jchris", "tleyden"]) 

// throw an error unless the user has the "admin" role
requireRole("admin") 

// throw an error unless the user has one of those roles
requireRole(["admin", "old-timer"]) 

// throw an error unless the user has access to read the "events" channel
requireAccess("events") 

// throw an error unless the can read one of these channels
requireAccess(["events", "messages"]) 

Here’s a simple sync function that validates whether the user is modifying a document in the old document’s owner list:

function (doc, oldDoc) {
  if (oldDoc) {
    requireUser(oldDoc.owner); // may throw({forbidden: "wrong user"})
  }
}

Using Walrus

Walrus is a simple, limited, in-memory database that you can use in place of Couchbase Server for unit testing during development.

Use the following command to start a Sync Gateway that connects to a single Walrus database called sync_gateway and listens on the default ports:

$ sync_gateway -url walrus:

To use a different database name, use the -bucket option. For example:

$ sync_gateway -url walrus: -bucket mydb

By default, Walrus does not persist data to disk. However, you can make your database persistent by specifying an existing directory to which Sync Gateway can periodically save its state. It saves the data to a file named /<directory>/sync_gateway.walrus. For example, the following command instructs Sync Gateway to save the data in a file named /data/sync_gateway.walrus:

$ mkdir /data
$ sync_gateway -url walrus:/data

You can use a relative path when specifying the directory for persistent data storage:

$ mkdir data
$ sync_gateway -url walrus:data

You can also specify the directory for persistent data storage in a configuration file. The config.json file would look similar to the following JSON fragment:

{
   "databases":{
      "couchchat":{
         "server":"walrus:data"         
   ...
}

Sync Gateway APIs

To interact with Couchbase Server Sync Gateway, you use the following APIs:

For information about controlling access to the REST APIs, see Administering the REST APIs.

Sync Function API

The sync function is the core API you’ll be interacting with on the Sync Gateway. For simple applications it might be the only server-side code you need to write. For more complex applications it is still a primary touchpoint for managing data routing and access control.

For more information about using sync functions, read about channels and the description of the CouchChat data model.

Default function

If you don’t supply a sync function, Sync Gateway uses this as a default:

function (doc) {
  channel(doc.channels);
}

Sync Function Arguments

The arguments enable the sync function to be used for validation as well as data routing. Your implementation can omit the oldDoc parameter if you do not need it (JavaScript ignores extra parameters passed to a function).

function (doc, oldDoc) {
  // your code here
}

The sync function arguments are:

  • doc—The document that is being saved. This matches the JSON that was saved by the mobile client and replicated to Sync Gateway. No metadata or other fields are added, although the _id and _rev fields are available.

  • oldDoc—If the document has been saved before, the revision that is being replaced is available in this argument. In the case of a document with a conflicting revision, the provisional winning revision is passed in oldDoc. If the document is being deleted, there is a _deleted property whose value is true.

Sync Function Calls

From within the sync function you create changes in the Sync Gateway configuration via callback functions. Each call manages a small amount of configuration state. It is also tied back to the document which initiated the call, so that when the document is modified, any configuration made by an old version of the document is replaced with configuration derived from the newer version. Via these APIs, documents are mapped to channels. They can also grant access to channels, either to users or roles. Finally, you can reject an update completely by throwing an error. The error message will be returned to the synchronizing client, which will log it or potentially display it to the user.

Validation via throw()

The sync function can prevent a document from persisting or syncing to any other users by calling throw() with an error object. This also prevents the document from changing any other gateway configuration. Here is an example sync function that disallows all writes to the database it is in.

function (doc) {
  throw ({forbidden : "read only!"})
}

The key of the error object may be either forbidden (corresponding to an HTTP 403 error code) or unauthorized (corresponding to HTTP 401 error). The forbidden error should be used if the user is already authenticated and the account they are syncing with is not permitted to modify or create the document. The unauthorized error should be used if the account is not authenticated. Some user agents will trigger a login workflow when presented with a 401 error.

A quick rule of thumb: most of the time you should use the throw({forbidden : "your message here"}) statement because most applications require users to be authenticated before any reads or writes can occur.

Map a document to a channel

The channel call routes the document to the named channel. It accepts either a channel name string, or an array of strings, if the document should be added to multiple channels in a single call. The channel function can be called zero or more times from the sync function, for any document. The default function (listed at the top of this document) routes documents to the channels listed on them. Here is an example that routes all “published” documents to the “public” channel.

function (doc, oldDoc) {
  if (doc.published) {
    channel ("public");
  } 
}

As a convenience, it is legal to call channel with a null or undefined argument; it simply does nothing. This allows you to do something like channel(doc.channels) without having to first check whether doc.channels exists.

Grant a user access to a channel

The access call grants access to channel to a given user or list of users. It can be called multiple times from a sync function.

The effects of the access call last as long as this revision is current. If a new revision is saved, the access calls made by the sync function will replace the original access. If the document is deleted, the access is revoked. The effects of all access calls by all active documents are effectively unioned together, so if any document grants a user access to a channel, that user has access to the channel. Note that revoking access to a channel will not delete the documents which have already been synced to a user’s device.

The access call takes two arguments, the user (or users) and the channel (or channels). These are all valid ways to call it:

access ("jchris", "mtv")
access ("jchris", ["mtv", "mtv2", "vh1"])
access (["snej", "jchris", "role:admin"], "vh1")
access (["snej", "jchris"], ["mtv", "mtv2", "vh1"])

As a convenience, either argument may be null or undefined, in which case nothing happens.

Here is an example function that grants access to a channel for all the users listed on a document:

function (doc, oldDoc) {

  access (doc.members, doc.channel_name);
  
  // we should also put this document on the channel it manages
  channel (doc.channel_name)

}

Grant access to a channel to all users with a given role

If a user name in an access call begins with the prefix role:, the rest of the name is interpreted as a role, not a user. The call then grants access to the channel(s) for all users with that role.

Grant a user a role

The role call grants a user a role, indirectly giving them access to all channels granted to that role. It can also affect the user’s ability to revise documents, if the access function requires role membership to validate certain types of changes.

Its use is similar to access:

role(user_or_users, role_or_roles);

The value of either parameter can be a string, an array of strings, or null. If the value is null, the call is a no-op.

For consistency with the access call, role names must always be prefixed with role:. An exception is thrown if a role name doesn’t match this.

Some examples:

role ("jchris", "role:admin")
role ("jchris", ["role:portlandians", "role:portlandians-owners"])
role (["snej", "jchris", "traun"], "role:mobile")

NOTE: Roles, like users, have to be explicitly created by an administrator. So unlike channels, which come into existence simply by being named, you can’t create new roles with a role() call. Nonexistent roles won’t cause an error, but have no effect on the user’s access privileges. (It is possible to create a role after the fact; as soon as it’s created, any pre-existing references to it take effect.)

Sync REST API

You use the Sync REST API to synchronize a local database with a remote database. The sync takes place over HTTP and uses JSON documents in the message bodies. For more information about the synchronization protocol, see Replication Algorithm. You can also see the URL mappings in the Sync Gateway source code.

To access the Sync REST API, you need to have a user account.

HTTP Requests

You can use the following requests on the remote database. Replace db with the dame of your database.

Push or Pull Requests:

  • Read the last checkpoint
    GET /db/_local/checkpointid

  • Save a new checkpoint
    PUT /db/_local/checkpointid

Push Requests:

  • Create remote database
    PUT /db

  • Find revisions that are not known to the remote database
    POST /db /_revs_diff

  • Upload revisions
    POST /db/_bulk_docs

  • Upload a single document with attachments
    PUT /db/docid?new_edits=false

Pull Requests:

  • Find changes since the last pull (feed can be normal or longpoll)
    GET /db/_changes?style=all_docs&feed=feed&since=since&limit=limit&heartbeat=heartbeat

  • Download a single document with attachments
    GET /db/docid?rev=revid&revs=true&attachments=true&atts_since=lastrev

  • Download first-generation revisions in bulk
    POST /db/_all_docs?include_docs=true

Admin REST API

The Admin REST API is a superset of the Sync REST API with the following major extensions:

  • The capability to view, create, and modify users and roles.
  • The capability to perform management tasks such as creating and deleting databases.
  • The capability to perform administrative tasks without authentication. The Admin REST API operates in superuser mode—you can view and modify anything without needing a password or session cookie.

By default, the Admin REST API runs on port 4985 (unless you change the adminInterface configuration parameter). Do not expose this port—It belongs behind your firewall. Anyone who can reach this port has free access to and control over your databases and user accounts.

Admin REST API Endpoints

PUT /$DB/ – Configures a new database. The body of the request contains the database configuration as a JSON object ()the same as an entry in the databases property of a configuration file. Note that this doesn’t create a Couchbase Server bucket—you need to do that before configuring the database.

DELETE /$DB/ – Removes a database. It doesn’t delete the Couchbase Server bucket or any of its data, though, so you could bring the database back later with a PUT.

/$DB/_user/$name – represents a user account. It supports GET, PUT, and DELETE, and you can also POST to /$DB/_user/. The body is a JSON object; for details see the [[Authentication]] page. The special user name GUEST applies to unauthenticated requests.

/$DB/_role/$name – represents a role. This API is similar to users.

/$DB/_session – POST to this endpoint to create a logon session. The request body is a JSON object containing the username in the name object and the duration of the session (in seconds) in the ttl object. The response is a JSON object with properties session_id (the session cookie string), expires (the time the session expires) and cookie_name (the name of the HTTP cookie to set).

/_compact – Compacts a database by removing obsolete document bodies. Needs to be run occasionally.

/_profile – POST to this endpoint to enable Go CPU profiling, which can be useful for diagnosing performance problems. To start profiling, send a JSON body with a file property whose value is a path to write the profile to. To stop profiling, send a request without a file property.

Tips

A quick way to tell whether you’re talking to the Admin REST API is by sending a GET / request and checking whether the resulting object contains an "ADMIN": true property.

HTTP requests logged to the console show the user name of the requester after the URL. If the request is made on the admin port, this is “(ADMIN)” instead.

Deployment Considerations

This section contains information about Sync Gateway deployments.

Limitations

Sync Gateway has the following limitations:

  • It cannot operate on pre-existing Couchbase buckets with app data in them because Sync Gateway has its own document schema and needs to create and manage documents itself. You can migrate existing data by creating a new bucket for the gateway and then using the Sync REST API to move your documents into it via PUT requests. You can’t make changes to the Couchbase bucket directly. You have to go through the Sync Gateway API.

  • Explicit garbage collection is required to free up space, via a REST call to /$DB/_compact. Garbage collection is not scheduled automatically, so you have to call it yourself.

  • Document IDs longer than 180 characters overflow the Couchbase Server maximum key length and cause an HTTP error.

Scaling Sync Gateway

Sync Gateway can be scaled up by running it as a cluster. This means running an identically configured instance of Sync Gateway on each of several machines, and load-balancing them by directing each incoming HTTP request to a random node. Sync Gateway nodes are “shared-nothing,” so they don’t need to coordinate any state or even know about each other. Everything they know is contained in the central Couchbase Server bucket.

All Sync Gateway nodes talk to the same Couchbase Server bucket. This can, of course, be hosted by a cluster of Couchbase Server nodes. Sync Gateway uses the standard Couchbase “smart-client” APIs and works with database clusters of any size.

Performance

Keep in mind the following notes on performance:

  • Sync Gateway nodes don’t keep any local state, so they don’t require any disk.

  • Sync Gateway nodes do not cache much in RAM. Every request is handled independently. The Go programming language does use garbage collection, so the memory usage might be somewhat higher than for C code. However, memory usage shouldn’t be excessive, provided the number of simultaneous requests per node is kept limited.

  • Go is good at multiprocessing. It uses lightweight threads and asynchronous I/O. Adding more CPU cores to a Sync Gateway node can speed it up.

  • As is typical with databases, writes are going to put a greater load on the system than reads. In particular, replication and channels imply that there’s a lot of fan-out, where making a change triggers sending notifications to many other clients, who then perform reads to get the new data.

  • We don’t currently have any guidelines for how many gateway or database nodes you might need for particular workloads. We’ll know more once we do more testing and tuning and get experience with real use cases.

Managing TCP Connections

Very large-scale deployments might run into challenges managing large numbers of simultaneous open TCP connections. The replication protocol uses a “hanging-GET” technique to enable the server to push change notifications. This means that an active client running a continuous pull replication always has an open TCP connection on the server. This is similar to other applications that use server-push, also known as “Comet” techniques, as well as protocols like XMPP and IMAP.

These sockets remain idle most of the time (unless documents are being modified at a very high rate), so the actual data traffic is low—the issue is just managing that many sockets. This is commonly known as the “C10k Problem” and it’s been pretty well analyzed in the last few years. Because Go uses asynchronous I/O, it’s capable of listening on large numbers of sockets provided that you make sure the OS is tuned accordingly and you’ve got enough network interfaces to provide a sufficiently large namespace of TCP port numbers per node.

Troubleshooting

The developers are hungry for feedback about Sync Gateway. If you run into any roadblocks, please let us know by filing an issue on one of our projects or via the mailing list. What seems insignificant to you may be hitting everyone but the core developers, so until you let us know, we can’t fix it.

Tools

In general, cURL, a command-line HTTP client, is your friend. You might also want to try HTTPie, a human-friendly command-line HTTP client. By using these tools, you can inspect databases and documents via the Sync REST API, and look at user and role access privileges via the Admin REST API.

An additional useful tool is the admin-port URL /databasename/_dump/channels, which returns an HTML table that lists all active channels and the documents assigned to them. Similarly, “/databasename/_dump/access” shows which documents are granting access to which users and channels.

Getting Help

If you’re having trouble, feel free to ask for help on the mailing list. If you’re pretty sure you’ve found a bug, please file a bug report.

Contributing

The Sync Gateway code is available on GitHub: https://github.com/couchbase/sync_gateway.

Building From Source

To build Sync Gateway from source, you must have Go 1.2 or later installed on your computer.

On Mac or Unix systems, you can build Sync Gateway from source as follows:

  1. Open a terminal window and change to the directory that you want to store Sync Gateway in.

  2. Clone the Sync Gateway GitHub repository:

     $ git clone https://github.com/couchbase/sync_gateway.git
    
  3. Change to the sync_gateway directory:

     $ cd sync_gateway
    
  4. Set up the submodules:

     $ git submodule init
     $ git submodule update
    
  5. Build Sync Gateway:

     $ ./build.sh
    

Sync Gateway is a standalone, native executable located in the ./bin directory. You can run the executable from the build location or move it anywhere you want.

To update your build later, pull the latest updates from GitHub, update the submodules, and run ./build.sh again.

Release Notes

1.0 Beta 3 (March 2014)

This is the third Beta release of Couchbase Sync Gateway 1.0.

Couchbase Sync Gateway is a ready-to-go, easy-to-scale sync layer that extends Couchbase Server to facilitate communication between Couchbase Server and your Couchbase Lite-backed apps. This version features syncing and basic admin features.

The beta release is available to all community-edition customers. You can find the Couchbase Sync Gateway documentation on the web at http://docs.couchbase.com/sync-gateway.

Features

We’ve made some major changes since Beta 2 to make Sync Gateway more performant, scalable, and co-exist better with Couchbase Server. Features introduced in Beta 3 include the following:

  • Bucket shadowing We have designed a co-existence path with Couchbase Server web clients using a workflow dubbed “bucket shadowing”. A Couchbase Server managed bucket and a Sync Gateway compatible bucket “shadow” each other. The Couchbase Server bucket can continue to be managed using regular Couchbase Server APIs. Also, if you’re already using Couchbase Server and want to make your existing data available to Couchbase Lite mobile clients, this is the recommended approach. More information can be found here.

  • New Configuration Properties Properties have been added to increase flexibility for compression, maxinum number of open file descriptors allowed and support for bucket shadowing. They are as follows:

    • compressResponses: set this to false to disable GZip compression of HTTP responses, if your gateway is behind a proxy server that applies its own compression.
    • maxFileDescriptors: The maximum number of open file descriptors/sockets. The gateway calls setrlimit at startup time to request 5000 file descriptors, but if you need more (which is likely for heavily loaded servers) you can set this property to request a higher number.
    • shadow: Configures bucket shadowing.
  • Admin API Enhancements We’ve added a number of Admin enhancements that include:

    • Inspection of document channel assignments and access grants. More detail available here.
    • You can see the database’s current sync function by doing a GET /db/_config.
    • GET /_expvar will return a (large) JSON response containing a lot of internal statistics about the gateway. We use this internally for performance testing but it could be useful to you too. See the expvars page for details.
  • Performance Enhancements We’ve added support to handle GZip-compressed HTTP requests and responses, WebSocket protocol for continuous _changes feed, and in-memory caching of recently requests document bodies as well as change history.

Fixes in Beta 3

Some fixes to highlight in this release:

  • Replication support

    • Fixed a bug that could cause the Sync Gateway to send a pull replicator the same changes over and over again, consuming CPU (on both sides) and network bandwidth.

    Issues: 262

  • Changes feed

    • Fixed a rare crash in the TruncateEncodedChangeLog function, probably triggered by lots of simultaneous writes to a channel.

    Issues: 243

  • Correctly assign document channels when importing existing docs from a Couchbase bucket or after changing the sync function.

Known Issues

  • GET /db no longer includes a doc_count property. It’s quite expensive to count the number of documents in a Couchbase bucket. This URL gets accessed at the beginning of every replication, so the overhead of including doc_count was significant, even though it appears to be unused.

1.0 Beta 2 (December 2013)

This is the second Beta release of Couchbase Sync Gateway 1.0.

Couchbase Sync Gateway is a ready-to-go, easy-to-scale sync layer that extends Couchbase Server to facilitate communication between Couchbase Server and your Couchbase Lite-backed apps. This version features syncing and basic admin features.

The beta release is available to all community-edition customers. You can find the Couchbase Sync Gateway documentation on the web at http://docs.couchbase.com/sync-gateway.

Features

The primary focus of the second Beta release for Sync Gateway has been performance enhancement, horizontal scaling, and increased stability.

Fixes in Beta 2

Overall performance fixes to improve product usability.

Known Issues

  • Authentication

    • If your Persona login fails with a 405, make sure you have set the personaOrigin URL on the command line or in the config file.

    Issues: 71

    • We are actively working on a known Mac issue for Facebook and Persona support stemming from a build bug. For Mac developers looking to use these third-party services, developers can use [Couchbase Cloud] (http://www.couchbasecloud.com/index/) or build Sync Gateway using these instructions while we fix the Mac installer.

    Issues: 71, 220

  • Web Client Support

    • Web apps will have trouble making XHR requests to Sync Gateway due to browser security restrictions unless the HTML is hosted at the same public host/port.

    Issues: 115

  • Attachment Support

    • We are actively working on an option to store attachments in third-party storage.

    Issues: 197

  • Higher ulimit

    • The ulimit can be tuned at the developer’s discretion.

    Issues: 218

  • Installation

    • Installing from Beta 1 to Beta 2 on Ubuntu allows for older instance of Sync Gateway to run. A restart of Sync Gateway will rectify this problem.

    Issues: 232

  • Rebalance support

    • If a Couchbase Server node is failed over and the cluster is not rebalanced to correct the situation, Sync Gateway will get errors on half the requests.

    Issues: 198

1.0 Beta (13 September 2013)

This is the Beta release of Couchbase Sync Gateway 1.0.

Couchbase Sync Gateway is a ready-to-go, easy-to-scale sync layer that extends Couchbase Server to facilitate communication between Couchbase Server and your Couchbase Lite-backed apps. This version features syncing and basic admin features.

The beta release is available to all community-edition customers. You can find the Couchbase Sync Gateway documentation on the web at http://docs.couchbase.com/sync-gateway.

Features

  • Dynamic sync capabilities via Sync Function APIs—Apps can begin to automatically sync data from the cloud without any manual setup by using Couchbase sync functions.

  • Easy Administration—Manage Sync Gateway via the Admin REST API when you need to.

  • Seamless scale-out—Easily scale your Sync Gateway tier as your application needs grow.

Fixes

None.

Known Issues

We are currently working on performance tuning and are aware of issues when Sync Gateway is scaled. If you run into a performance issue, please let us know.