This guide provides a platform-independent overview of Couchbase Lite that focuses on what it is and how it works with your data.
Couchbase Lite is a lightweight, document-oriented (NoSQL), syncable database engine suitable for embedding into mobile apps.
Couchbase Lite provides native APIs for seamless iOS (Objective-C) and Android (Java) development. In addition, it includes the Couchbase Lite Plug-in for PhoneGap, which enables you to build iOS and Android apps that you develop by using familiar web-application programming techniques and the PhoneGap mobile development framework.
Major features of Couchbase Lite:
The world has gone mobile—both businesses and consumers have leapt on the bandwagon. By using Couchbase Lite, you can provide your customers with a seamless mobile experience.
Businesses are moving to mobile solutions to provide better customer experiences and operate their businesses more efficiently.
Today’s consumers rely on mobile apps to organize their lives and keep up with family and friends. They have smartphones and tablets. They want to keep their data synched between multiple devices and multiple people. For example, all members of a family could use an app that helps them coordinate schedules and shopping lists.
By using flexible JSON documents rather than a rigid schema, your database can evolve over time without impacting the user experience. Your users can count on having an amazing app experience with a fast and unbreakable local database.
Couchbase Lite provides an ultra-lightweight, reliable, secure JSON database built for all your online and offline mobile application needs.
Couchbase Lite provides a sync solution that already works. It’s easy to set up, easy to manage, and easy to scale. Data syncing is crucial for mobile apps because it:
Lets users work with their data on multiple devices, from phones to desktop computers.
Lets groups of users collaborate on shared data.
Lets companies update data sets (whether corporate databases or restaurant directories) in one central place and have the updates delivered efficiently to clients.
Makes apps more responsive, and even lets users work offline, by taking network I/O out of the critical path. The app’s UI operates on local data, and syncing runs in the background.
However, syncing is very difficult to implement properly. It requires special metadata (like vector clocks or revision trees), has to handle network partition and data conflicts, and its algorithms have to work incrementally and be highly failure-tolerant. Some mobile developers have waded into ad hoc sync implementations and found themselves in over their heads, with delayed or canceled products. Couchbase Lite has sync compatibility with a solution that already works, Couchbase Sync Gateway.
Couchbase has been working with community users and customers on use cases like these:
Medical Records—medical data is a great fit for schemaless JSON storage. It’s also critical that it be available wherever the health care provider goes, regardless of network conditions.
Customer Loyalty and Point of Sale—we see a lot of these apps already using our sync technology, and we’ve been working with some developers closely to ensure a smooth ride.
Airline—pilots and flight attendants benefit from having easy access to data about passengers and flight plans, with the ability to dynamically refresh the data when they are on the ground.
Fleet Management—tracking vehicle telemetry and routing it to the cloud when connections are available is a great fit for Couchbase Mobile.
Social Media—chat and game companies often take a portfolio approach. By offloading the details of pushing data across mobile networks, they can focus on rolling out compelling content that uses a common backbone.
Couchbase Lite enables sync on iOS and Android devices with a flexible, embedded JSON-based database that works with Sync Gateway and Couchbase Server to handle the server side of your app synchronization connections. In production, you run Sync Gateway and use Couchbase Server for storage so you can handle a growing user base with confidence.
The mobile technology stack consists of:
Your Couchbase Lite-powered application—with Couchbase Lite embedded you have fine-grained control over data access and routing. For each document in the database, you specify a set of channels the document belongs to and for each user or device, you control which channels they can access.
Sync Gateway—a server that manages HTTP-based data access for mobile clients. It handles access control and data routing, so that a single large Couchbase Server cluster can manage data for multiple users and complex applications.
Couchbase Server—a high performance, scalable, NoSQL document database that’s been battle-tested in heavy traffic, mission-critical deployments serving millions of users.
The following diagram illustrates the mobile technology stack:
Couchbase Server should be deployed behind your firewall (as databases normally are). Sync Gateway should be deployed on a server that can be accessed by mobile devices over the Internet and can also reach Couchbase Server. Mobile devices connect to Sync Gateway, which enforces access control and update validation policies.
This section describes how Couchbase Lite structures and works with data. If you’re familiar with relational databases and SQL, you’ll notice that Couchbase Lite works differently and has its own database terminology. The following table compares the terminology:
|Couchbase Lite Term||Relational / SQL Term|
|Document type (informal)||Table|
|Document ID||Primary key|
|GET or view query||Query|
Couchbase Lite is a document database. A document has a much more flexible data format than a SQL database row, generally contains all the information about a data entity (including compound data) rather than being normalized across tables, and can have arbitrary-sized binary attachments.
A document is a JSON object, similar to a dictionary data structure, that consists of arbitrary key-value pairs. There’s no schema—every document can have its own individual set of keys, although almost all databases adopt one or more informal schemas.
Whatever its contents, though, every document has a special property called
_id. This property is the document ID, which is the document’s unique identifier in its database. A document ID is similar to a SQL primary key, except that primary keys are usually integers while document IDs are strings. When you create a document, you can either provide your own ID or let Couchbase Lite assign one. If you provide your own document IDs, you can use any string you want, such as a universally unique identifier (UUID) or a string that is meaningful to your application.
Documents have some other special properties, and their names always start with an underscore. The leading underscore denotes a reserved property—don’t use an underscore prefix for any of your own properties.
A Couchbase Lite database is a collection of documents, and serves as a namespace to look them up by their unique document IDs. The database supports the typical CRUD operations (create, read, update, and delete) .
Unlike relational databases, Couchbase Lite databases don’t contain tables. They’re not necessary—tables exist to define a schema for their rows, and Couchbase Lite documents don’t have a schema. You can freely put different types of documents, containing different properties appropriate to their types, into the same database. It is very convenient, though, to be able to easily distinguish different types of documents (especially in view functions), so there’s an informal convention of using a
type property whose value is a string identifying the document type.
Querying is probably the hardest thing about Couchbase Lite for SQL users to get used to. In SQL, you use a complex query language to specify exactly what data you want, then run the query and get back the data. In Couchbase Lite it’s a two-stage process based on a technique called MapReduce.
First you define a view that uses a map function to extract information out of every document. The map function is written in the same language as your app—most likely Objective-C or Java—so it’s very flexible. The result of applying the map function to the database is an ordered set of key-value pairs. For example, a map function might grind through an address-book database and produce a set of mappings from names to phone numbers. The view’s output is stored persistently in the database and updated incrementally as documents change. It’s very much like the type of index a SQL database creates internally to optimize queries.
The second step is querying the view to get actual data. You give a range of keys, and get back all the key-value pairs in that range, sorted by key. Optionally, you can use the view’s reduce function to aggregate values together, or group matching values into one. It’s very simple compared to the baroque possibilities of SQL
SELECT statements—the power of views comes both from the flexibility of the map and reduce functions, and from the ability to use compound (array) keys to sort the results in interesting ways.
Remember: a view is not a query, it’s an index. Views are persistent, and need to be updated (incrementally) whenever documents change, so having large numbers of them can be expensive. Instead, it’s better to have a smaller number of views that can be queried in interesting ways.
Replication is a key feature of Couchbase Lite and enables document syncing. Replication is conceptually simple—take everything that’s changed in database A and copy it over to database B.
Replication is basically unidirectional. A replication from a remote to a local database is called a pull, and a replication from a local to a remote database is called a push. Bidirectional replication, or sync, is done by configuring both a pull and a push between the same two databases.
Although any one replication involves only two databases, it’s possible to set up chains of replications between any number of databases. This creates a directed graph, with databases as nodes and replications as arcs. Many topologies are possible, from a star with one master database and a number of replicas, to a mesh network where changes made at any node eventually propagate to all the others.
When working with replication, you need to consider several factors:
Push versus pull. This is really just a matter of whether A or B is the remote database.
Continuous versus one-shot. A one-shot replication proceeds until all the current changes have been copied, then finishes. A continuous replication keeps the connection open, idling in the background and watching for more changes. As soon as the continuous replication detects any changes, it copies them. Couchbase Lite’s replicator is aware of connectivity changes. If the device goes offline, the replicator watches for the server to become reachable again and then reconnects.
Persistent versus non-persistent. Non-persistent replications, even continuous ones, are forgotten after the app quits. Persistent replications are remembered in a special
_replicator database. This is most useful for continuous replications: by making them persistent, you ensure they are always ready and watching for changes every time your app launches.
Filters. Sometimes you want only particular documents to be replicated, or you want particular documents to be ignored. To do this, you can define a filter function. The function takes a document’s contents and returns
true if it should be replicated.
At a high level, replication is simply a matter of finding all the changes that have been made to the source database since the last replication, fetching them, and applying them to the target database. The first step uses the source’s changes feed, passing in a sequence number saved during the previous replication. After all the updates are applied, the checkpoint is updated to the latest sequence number from the source.
Because replication uses document IDs to identify matching documents, it implicitly assumes that both databases share the same document-ID namespace. (And if they didn’t for some reason, they will afterwards, whether or not that was intended!) This is why UUIDs are so popular as document IDs—they pretty much guarantee that new documents created on different servers are distinct. Other naming schemes such as serial numbers or time stamps might cause unintentional ID collisions.
One important aspect of replication is the way it deals with conflicts. A database prevents local conflicts in a document by using MVCC—an attempt to add a conflicting change to a document is rejected. However, there’s no way to enforce this across multiple independent databases on different servers and devices. (Well, there are ways, but they involve distributed locking, which scales very poorly and doesn’t support offline clients at all.) The result is that the same document (as identified by its ID) can be updated with different data on different instances of Couchbase Lite. The conflict probably won’t be detected until one database replicates to the other. What happens then? It’s too late to prevent the conflict. Couchbase Lite itself doesn’t know how to reconcile it by merging the two documents together because that requires knowledge of the application schema. What it does instead is simply keep both revisions, and leave it up to the application to review them and merge them together later.
It’s actually not obvious that both revisions are present. When the app retrieves a document by its ID, it would be confusing if it got two copies. Instead, Couchbase Lite somewhat arbitrarily chooses one revision as the “winner” (it’s the one whose revision ID is lexicographically higher) and treats that one as the document’s default value. Couchbase Lite has API calls to detect the conflict state and to fetch the document’s revision tree, which really is a tree now that it’s branched. The application can then fetch the “losing” revision by its revision ID, compare the two, and then update the document with the appropriate merged contents, thereby resolving the conflict.
One significant difference from other databases is document versioning. Couchbase Lite uses a technique called Multiversion Concurrency Control (MVCC) to manage conflicts between multiple writers. This is the same technique used by version-control systems like Git or Subversion, and by WebDAV. Document versioning is similar to the check-and-set mechanism (CAS) of Couchbase Server, except that in Couchbase Lite versioning is required rather than optional and the token is a UUID rather than an integer.
Every document has a special field called
_rev that contains the revision ID. The revision ID is assigned automatically each time the document is saved. Every time a document is updated, it gets a different and unique revision ID.
When you save an update to an existing document, you must include its current revision ID. If the revision ID you provide isn’t the current one, the update is rejected. When this happens, it means some other client snuck in and updated the document before you. You need to fetch the new version, reconcile any changes, incorporate the newer revision ID, and try again.
Keep in mind that Couchbase Lite is not a version control system and you must not use the versioning feature in your application (for example, you can’t use it to store the revision history of pages in a wiki). The old revisions are just cached—they are periodically thrown away when the database is compacted, and they’re never replicated. They’re not there to use in your data model, they’re there only to help with concurrency and resolving conflicts during replication.
Couchbase Lite provides a mechanism called the changes feed to track changes to a database. Couchbase Lite assigns a sequence identifier, similar to a serial number, to each document update. The database’s changes feed is a query that returns metadata about all document updates that have occurred since a given sequence identifier. The client can then store the latest sequence identifier reported in the feed, and the next time it fetches the feed it passes in that sequence identifier so it can get only the new changes. Couchbase Lite also provides a mechanism to leave the feed running and be notified of all new changes in real time.
The changes feed powers sync between devices or from the Sync Gateway to a device. It can also be used by asynchronous background processes to be notified of updates by Sync Gateway. For instance, to send an email when a user creates a new document, you could have a script listen to the changes feed from Sync Gateway and send emails when it sees a relevant document.
This section provides answers to questions frequently asked about Couchbase Lite. The questions are grouped into the following categories:
Couchbase Lite is an ultra-lightweight and flexible mobile database that easily syncs with the cloud, built for all your mobile application needs.
Couchbase Lite was originally known as TouchDB. In January 2013 we changed the name to Couchbase Lite. The original 1.0 release of TouchDB has kept the old naming. Later releases are known as Couchbase Lite.
The reference Objective-C implementation runs on iOS 6.0 or later and Mac OS X 10.7 or later. There is also a Java version for Android devices. You can also build HTML5 apps with the Couchbase Lite PhoneGap plug-in.
The released 1.0-beta version of Couchbase Lite is ready to use in apps. As with any open source project, the latest GitHub commits do represent bleeding-edge functionality and should be used with caution.
iOS demo apps:
Android demo apps:
PhoneGap demo apps:
Couchbase Lite itself is released under the Apache License, Version 2.0.
Some libraries that Couchbase Lite uses (FMDB, MYUtilities, and CocoaHTTPServer) have MIT or BSD licenses. We might seek relicensing or rewrite those parts of the code to get everything under a single Apache license.
Yes! You can see a list on the Couchbase Lite In The Wild page.
Yes! The Mobile Couchbase Google Group is the best place to ask questions. You can access it on the web or subscribe to it as a mailing list.
As of October 2013, the iOS version of Couchbase Lite compiles to about 550 KB of optimized ARM7 code.
A: For a cold launch, it takes about 100 ms to initialize the library and open a small database. If the app has been launched and quit recently, leaving stuff in cache, it’s about 60 ms. This is on an iPad 2—older devices are a bit slower.
A: Couchbase Lite is fast enough for the kinds of data sets mobile apps use. It’s effectively instantaneous for small data sets. It won’t handle big data as well as Couchbase Server, but it keeps up pretty well with tens or hundreds of thousands of documents. Arbitrarily large binary attachments are kept as files in the file system.
There aren’t any hard limits in Couchbase Lite itself. The most likely practical limit is the available disk and flash storage on the device, and of course app responsiveness as query times increase. Couchbase Lite does not store attachments inside the database file. For information about limitations in SQLite, see Implementation Limits for SQLite.
In the ways that matter, yes. The REST API is compatible, although you talk to the engine in-process rather than over a socket. Some of the more server-centric features of CouchDB, like user accounts, aren’t supported.
Yes, its replication protocol is entirely compatible. That’s a very important goal. Apps using Couchbase Lite can sync with servers running Apache CouchDB and with Couchbase Server via Sync Gateway.
Sync Gateway acts as a server-side bridge between Couchbase Server and Couchbase Lite. After your data is synced to Couchbase Server, you can use MapReduce to build indexes across the full data set.
Yes. Revision trees are implemented and preserved across replication.
There’s an HTTP server extension called CouchbaseLiteListener. It’s mostly there to enable Couchbase Lite-to-Couchbase Lite (peer-to-peer) replication, make testing easier, and support PhoneGap-style HTML5 development.
CouchApps running in PhoneGap is definitely an interesting mobile platform. We have a [Couchbase Lite Plug-in for PhoneGap] (https://github.com/couchbaselabs/Couchbase-Lite-PhoneGap-Plugin) that enables you to use Couchbase Lite with PhoneGap. Most CouchApps should be able to run with only minor modifications.
Largely because SQLite is already available as a shared library on every platform we’re interested in. This keeps our code size down and simplifies the build process. It also comes with useful extensions like full-text indexing and R-trees (for geo-queries).
Additionally, both Berkeley and Kyoto have GPL-like licenses that are less friendly to commercial developers (especially iOS developers) and incompatible with the Apache license of Couchbase Lite itself.
Because Couchbase Lite is a lot smaller, starts up a lot more quickly, and is easily embedded into an app. Those are important factors for mobile app developers (and some desktop app developers too). If you’re working on server-side software, those factors probably don’t matter to you or at least don’t outweigh the drawbacks.
World-class, highly-flexible data sync capabilities that go way beyond what you can get from iCloud. Another factor is that the API is simpler and easier to use than Core Data.
As with the previous comparison to Core Data, the big reason is sync. If your users want to work with their data on multiple devices or platforms (including the Web) or have it transparently backed up, the replication capabilities in Couchbase Lite make it very easy compared to the pain of implementing sync yourself or trying to duct-tape your custom SQLite database to the iCloud APIs.
For an iOS example, see Deleting Replications.
It depends on how you configure the replication. You can set up a continuous replication that is always active when the device is online, or you can trigger a one-shot replication whenever you want.
Yes, by using regular HTTP gzip transfer encoding. A proxy like nginx or Apache can apply this encoding transparently.
It sends the whole document. Typically documents aren’t that big. If a document contains attachments, only the ones that have changed are transferred.
If the client has an active replication, changes from the server are pushed to it within a second or two. On iOS 7, your server can send push notifications that invisibly wake up the app and let it replicate the new changes.