User Profile Sample: Data Sync Fundamentals

    +

    Introduction

    Couchbase Sync Gateway is a key component of the Couchbase Mobile stack. It is an internet-facing synchronization mechanism that securely syncs data across devices as well as between devices and the cloud. Couchbase Mobile is built upon a websocket based replication protocol.

    The core functions of the Sync Gateway include

    • Data Synchronization across devices and the cloud

    • Authorization

    • Access Control

    • Data Validation

    What You Will Learn

    This tutorial will demonstrate how to -

    • Setup the Couchbase Sync Gateway to sync content between multiple Couchbase Lite enabled clients. We will will cover the basics of the Sync Gateway Configuration.

    • Configure your Sync Gateway to enforce data routing, access control and authorization. We will cover the basics of the Sync Function API.

    • Configure your Couchbase Lite clients for replication with the Sync Gateway.

    • Use "Live Queries" or Query events within your Couchbase Lite clients to be asynchronously notified of changes.

    We will be using an Android app as an example of a Couchbase Lite enabled client.

    You can learn more about Sync Gateway here in the Sync Gateway Documentation

    Prerequisites

    This tutorial assumes familiarity with building Android apps using Java and with the basics of Couchbase Lite.

    • If you are unfamiliar with the basics of Couchbase Lite, it is recommended that you walk through the following tutorials:

    To follow the tutorial it will be useful to have:

    • Android Studio

    • Android device or emulator running API level 29 or above

    • Android SDK 29+

    • Android Build Tools 29+

    • JDK 8

    • git (Optional)
      This is required if you want prefer to pull the source code from GitHub repo.

    • curl HTTP client
      We use curl in our tutorial. Download latest version from curl website, or use the HTTP client of your choice.

    System Overview

    Userprofile Sample App

    We will be working with a simple "User Profile" app which we introduced in the Standalone tutorial and extended in the Query tutorial.

    In this tutorial, we will be extending that app to support data sync. It will now also:

    • Allows users to log in and create or update his/her user profile information.
      The user profile view is automatically updated every time the profile information changes in the underlying database

    • The user profile information is synced with a remote Sync Gateway which then syncs it to other devices (subject to access control and routing configurations specified in the sync function)

    App with Sync

    Sample App Architecture

    The sample app follows the MVP pattern, separating the internal data model, from a passive view through a presenter that handles the logic of our application and acts as the conduit between the model and the view.

    MVP Architecture

    In the Android Studio project, the code is structured by feature. You can select the Android option in the left navigator to view the files by package.

    MVP Android Studio

    Each package contains 3 different files:

    • Activity: This is where all the view logic resides.

    • Presenter: This is where all the business logic resides to fetch and persist data to a web service or the embedded Couchbase Lite database.

    • Contract: An interface that the Presenter and Activity implement.

    MVP Package

    App Installation

    Clone the sync branch of the User Profile Demo project from GitHub. Type the following command in your terminal

    git clone -b sync https://github.com/couchbaselabs/userprofile-couchbase-mobile-android

    Installing Couchbase Lite

    This sample project already contains the appropriate additions for downloading, and utilizing the Android Couchbase Lite dependency module. However, in the future, to include Couchbase Lite support within an Android app include the following in app/build.gradle.

      dependencies {
        ...
    
        implementation 'com.couchbase.lite:couchbase-lite-android-ee:3.0.0'
    }
    Try it out
    • Open build.gradle using Android Studio.

    • Build and run the project.

    • Verify that you see the login screen.

      User Profile Login Screen Image

    Data Model

    If have followed the Query tutorial, you can skip this section and proceed to the Backend Installation. section. We have not made any changes to the Data model for this tutorial.

    Couchbase Lite is a JSON Document Store. A Document is a logical collection of named fields and values.The values are any valid JSON types. In addition to the standard JSON types, Couchbase Lite supports some special types like Date and Blob. While it is not required or enforced, it is a recommended practice to include a "type" property that can serve as a namespace for related.

    The "User Profile" Document

    The app deals with a single Document with a "type" property of "user". The document ID is of the form "user::<email>". An example of a document would be:

    {
        "type":"user",
        "name":"Jane Doe",
        "email":"jame.doe@earth.org",
        "address":"101 Main Street",
        "image":CBLBlob (image/jpg),
        "university":"Missouri State University"
    }

    UserProfile

    For the purpose of this tutorial the "user" Document is first stored within an Object of type Map<String, Object>.

    Map<String, Object> profile = new HashMap<>();
    profile.put("name", nameInput.getText().toString());
    profile.put("email", emailInput.getText().toString());
    profile.put("address", addressInput.getText().toString());
    profile.put("university", universityText.getText().toString());
    profile.put("type", "user");
    byte[] imageViewBytes = getImageViewBytes();
    
    if (imageViewBytes != null) {
        profile.put("imageData", new com.couchbase.lite.Blob("image/jpeg", imageViewBytes));
    }

    The "University" Document

    The app comes bundled with a collection of Documents of type "university". Each Document represents a university.

    {
        "type":"university","web_pages": [
          "http://www.missouristate.edu/"
        ],
        "name": "Missouri State University",
        "alpha_two_code": "US",
        "state-province": MO,
        "domains": [
          "missouristate.edu"
        ],
        "country": "United States"
    }

    The University Record

    When "university" Document is retrieved from the database it is stored within an Object of type Map<String, Object>.

    Map<String, Object> properties = new HashMap<>();
    properties.put("name", row.getDictionary("universities").getString("name"));
    properties.put("country", row.getDictionary("universities").getString("country"));
    properties.put("web_pages", row.getDictionary("universities").getArray("web_pages"));

    Backend Installation

    We will install Couchbase Server and Sync Gateway using Docker.

    Prerequisites

    • You must have Docker installed on your laptop. For more on Docker — see: Get Docker

    • On Windows, you may need admin privileges.

    • Ensure that you have sufficient memory and cores allocated to docker. At Least 3GB of RAM is recommended.

    Docker Network

    Create a docker network named “workshop”

    docker network ls
    
    docker network create -d bridge workshop

    Couchbase Server

    Install

    We have a custom docker image priyacouch/couchbase-server-userprofile:7.0.0-dev of Couchbase Server, which creates an empty bucket named “userprofile” and an RBAC user “admin” with “sync gateway” role.

    Alternatively, you can follow the instructions in our documentation — see: Get Started - Prepare, to install Couchbase Server and configure it with the relevant bucket.

    1. Optionally, remove any existing Docker container

      docker stop cb-server && docker rm cb-server
    2. Start Couchbase Server in a Docker container

      docker run -d --name cb-server \
      --network workshop \
      -p 8091-8094:8091-8094 -p 11210:11210 \
      priyacouch/couchbase-server-userprofile:7.0.0-dev

    Test Server Install

    The server could take a few minutes to deploy and fully initialize; so be patient.

    1. Check the Docker logs using the command:

      docker logs -f cb-server

      When the setup is completed, you should see output similar to that shown in Figure 1.

      log output
      Figure 1. Server set-up output
    2. Now check the required data is in place:

      1. Open up http://localhost:8091 in a browser

      2. Sign in as “Administrator” and “password” in login page

      3. Go to “buckets” menu and confirm “userprofile” bucket is created

        confirm bucket created
      4. Go to “security” menu and confirm “admin” user is created

        confirm admin user created

    Sync Gateway

    Now we will install, configure and run Sync Gateway.

    Configuration

    When using Sync Gateway 3.0, we can opt to provide a bootstrap configuration — see: Sync Gateway Configuration. We would then provision database, sync and other configuration using the Admin REST endpoints Alternatively, we can continue to run in legacy-mode, using the Pre-3.0 configuration.

    In this tutorial — for the purposes of backward compatibility — we will run 3.x using its legacy configuration option. That is, we will be running with the disable_persistent_config option in the configuration file set to true. You can, if you wish, run a 2.8 version of Sync Gateway instead.

    The configuration files corresponding to this sample application are shown in Table 1. They are available in the "sync" branch of the github repo hosting the app, which you cloned — look in:
    /path/to/cloned/repo/userprofile-couchbase-mobile/content/modules/userprofile-sync/examples/

    Table 1. Available configuration files

    Release

    Filename

    3.x

    2.x

    Deploy

    Configure and launch Sync Gateway in a Docker container.

    1. Switch to the the folder containing the cloned configuration files, using:

      cd /path/to/cloned/repo/userprofile-couchbase-mobile-android/content/modules/userprofile-sync/examples
    2. Make sure no Sync Gateway container exists, using:

      docker stop sync-gateway && docker rm sync-gateway
    3. Launch Sync Gateway in a Docker container

      You should see configuration files for the latest major version and the previous major version in this folder — see: Table 1. Choose an appropriate version.

      For non-Windows Systems
      • Sync Gateway 3.0

      • Sync Gateway 2.x

      Configure and run Sync Gateway 3.0 in Docker using the configuration file sync-gateway-config-userprofile-demo-3-x-legacy.json.

      Note the use of disable_persistent_config in the configuration file to force legacy configuration mode.

      docker run -p 4984-4986:4984-4986 \
      --network workshop \
      --name sync-gateway \
      -d \
      -v `pwd`/sync-gateway-config-userprofile-demo-3-x-legacy.json:\
      /etc/sync_gateway/sync_gateway.json \
      couchbase/sync-gateway:3.0.0-enterprise \
      /etc/sync_gateway/sync_gateway.json

      Configure and run Sync Gateway 2.8 in Docker

      docker run -p 4984-4986:4984-4986 \
      --network workshop \
      --name sync-gateway \
      -d \
      -v `pwd`/sync-gateway-config-userprofile-demo-2-x.json:\
      /etc/sync_gateway/sync_gateway.json \
      couchbase/sync-gateway:2.8.4-enterprise \
      /etc/sync_gateway/sync_gateway.json
      For Windows Systems
      • Sync Gateway 3.0

      • Sync Gateway 2.x

      Configure and run Sync Gateway 3.0 in legacy mode

      docker run -p 4984-4986:4984-4986 ^
      --network workshop ^
      --name sync-gateway ^
      -d -v %cd%sync-gateway-config-userprofile-demo-3-x-legacy.json:^
      /etc/sync_gateway/sync_gateway.json ^
      couchbase/sync-gateway:3.0.0-enterprise ^
      /etc/sync_gateway/sync_gateway.json

      Configuring and running Sync Gateway 2.8

      docker run -p 4984-4986:4984-4986 ^
      --network workshop ^
      --name sync-gateway ^\
      -d ^
      -v %cd%/sync-gateway-config-userprofile-demo-2-x.json:^
      etc/sync_gateway/sync_gateway.json ^
      couchbase/sync-gateway:2.8.4-enterprise ^
      /etc/sync_gateway/sync_gateway.json

    Test the Installation

    Now we can confirm that the Sync Gateway is up and running.

    1. Check the log messages

      docker logs -f sync-gateway

      You will see a series of log messages. Make sure there are no errors.

    2. Open up http://localhost:4984 in a browser.
      You should see equivalent of the following message

      {"couchdb":"Welcome","vendor":{"name":"Couchbase Sync Gateway","version":"3.0"},"version":"Couchbase Sync Gateway/3.0.0(145;e3f46be) EE"}

    Now that we have the server and the sync gateway installed, we can verify data sync between Couchbase Lite enabled apps.

    Sync Function

    The Sync Function is a Javascript function that is specified as part of the Sync Gateway Configuration. The Sync Function handles data validation, authorization, access control and data routing.

    • Open the sync-gateway-config-userprofile-demo-2-x.json* file using any text editor of your choice. This configuration file is located in the app bundle at /path/to/UserProfileDemo/modules/userprofile/examples/src.

    • Locate the sync setting and follow along with the rest of the sections below

    Authorization

    We use the requireUser() API to verify that the email property specified in the Document matches the Id of the user making the request. The Id of the user making the request is specified in the Authorization header. We will be using Basic Authentication in our application.

    function sync(doc, oldDoc) {
    
       /* Authorization */
    
      // Verify the user making the request is the same as the one in doc's email
      requireUser(doc.email);
    
    }

    Data Validation

    In this case, we are doing some basic validation of the contents of the Document

    function sync(doc, oldDoc) {
    
       /* Data Validation */
    
       if (!isDelete()) {
          // Validate the presence of email fields
          validateNotEmpty("email", doc.email); (1)
    
          // Check if document is being created / added for first time
          // We allow any user to create the document
          if (isCreate()) {
    
            // Validate that the document Id _id is prefixed by owner.
            var expectedDocId = "user" + "::" + doc.email;
    
            if (expectedDocId != doc._id) { (2)
                throw({forbidden: "user doc Id must be of form user:email"});
            }
    
          } else {
             // Validate that the email hasn't changed.
            validateReadOnly("email", doc.email, oldDoc.email); (3)
          }
    
        }
      }
    
    
      // Verify that specified property exists
      function validateNotEmpty(key, value) {
        if (!value) {
          throw({forbidden: key + " is not provided."});
        }
      }
    
      // Verify that specified property value has not changed during update
      function validateReadOnly(name, value, oldValue) {
        if (value != oldValue) {
          throw({forbidden: name + " is read-only."});
        }
      }
    1 Verify that the email property is not null. If it’s null, we throw a JS exception (see validateNotEmpty() function)
    2 If this a new document, then verify that the Id of the Document is of the required format (i.e. "user::<email>"). We throw an exception if that’s not the case.
    3 If this is a document update, then verify that the email property value has not changed. Again, we throw an exception if that’s not the case.

    You can learn more about the Sync Function in the Sync Function API

    Data Routing

    Channels are a mechanism to "tag" documents and is typically used to segregate documents based on the contents of the document. Combined with the access() and requireAccess() API, it can be used to enforce Access Control. As we shall see in a later section, clients can use channels to pull only a subset of documents.

      /* Routing */
      // Subsequent updates to document must be authorized
      var email = getEmail();
    
      // Add doc to the user's channel.
      channel("channel." + email); (1)
    
      // get email Id property
      function getEmail() {
        return (isDelete() ? oldDoc.email : doc.email);
      }
    1 The channel comes into existance the first time a document is added to it. In our case, the channel name is generated from the email property specified in the document

    Access Control

    You can enforce access control to channels using the access() API. This will ensure that only users with access to a specific channel will be able to retrieve documents in the channel.

      /* Access Control */
      // Give user read access to channel
       if(!isDelete()) {
        // Deletion of user document is essentially deletion of user
           access(email,"channel." + email)
       }

    Starting Replication

    Two-way Replication between the app and the Sync Gateway is enabled when user logs into the app.

    • Open the DatabaseManager.java file and locate the startPushAndPullReplicationForCurrentUser() function.

      public static void startPushAndPullReplicationForCurrentUser(String username, String password)
    • Next, we create an instance of the ReplicatorConfiguration instance that specifies the source and target database and you can optionally, override the default configuration settings.

      ReplicatorConfiguration config = new ReplicatorConfiguration(userprofileDatabase, new URLEndpoint(url)); (1)
      config.setType(ReplicatorType.PUSH_AND_PULL); (2)
      config.setContinuous(true); (3)
      
      config.setAuthenticator(new BasicAuthenticator(username, password.toCharArray())); (4)
      config.setChannels(Arrays.asList("channel." + username)); (5)
    1 Initialize with source as the local Couchbase Lite database and the remote target as the Sync Gateway
    2 Replication type of PUSH_AND_PULL indicates that we require two-way sync. A value of .PUSH specifies that we only pull data from the Sync Gateway. A value of .PULL specifies that we only push data.
    3 The continuous mode is specified to be true, which means that changes are synced in real-time. A value of false implies that data is only pulled from the Sync Gateway.
    4 This is where you specify the authentication credentials of the user. In the Authorization section, we discussed that the Sync Gateway can enforce authorization check using the requireUser API.
    5 The channels are used to specify the channels to pull from. Only documents belonging to the specified channels are synced. This is subject to Access Control rights enforced at the Sync Gateway. This means that if a client does not have access to documents in a channel, the documents will not be synched even if the client specifies it in the replicator configuration.
    • Initialize the Replicator with the ReplicatorConfiguration

      replicator = new Replicator(config);
    • We attach a callback listener to the Replicator to be asynchronously notified of state changes. This could be useful for instance, to inform the user of the progress of the replication. This is an optional step

      replicatorListenerToken = replicator.addChangeListener(new ReplicatorChangeListener() {
          @Override
          public void changed(ReplicatorChange change) {
      
              if (change.getReplicator().getStatus().getActivityLevel().equals(ReplicatorActivityLevel.IDLE)) {
                  Log.e("Replication Comp Log", "Scheduler Completed");
              }
              if (change.getReplicator().getStatus().getActivityLevel().equals(ReplicatorActivityLevel.STOPPED)
                      || change.getReplicator().getStatus().getActivityLevel().equals(ReplicatorActivityLevel.OFFLINE)) {
                  Log.e("Rep Scheduler  Log", "ReplicationTag Stopped");
              }
          }
      });
    • Start the replicator

      replicator.start();

    Stopping Replication

    When user logs out of the app, the replication is stopped before the database is closed.

    • Open the DatabaseManager.java file and locate the stopAllReplicationForCurrentUser() function.

      public static void stopAllReplicationForCurrentUser()
    • Stop the replicator and remove any associated change listeners

      replicator.removeChangeListener(replicatorListenerToken);
      replicator.stop();
      All open replicators must be stopped before database is closed. There will be an exception if you attempt to close the database without closing the active replicators.

    Query Events / Live Queries

    Couchbase Lite apps can set up live queries in order to be asynchronously notified of changes to the database that affect the results of the query. This would be very useful for instance, to keep a UI View up-to-date with the results of a query.

    In our app, the user profile view is kept up-to-date with a live query that fetches the user profile data that is used to populate the view. This means that, if the replicator pulls down changes to the user profile, it will be automatically reflected in the view.

    • Open the UserProfilePresenter.java file and locate the fetchProfile() function.

       public void fetchProfile()
    • Build the Query using QueryBuilder API. If you are unfamiliar with this API, please check out this tutorial.

      Query query = QueryBuilder
                      .select(SelectResult.all())
                      .from(DataSource.database(database))
                      .where(Meta.id.equalTo(Expression.string(docId))); (1)
      1 We query for documents based on document Id. In our app, there should be exactly one user profile document corresponding to this Id.
    • Attach listener callback to the query to make it live

      query.addChangeListener(new QueryChangeListener() {
      
          @Override
          public void changed(QueryChange change) { (1)
              ResultSet rows = change.getResults();
      
              Result row = null;
              Map<String, Object> profile = new HashMap<>(); (2)
      
              profile.put("email", DatabaseManager.getSharedInstance().currentUser);
      
              while ((row = rows.next()) != null) {
                  Dictionary dictionary = row.getDictionary("userprofile"); (3)
      
                  if (dictionary != null) {
                      profile.put("name", dictionary.getString("name")); (4)
                      profile.put("address", dictionary.getString("address")); (4)
                      profile.put("imageData", dictionary.getBlob("imageData")); (4)
                      profile.put("university", dictionary.getString("university")); (4)
                      profile.put("type", dictionary.getString("type")); (4)
                  }
              }
      
              mUserProfileView.showProfile(profile);
          }
      });
      1 Attach a listener callback to the query. Attaching a listerner automatically makes it live so any time there is a change in the user profile data in the underlying database, the callback would be invoked
      2 Create an instance of [UserRecord]. This will be populated with the query results.
      3 The SelectResult.all() method is used to query all the properties of a document. In this case, the document in the result is embedded in a dictionary where the key is the database name, which is "userprofile". So we retrieve the Dictionary at key "userprofile".
      4 We use appropriate type getters to retrieve values and populate the UserRecord instance

    Exercises

    Exercise 1

    In this exercise, we will observe how changes made on one app are synced across to the other app

    • The app should be running in two simulators side by side

    • Log into both the simulators with same userId and password. Use the values "demo@example.com" and "password" for user Id and password fields respectively

    • On one simulator, enter values in the user and address fields.

    • Confirm that changes show up in the app on the other simulator.

    • Similarly, make changes to the app in the other simulator and confirm that the changes are synced over to the first simulator.

    Exercise 2

    In this exercise, we will observe changes made via Sync Gateway are synced over to the apps.

    • Make sure you complete Exercise 1.
      This is to ensure that you have the appropriate user profile document (with document Id of "user::<emailId>") created through the app and synced over to the Sync Gateway.

    • Open the command terminal and issue the following command to get the user profile document via [GET Document REST API].
      We will be using curl to issue the request. If you haven’t done so, please install curl as indicated in the Prerequisites section

      curl -X GET \
        http://localhost:4985/userprofile/user::demo@example.com \
        -H 'Accept: application/json' \
        -H 'Cache-Control: no-cache' \
        -H 'Content-Type: application/json'
    • Your response should look something like the response below.
      The exact contents depends on the user profile information that you provided via your mobile app.

      {
          "_attachments": { (1)
              "blob_1": {
                  "content_type": "image/jpeg",
                  "digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
                  "length": 14989,
                  "revpos": 2,
                  "stub": true
              }
          },
          "_id": "user::demo@example.com",
          "_rev": "2-3a76cfa911e2c54d1e82b29dbffc7f4e5a9bc265", (2)
          "address": "",
          "email": "demo@example.com",
          "image": {
              "@type": "blob",
              "content_type": "image/jpeg",
              "digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
              "length": 14989
          },
          "name": "",
          "type": "user",
          "university": "Missouri State University"
      }
      1 If you had updated an image via the mobile app, you should see an * "_attachments"* property. This entry holds an array of attachments corresponding to each image blob entry added by the mobile app. This property is added by the Sync Gateway when it processes the document. You can learn more about how image Blob types are mapped to attachments [here].
      2 Record the revision Id of the document. You will need this when you update the document
    • In the command terminal, issue the following command to update the user profile document via [PUT Document REST API]

      curl -X PUT \
        'http://localhost:4985/userprofile/user::demo@example.com?rev=3-12d203d6024c8b844c5ed736c726ac63379e05dc' \
        -H 'Accept: application/json' \
        -H 'Cache-Control: no-cache' \
        -H 'Content-Type: application/json' \
        -d '{
          "address": "101 Main Street", (1)
          "email": "demo@example.com",
          "image": {
              "@type": "blob",
              "content_type": "image/jpeg",
              "digest": "sha1-S8asPSgzA+F+fp8/2DdIy4K+0U8=",
              "length": 14989
          },
          "name": "",
          "type": "user",
          "university": "Missouri State University"
      }'
      1 I updated the university field via the REST API. You can choose to update any other profile information
    • Confirm that you get a HTTP "201 Created" status code

    • As soon as you update the document via the Sync Gateway REST API, confirm that the changes show up in the mobile app on the simulator.

      App Sync

    Handling Conflicts during Data Synchronization

    Data conflicts are inevitable in an environment where you can potentially have multiple writes updating the same data concurrently. Couchbase MObile supports Automated Conflict Resolution.

    You can learn more about automated conflict resolution in this blog post.

    Learn More

    Congratulations on completing this tutorial!

    This tutorial walked you through an example of how to use a Sync Gateway to synchronize data between Couchbase Lite enabled clients. We discussed how to configure your Sync Gateway to enforce relevant access control, authorization and data routing between Couchbase Lite enabled clients.

    Check out the following links for further details