A newer version of this documentation is available.

View Latest

Document Expiry

    +

    Goal: When a document in an existing bucket is about to expire, a new document or is created in a different bucket.

    Implementation:

    Create a JavaScript Function that contains an OnUpdate handler, which runs whenever a document is created (or mutated). The handler calls a timer routine, which executes a callback function, two minutes prior to any document’s established expiration. This function retrieves a specified value from the document, and stores a document with the same key, in a specified target bucket. The original document in the source bucket is not changed (and will be deleted).

    Preparations:

    For this example, three buckets 'source', 'target', and 'metadata', are required (note the metadata bucket for Eventing can be shared with other Eventing functions). Make all three buckets with minimum size of 100MB.

    For steps to create buckets, see Create a Bucket.

    The 'metadata' bucket is for the sole use of the Eventing system, do not add, modify, or delete documents from this bucket. In addition do not drop or flush or delete the bucket while you have any deployed Eventing functions.

    You will need to run cbc (the command-line KV client) or alternatively an SDK python script or Java program to create or update a document in the 'source' bucket, with an expiration time of 600 seconds.

    Procedure:

    1. The example requires a document to be created in the 'source' bucket with a key of SampleDocument2, a value of {'a_key': 'a_value'}, and most importantly that the document’s expiration (or TTL) set to 600 seconds or 10 minutes).

      There are several methods to make a test document with an expiration set. The easiest is most likely using cbc. However you can use any Couchbase SDK (the command-line KV client is compiled from the C SDK). For example you can use a Python script or a Java program.

      • The cbc binary, or KV client

      • Python SDK script

      • Java SDK program

      On Linux
      /opt/couchbase/bin/cbc \
          create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \
          -u Administrator -P password \
          --expiry=600

      on macOS

      /Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbc \
           create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \
          -u Administrator -P password \
          --expiry=600

      on Windows

      "C:\Program Files\Couchbase\Server\bin\cbc" ^
           create SampleDocument2 -V "{'a_key': 'a_value'}" -U couchbase://localhost/source ^
          -u Administrator -P password ^
          --expiry=600

      Use the command-line KV client, e.g. the cbc binary, and cut-n-paste one of the above commands to create the needed sample document.

      We are passing --expiry the number of time in seconds from now at which the item should expire. However if you want an expiry over 30 days you must use the number of seconds since Unix Epoch.

      On macOS (or OS-X) if you get a 'dyld: Library not loaded' when running cbc a solution is documented in MB-37768.

      For information on the cbc tool, refer to Using the command-line KV client.

      # This is a Python 2.5 SDK example
      from couchbase.cluster import Cluster
      from couchbase.cluster import PasswordAuthenticator
      import time
      cluster = Cluster('couchbase://localhost:8091')
      authenticator = PasswordAuthenticator('Administrator', 'password')
      cluster.authenticate(authenticator)
      cb = cluster.open_bucket('source')
      cb.upsert('SampleDocument2', {'a_key': 'a_value'})
      cb.touch('SampleDocument2', ttl=10*60)

      Issue the command python to start a Python session then cut-n-paste the above to create the needed sample document. Type ^D (or ctrl-D) to close the Python session.

      For information on the Couchbase Python SDK, refer to Start Using the Python SDK.

      A Java 2.7.11 SDK program example

      // Original Bucket API
      import java.time.Duration;
      import com.couchbase.client.java.Bucket;
      import com.couchbase.client.java.Cluster;
      import com.couchbase.client.java.CouchbaseCluster;
      import com.couchbase.client.java.document.JsonDocument;
      import com.couchbase.client.java.document.json.JsonObject;
      public class UpsertAndSetExpiry {
          public static void main(String... args) throws Exception {
              Cluster cluster  = CouchbaseCluster.create("localhost");
              cluster.authenticate("Administrator", "password");
              Bucket bucket = cluster.openBucket("source");
              String docID = "SampleDocument2";
              Duration dura = Duration.ofMinutes(10);
              try {
                  JsonDocument doc = JsonDocument.create(
                      docID, (int)dura.getSeconds(),
                      JsonObject.create().put("a_key", "a_value") );
                  bucket.upsert(doc);
                  System.out.println("docID: " + docID + " expires in " + dura.getSeconds());
              } catch (Exception e) {
                  System.out.println("upsert error for docID: " + docID + " " + e);
              }
              bucket.close();
              cluster.disconnect();
          }
      }

      A Java 3.0 SDK program example

      // New Collections API
      package com.jonstrabala;
      import java.time.Duration;
      import com.couchbase.client.java.*;
      import com.couchbase.client.java.json.JsonObject;
      import static com.couchbase.client.java.kv.UpsertOptions.upsertOptions;
      public class UpsertAndSetExpiry_v3 {
          public static void main(String... args) throws Exception {
          	Cluster cluster = Cluster.connect("localhost", "Administrator", "password");
          	Bucket bucket = cluster.bucket("source");
          	Collection collection = bucket.defaultCollection();
          	String docID = "SampleDocument2";
          	Duration dura = Duration.ofMinutes(10);
          	try {
          		collection.upsert(
          			docID, JsonObject.create().put("a_key", "a_value"),
          			upsertOptions().expiry(dura) );
          		System.out.println("docID: " + docID + " expires in " + dura.getSeconds());
          	} catch (Exception e) {
          		System.out.println("upsert error for docID: " + docID + " " + e);
          	}
              bucket = null;
              collection = null;
          	cluster.disconnect(Duration.ofSeconds(2000));
          }
      }

      Download the proper SDK and then compile and run one of the above Java programs

      For information on the Couchbase Java SDK, refer to Start Using the Java SDK.

    2. You now have a document in bucket 'source' with an expiration set.

    3. To verify that your new document was created, access the Couchbase Web Console > Buckets page and click the Documents link of the source bucket. The new document gets displayed automatically (as this page will attempt to list the first few items).

    4. [Optional Step] Click on the document’s id, SampleDocument2 to view the documents Data and also the documents Metadata information. Note that the "expiration" field in the Metadata is non-zero (set to a Unix timestamp in seconds since epoch).

    5. From the Couchbase Web Console > Eventing page, click ADD FUNCTION, to add a new Function. The ADD FUNCTION dialog appears.

    6. In the ADD FUNCTION dialog, for individual Function elements provide the below information:

      • For the Source Bucket drop-down, select source.

      • For the Metadata Bucket drop-down, select metadata.

      • Enter add_timer_before_expiry as the name of the Function you are creating in the Function Name text-box.

      • [Optional Step] Enter text Function that adds timer before document expiry, in the Description text-box.

      • For the Settings option, use the default values.

      • For the Bindings option, add two bindings.

        • For the first binding, select "bucket alias", specify src as the "alias name" of the bucket, and select source as the associated bucket, and select "read only".

        • For the first binding, select "bucket alias", specify tgt as the "alias name" of the bucket, and select target as the associated bucket, and select "read and write".

      • After configuring your settings your screen should look like:

        docexpiry 01 settings
    7. After providing all the required information in the ADD FUNCTION dialog, click Next: Add Code. The add_timer_before_expiry dialog appears.

      • The add_timer_before_expiry dialog initially contains a placeholder code block. You will substitute your actual add_timer_before_expiry code in this block.

        docexpiry 02 editor with default
      • Copy the following Function, and paste it in the placeholder code block of add_timer_before_expiry dialog.

        function OnUpdate(doc, meta) {
            // Only process for those documents that have a non-zero TTL
            if (meta.expiration == 0 ) return;
            // Get the TTL and compute 2 minutes prior to the TTL, note JavaScript Date() takes msec.
            var twoMinsPrior = new Date((meta.expiration - 2*60) * 1000);
            // Create a context and then create a timer with our context
            var context = { docID : meta.id, expiration : meta.expiration };
            createTimer(DocTimerCallback, twoMinsPrior , meta.id, context);
            log('OnUpdate add Timer 2 min. prior to TTL to DocId:',  meta.id);
        }
        function DocTimerCallback(context) {
            log('DocTimerCallback 1 on DocId:', String(context.docID));
            // create a new document with the same ID but in the target bucket
            tgt[context.docID] = "To Be Expired in 2 min., Key's Value is:" + JSON.stringify(src[context.docID]);
            log('DocTimerCallback 2 src expiry:', new Date(context.expiration  * 1000));
            log('DocTimerCallback 3 tgt archive via Key:', String(context.docID));
        }

        After pasting, the screen appears as displayed below:

        docexpiry 03 editor with code
      • Click Save.

      • To return to the Eventing screen, click the '< back to Eventing' link (below the editor) or click Eventing tab.

    8. From the Eventing screen, click Deploy.

      • In the Confirm Deploy Function dialog, select Everything from the Feed boundary option.

      • Click Deploy Function.

    9. The Eventing function is deployed and starts running within a few seconds. From this point, the defined Function is executed on all existing documents and on subsequent mutations.

    10. Look at the Log for add_timer_before_expiry once it deploys (the "Log" link will appear once the function is deployed)

      2020-01-13T13:50:47.149-08:00 [INFO] "OnUpdate add Timer 2 min. prior to TTL to DocId:" "SampleDocument2"
    11. Now look at the Buckets in the UI the 'metadata' bucket will have 2048 documents related to the Eventing function and three (3) additional documents related to the timer, and of course the key thing is that you should see one (1) document in the 'source' bucket (poked in via the Python script).

      docexpiry 04 buckets
    12. Wait a few minutes, return to Eventing in the UI and Look at the Log again for add_timer_before_expiry at two minutes before the TTL was scheduled the timer will have fired and executed DocTimerCallback (note the logs display by the "Log" link are in reverse time order)

      2020-01-13T13:51:58.783-08:00 [INFO] "DocTimerCallback 3 tgt archive via Key:" "SampleDocument2"
      2020-01-13T13:51:58.783-08:00 [INFO] "DocTimerCallback 2 src expiry:" "2020-01-13T21:53:46.000Z"
      2020-01-13T13:51:58.781-08:00 [INFO] "DocTimerCallback 1 on DocId:" "SampleDocument2"
      2020-01-13T13:50:47.149-08:00 [INFO] "OnUpdate add Timer 2 min. prior to TTL to DocId:" "SampleDocument2"

      The final result, is a new document containing data from the original, named SourceDocument2 being written to the bucket 'target' with the same Key.

    13. Now look at the Buckets in the UI again you will see one (1) document in the 'source' bucket and one (1) document in the 'target bucket'.

      docexpiry 05 buckets
    14. Wait a few more minutes (actual just bit more than two minutes) past the 120 second window, then check the document within the bucket 'source', you will find that it is missing and will not be accessible as it has expired due to the defined TTL on the document.

      If you don’t actually try to access the document in the bucket 'source' by clicking on the *Documents" link the UI will indicate it still exists until the expiry pager removes the tombstone for the deleted or expired documents (or an attempt to access it is made).
      docexpiry 06 buckets
    15. Cleanup, go to the Eventing portion of the UI and undeploy the Function add_timer_before_expiry, this will remove the 2048 documents from the 'metadata' bucket (in the Bucket view of the UI). Remember you may only delete the 'metadata' bucket if there are no deployed Eventing functions.