Delete v Expiry

    +

    Goal: Differentiate between a deletion or an expiration.

    Implementation:

    Create a JavaScript Function that contains an OnDelete handler, which runs whenever a document is deleted or expired (or mutated). The handler merely logs whether the document was deleted or was expired via a bucket or document TTL. This function accesses the 'expired' field of the second argument to the handler.

    Preparations:

    For this example, two buckets 'source' and 'metadata', are required (note the metadata bucket for Eventing can be shared with other Eventing functions). Make both buckets with minimum size of 100MB.

    For steps to create buckets, see Create a Bucket.

    The 'metadata' bucket is for the sole use of the Eventing system, do not add, modify, or delete documents from this bucket. In addition do not drop or flush or delete the bucket while you have any deployed Eventing functions.

    You will need to run cbc (the command-line KV client) or alternatively an SDK python script or Java program to create or update a document in the 'source' bucket, with an expiration time of 600 seconds.

    Procedure:

    1. The example requires a document to be created in the 'source' bucket with a key of SampleDocument2, a value of {'a_key': 'a_value'}, and most importantly that the document’s expiration (or TTL) set to 600 seconds or 10 minutes).

      There are several methods to make a test document with an expiration set. The easiest is most likely using cbc. However you can use any Couchbase SDK (the command-line KV client is compiled from the C SDK). For example you can use a Python script or a Java program.

      • The cbc binary, or KV client

      • Python SDK script

      • Java SDK program

      On Linux
      /opt/couchbase/bin/cbc \
          create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \
          -u Administrator -P password \
          --expiry=600

      on macOS

      /Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbc \
           create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \
          -u Administrator -P password \
          --expiry=600

      on Windows

      "C:\Program Files\Couchbase\Server\bin\cbc" ^
           create SampleDocument2 -V "{'a_key': 'a_value'}" -U couchbase://localhost/source ^
          -u Administrator -P password ^
          --expiry=600

      Use a command-line KV client, e.g. the cbc binary, and copy and paste one of the above commands to create the needed sample document.

      We are passing --expiry the number of time in seconds from now at which the item should expire. However if you want an expiry over 30 days you must use the number of seconds since Unix Epoch.

      On macOS (or OS-X) if you get a 'dyld: Library not loaded' when running cbc a solution is documented in MB-37768.

      For information on the cbc tool, refer to Using the command-line KV client.

      # This is a Python 2.5 SDK example
      from couchbase.cluster import Cluster
      from couchbase.cluster import PasswordAuthenticator
      import time
      cluster = Cluster('couchbase://localhost:8091')
      authenticator = PasswordAuthenticator('Administrator', 'password')
      cluster.authenticate(authenticator)
      cb = cluster.open_bucket('source')
      cb.upsert('SampleDocument2', {'a_key': 'a_value'})
      cb.touch('SampleDocument2', ttl=10*60)

      Issue the command python to start a Python session then copy and paste the above to create the needed sample document. Type ^D (or Ctrl+D) to close the Python session.

      For information on the Couchbase Python SDK, refer to Start Using the Python SDK.

      A Java 2.7.11 SDK program example

      // Original Bucket API
      import java.time.Duration;
      import com.couchbase.client.java.Bucket;
      import com.couchbase.client.java.Cluster;
      import com.couchbase.client.java.CouchbaseCluster;
      import com.couchbase.client.java.document.JsonDocument;
      import com.couchbase.client.java.document.json.JsonObject;
      public class UpsertAndSetExpiry {
          public static void main(String... args) throws Exception {
              Cluster cluster  = CouchbaseCluster.create("localhost");
              cluster.authenticate("Administrator", "password");
              Bucket bucket = cluster.openBucket("source");
              String docID = "SampleDocument2";
              Duration dura = Duration.ofMinutes(10);
              try {
                  JsonDocument doc = JsonDocument.create(
                      docID, (int)dura.getSeconds(),
                      JsonObject.create().put("a_key", "a_value") );
                  bucket.upsert(doc);
                  System.out.println("docID: " + docID + " expires in " + dura.getSeconds());
              } catch (Exception e) {
                  System.out.println("upsert error for docID: " + docID + " " + e);
              }
              bucket.close();
              cluster.disconnect();
          }
      }

      A Java 3.0 SDK program example

      // New Collections API
      package com.jonstrabala;
      import java.time.Duration;
      import com.couchbase.client.java.*;
      import com.couchbase.client.java.json.JsonObject;
      import static com.couchbase.client.java.kv.UpsertOptions.upsertOptions;
      public class UpsertAndSetExpiry_v3 {
          public static void main(String... args) throws Exception {
          	Cluster cluster = Cluster.connect("localhost", "Administrator", "password");
          	Bucket bucket = cluster.bucket("source");
          	Collection collection = bucket.defaultCollection();
          	String docID = "SampleDocument2";
          	Duration dura = Duration.ofMinutes(10);
          	try {
          		collection.upsert(
          			docID, JsonObject.create().put("a_key", "a_value"),
          			upsertOptions().expiry(dura) );
          		System.out.println("docID: " + docID + " expires in " + dura.getSeconds());
          	} catch (Exception e) {
          		System.out.println("upsert error for docID: " + docID + " " + e);
          	}
              bucket = null;
              collection = null;
          	cluster.disconnect(Duration.ofSeconds(2000));
          }
      }

      Download the proper SDK and then compile and run one of the above Java programs.

      For information on the Couchbase Java SDK, refer to Start Using the Java SDK.

    2. You now have a document in bucket 'source' with an expiration set.

    3. To verify that your new document was created, access the Couchbase Web Console > Buckets page and click the Documents link of the source bucket. The new document gets displayed automatically (as this page will attempt to list the first few items).

    4. [Optional Step] Click on the document’s id, SampleDocument2, to view the document’s data and also the document’s metadata information. Note that the "expiration" field in the metadata is non-zero (set to a Unix timestamp in seconds since epoch).

    5. From the Couchbase Web Console > Eventing page, click ADD FUNCTION, to add a new Function. The ADD FUNCTION dialog appears.

    6. In the ADD FUNCTION dialog, for individual Function elements provide the below information:

      • For the Source Bucket drop-down, select source.

      • For the Metadata Bucket drop-down, select metadata.

      • Enter delete_v_expiry as the name of the Function you are creating in the Function Name text-box.

      • [Optional Step] Enter text Function to show how to determine if a document removal is an expiration or a deletion, in the Description text-box.

      • For the Settings option, use the default values.

      • Skip the Bindings options

      • After configuring your settings your screen should look like:

        del v expiry 01 settings
    7. After providing all the required information in the ADD FUNCTION dialog, click Next: Add Code. The delete_v_expiry dialog appears.

      • The delete_v_expiry dialog initially contains a placeholder code block. You will substitute your actual delete_v_expiry code in this block.

        del v expiry 02 editor with default
      • Copy the following Function, and paste it in the placeholder code block of delete_v_expiry dialog.

        function OnDelete(meta, options) {
            if (options.expired) {
                log("doc expired:",meta.id);
            } else {
                log("doc deleted:",meta.id);
            }
        }

        After pasting, the screen appears as displayed below:

        del v expiry 03 editor with code
      • Click Save.

      • To return to the Eventing screen, click the '< back to Eventing' link (below the editor) or click Eventing tab.

    8. From the Eventing screen, click Deploy.

      • In the Confirm Deploy Function dialog, select Everything from the Feed boundary option.

      • Click Deploy Function.

    9. The Eventing function is deployed and starts running within a few seconds. From this point, the defined Function is executed on all existing documents and on subsequent mutations.

    10. When its expiration time is reached, an item is deleted as soon as one of the following occurs:

      • An attempt is made to access the item.

      • The expiry pager is run (default every 60 minutes).

      • Compaction is run.

    11. Therefore we first wait the 10 minute period from the creation of SampleDocument2 and then we will try to access the document to expedite the expiry occurrence.

    12. Access the Couchbase Web Console > Buckets page.

      • You may see document count or one or zero depending if the expiry pager has run.

        del v expiry 04 view bkt
    13. Click on the Documents link for the bucket source. If the expiration on the document SampleDocument2 has occurred there will be no documents found.

    14. Access the Couchbase Web Console > Buckets page for a second time.

      • You should see the document count is zero for the bucket source as we have attempted to access the item and it has been recognized and marked as an expired tombstone.

        del v expiry 04b view bkt
    15. Access the Couchbase Web Console > Eventing page, click the function name delete_v_expire.

      • You should see the following statistics under the Deployment Statistics:

        del v expiry 04 expiration
    16. Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).

      • You should see that the document has expired.

        2020-08-02T17:28:02.635-07:00 [INFO] "doc expired:" "SampleDocument2"
        del v expiry 05 log expired
    17. Now let’s create another document and perform a normal delete on it. Access the Couchbase Web Console > Buckets page and click the Documents link of the source bucket.

      • You should see no user records.

      • Click Add Document in the upper right banner.

      • In the Add Document dialog, specify the name SampleDocument3 as the New Document ID.

      • Click Save.

      • In the Edit Document dialog, the following text is displayed:

        {
        "click": "to edit",
        "with JSON": "there are no reserved field names"
        }
      • Click Save.

    18. Now click the trash can icon to delete the SampleDocument3 you just created. Select Continue to confirm the deletion.

    19. Access the Couchbase Web Console > Eventing page, click the function name delete_v_expire.

      • You should see following statistics under the Deployment Statistics:

        del v expiry 06 deletion
    20. Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).

      • Here we see the document was deleted.

        2020-08-02T17:29:25.223-07:00 [INFO] "doc deleted:" "SampleDocument3"
        del v expiry 06 log deleted
    21. To clean up, go to the Eventing portion of the UI and undeploy the Function delete_v_expiry. This will remove the 1024 documents from the 'metadata' bucket (in the Bucket view of the UI). Remember you may only delete the 'metadata' bucket if there are no deployed Eventing functions.