Delete v Expiry

    +

    Goal: Differentiate between a deletion or an expiration

    Implementation:

    Create a JavaScript Function that contains an OnDelete handler, which runs whenever a document is deleted or expired (or mutated). The handler merely logs whether the document was deleted or was expired from a collection. This function accesses the 'expired' field of the second optional argument to the handler.

    Preparations:

    For this example, two (2) buckets 'bulk' and 'rr100' are required where the later is intended to be 100% resident. Create the buckets with a minimum size of 100MB. For information on buckets, see Create a Bucket. Within the buckets we need two (2) keyspaces 'bulk.data.source' and 'rr100.eventing.metadata' (we loosely follow this organization).

    If you run a version of Couchbase prior to 7.0 you can just create the buckets 'source' and 'metadata' and run this example. Furthermore if your cluster was subsequently upgraded from say 6.6.2 to 7.0 your data would be moved to 'source._default._default' and 'metadata._default._default' and your Eventing Function would be seamlessly upgraded to use the new keyspaces and continue to run correctly.

    For complete details on how to set up your keyspaces refer to creating buckets and creating scopes and collections.

    The Eventing Storage keyspace, in this case 'rr100.eventing.metadata', is for the sole use of the Eventing system, do not add, modify, or delete documents from it. In addition do not drop or flush or delete the containing bucket (or delete this collection) while you have any deployed Eventing functions. In a single tenancy deployment this collection can be shared with other Eventing functions.

    You will need to run cbc (the command-line KV client) or alternatively an SDK python script or Java program to create or update a document in the 'source' bucket, with an expiration time of 600 seconds.

    Procedure:

    1. The example requires a document to be created in the 'source' collection with a key of SampleDocument2, a value of {'a_key': 'a_value'}, and most importantly that the document’s expiration (or TTL) set to 600 seconds or 10 minutes).

      There are several methods to make a test document with an expiration set. The easiest is most likely using N1QL. However you can use cbc or any Couchbase SDK (the command-line KV client is compiled from the C SDK). For example you can use a Python script or a complied Java program.

      • N1QL UPDATE

      • The cbc binary, or KV client

      • Python SDK script

      • Java SDK program

      Using the Query Workbench
      UPSERT INTO `bulk`.`data`.`source` (KEY, VALUE) VALUES ("SampleDocument2", {"a_key":"a_value"}, {"expiration":600});

      Issue the above commandin the Query Workbench of the UI.

      For information on setting document expiration times via N1QL, refer to Insert a document with expiration

      On Linux
      /opt/couchbase/bin/cbc \
          create SampleDocument2 -V '{"a_key": "a_value"}' \
          -U couchbase://localhost/source \
          --scope=data --collection=source \
          -u Administrator -P password \
          --expiry=600

      on macOS

      /Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbc \
          create SampleDocument2 -V '{"a_key": "a_value"}' -U couchbase://localhost/source \
          --scope=data --collection=source \
          -u Administrator -P password \
          --expiry=600

      on Windows

      "C:\Program Files\Couchbase\Server\bin\cbc" ^
          create SampleDocument2 -V "{'a_key': 'a_value'}" -U couchbase://localhost/source ^
          --scope=data --collection=source ^
          -u Administrator -P password ^
          --expiry=600

      Use the command-line KV client, e.g. the cbc binary, and cut-n-paste one of the above commands to create the needed sample document.

      We are passing --expiry the number of time in seconds from now at which the item should expire. However if you want an expiry over 30 days you must use the number of seconds since Unix Epoch.

      On macOS (or OS-X) if you get a 'dyld: Library not loaded' when running cbc a solution is documented in MB-37768.

      For information on the cbc tool, refer to Using the command-line KV client.

      #!/usr/bin/python3
      import sys
      import couchbase.collection
      import couchbase.subdocument as SD
      from couchbase.cluster import Cluster, ClusterOptions
      from couchbase_core.cluster import PasswordAuthenticator
      from couchbase.durability import ServerDurability, Durability
      from datetime import timedelta
      
      pa = PasswordAuthenticator('Administrator', 'password')
      cluster = Cluster('couchbase://127.0.0.1', ClusterOptions(pa))
      bucket = cluster.bucket('bulk')
      collection = bucket.scope('data').collection('source')
      
      try:
        document = dict( a_key="a_value" )
        result = collection.upsert(
          'SampleDocument2',
          document,
          expiry=timedelta(minutes=10)
        )
        print("UPSERT SUCCESS")
        print("cas result:", result.cas)
      except:
        print("exception:", sys.exc_info())

      Make an executable script as above and then it. Alternatively run the command python3 to start a Python session then cut-n-paste the above line (without the #!/usr/bin/python3 line) to create the needed sample document and then ^D (or ctrl-D) to close the Python session.

      For information on the Couchbase Python SDK, refer to Start Using the Python SDK.

      A Java 3.0 SDK program example

      // Must use the Collections API
      package com.jonstrabala;
      import java.time.Duration;
      import com.couchbase.client.java.*;
      import com.couchbase.client.java.json.JsonObject;
      import static com.couchbase.client.java.kv.UpsertOptions.upsertOptions;
      public class DocExpiryTestCC {
          public static void main(String... args) throws Exception {
          	// Note, if not on the server you need to change "localhost" to your DNS name or IP
          	Cluster cluster = Cluster.connect("localhost", "Administrator", "password");
          	Bucket bucket = cluster.bucket("bulk");
          	// Collection collection = bucket.defaultCollection();
          	Collection collection = bucket.scope("data").collection("source");
          	String docID = "SampleDocument2";
          	Duration dura = Duration.ofMinutes(10);
          	try {
          		collection.upsert(
          			docID, JsonObject.create().put("a_key", "a_value"),
          			upsertOptions().expiry(dura) );
          		System.out.println("docID: " + docID + " expires in " + dura.getSeconds());
          	} catch (Exception e) {
          		System.out.println("upsert error for docID: " + docID + " " + e);
          	}
              bucket = null;
              collection = null;
          	cluster.disconnect(Duration.ofSeconds(2000));
          }
      }

      Download the proper SDK and then compile and run one of the above Java programs

      For information on the Couchbase Java SDK, refer to Start Using the Java SDK.

    2. You now have a document in collection 'source' (keyspace bulk.data.source) with an expiration set.

    3. To verify that your new document was created, access the Couchbase Web Console > Documents page and click the Documents then select the keyspace bulk.data.source. The new document gets displayed automatically (as this page will attempt to list the first few items). You will see one (1) document in the bulk.data.source keyspace (this will disappear on the document’s expiry of 10 minutes).

      del v expiry 05 buckets
    4. [Optional Step] Click on the document’s id, SampleDocument2 to view the documents Data and also the documents Metadata information. Note that the "expiration" field in the Metadata is non-zero (set to a Unix timestamp in seconds since epoch).

    5. From the Couchbase Web Console > Eventing page, click ADD FUNCTION, to add a new Function. The ADD FUNCTION dialog appears.

    6. In the ADD FUNCTION dialog, for individual Function elements provide the below information:

      • For the Listen To Location drop-down, select bulk, data, source as the keyspace.

      • For the Eventing Storage drop-down, select rr100, eventing, metadata as the keyspace.

      • Enter delete_v_expiry as the name of the Function you are creating in the Function Name text-box.

      • Leave the "Deployment Feed Boundary" as Everything.

      • [Optional Step] Enter text Function to show how to determine if a document removal is an expiration or a deletion, in the Description text-box.

      • For the Settings option, use the default values.

      • Skip the Bindings options

      • After configuring your settings the ADD FUNCTION dialog should look like this:

        del v expiry 01 settings
    7. After providing all the required information in the ADD FUNCTION dialog, click Next: Add Code. The delete_v_expiry dialog appears.

      • The delete_v_expiry dialog initially contains a placeholder code block. You will substitute your actual delete_v_expiry code in this block.

        del v expiry 02 editor with default
      • Copy the following Function, and paste it in the placeholder code block of delete_v_expiry dialog.

        function OnDelete(meta, options) {
            if (options.expired) {
                log("doc expired:",meta.id);
            } else {
                log("doc deleted:",meta.id);
            }
        }

        After pasting, the screen appears as displayed below:

        del v expiry 03 editor with code
      • Click Save and Return.

    8. From the Eventing screen, click the delete_v_expiry function to select it, then click Deploy.

      del v expiry 03a deploy
      • Click Deploy Function.

    9. The Eventing function is deployed and starts running within a few seconds. From this point, the defined Function is executed on all existing documents and on subsequent mutations.

    10. When its expiration time is reached, an item is deleted as soon as one of the following occurs:

      • An attempt is made to access the item.

      • The expiry pager is run (default every 60 minutes).

      • Compaction is run.

    11. Therefore we first wait the full 10 minute period from the creation of SampleDocument2 and then we will try to access the document to expedite the expiry occurrence.

    12. Access the Couchbase Web Console > Buckets page.

      • You may see document count of one or zero depending in the Bucket bulk if the expiry pager has run.

        del v expiry 04 view bkt
    13. Click on the Documents in the UI you will see one (1) document in the bulk.data.source keyspace (this will disappear on the document’s expiry of 10 minutes). If the expiration on the document SampleDocument2 has occurred there will be no documents found.

      del v expiry 04b view doc
    14. Access the Couchbase Web Console > Buckets page for a second time.

      • You should see the document count is zero for the bucket source as we have attempted to access the item and it has been recognized and marked as an expired tombstone.

        del v expiry 04b view bkt
    15. Access the Couchbase Web Console > Eventing page, click the function name delete_v_expire.

      • You should see the following statistics under the Deployment Statistics:

        del v expiry 04 expiration
    16. Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).

      • You should see that the document has expired.

        2021-07-17T18:58:54.269-07:00 [INFO] "doc expired:" "SampleDocument2"
        del v expiry 05 log expired
    17. Now let’s create another document and perform a normal delete on it. Access the Couchbase Web Console > Buckets page and click the Scopes and Collections link of the bulk bucket.

      • Click Documents in the upper right banner for the data scope.

      • Select the keyspace bulk, data, source

      • You should see no user records.

      • Click Add Document in the upper right banner

      • For the ID in the Create New Document dialog specify SampleDocument

        ID [ SampleDocument3        ]
      • For the document body in the Create New Document dialog, the following text is displayed:

        {
        "click": "to edit",
        "with JSON": "there are no reserved field names"
        }
      • just keep the above text

      • Click Save.

    18. Now click the trash can icon to delete the SampleDocument3 you just created. Select Continue to confirm the deletion.

    19. Access the Couchbase Web Console > Eventing page, click the function name delete_v_expire.

      • You should see following statistics under the Deployment Statistics:

        del v expiry 06 deletion
    20. Click the "Log" link for delete_v_expiry to view the activity (the "Log" link will appear in the upper right of the Function’s controls once the function is deployed).

      • Here we see the document was deleted.

        2021-07-17T19:04:32.069-07:00 [INFO] "doc deleted:" "SampleDocument3"
        del v expiry 06 log deleted

    Cleanup:

    Go to the Eventing portion of the UI and undeploy the Function delete_v_expiry, this will remove the 1280 documents for each function from the 'rr100.eventing.metadata' colection (in the Bucket view of the UI). Remember you may only delete the 'rr100.eventing.metadata' keyspace if there are no deployed Eventing Functions.

    Now flush the 'bulk' bucket if you plan to run other examples (you may need to Edit the bucket 'bulk' and enable the flush capability).