Language Constructs

Enterprise Edition

The language constructs are fundamental units of a language. This topic discusses the JavaScript constructs that have been removed and new constructs that have been added in order to support the requirements of Couchbase Functions.

Using JavaScript, you can write your custom Functions. Couchbase Functions inherit support for most ECMAScript constructs by using Google v8 as the execution container. However, to support the ability to shard and scale Function-execution automatically, some capabilities have been removed. Additionally, to optimize language-utilization of the server environment, some new constructs have been added.

While every effort is made to ensure the accuracy of the content of the Eventing documentation herein, it should be noted that the controlling technical document is the Couchbase 7.0 Eventing Specification available on GitHub.

Removed Language Features

The following JavaScript features have been removed and cannot be used in Eventing Functions:

Global State
Asynchrony
Browser and other Extensions
Library Imports

Global State

Eventing Functions do not allow global variables. All state must be saved and retrieved from persistence providers. In Couchbase Server, the Data Service is used as a persistence provider. Therefore, all global states are contained in the Data Service bucket(s) made available to the Eventing Functions through bindings. This restriction is mandatory for the Eventing Function logic to remain agnostic of the rebalance operation.

var count = 0;                         // Not allowed - global variable.
function OnUpdate(doc, meta) {
  count++;
}

Note the use of 'Constant alias' bindings in the Function’s settings can be used to provide global constants accessible within a Function’s JavaScript. For example you might have a Constant alias of debug with a value of true (or false) to control verbose logging this would behave just like adding a statement const debug = true; at the beginning of your JavaScript code.

Asynchrony

Asynchrony, particularly asynchronous callbacks, to be useful needs to retain access to their parent scope. As such asynchrony forms a node specific, long-running state that prevents the capture of the entire state in the persistence providers. Therefore, Eventing Functions are restricted to executing as short-running, straight-line code, without sleep and wakeups.

function OnUpdate(doc, meta) {
  setTimeout(function(){}, 300);     // Not allowed - asynchronous flow.
}

Limited asynchrony is added back through time observers (or Timers). Time observers are designed specifically not to make the state node specific.

Browser and other Extensions

Eventing Functions execute as server-side code on Couchbase Server similar to the JavaScript code that is used in browsers.

Because Eventing Functions do not execute in the context of a browser, the extensions that browsers add to the core language, such as window methods, DOM events etc. are not available. The Couchbase Server prevents these browser extensions from executing in an Eventing Function. However a limited subset is added back (such as function timers in lieu of setTimeout, and curl calls in lieu of XHR).

For example some code that runs in the browser is excluded from use in Eventing Functions. The ‘window’ term in the code window.XMLHttpRequest(), is not a server-side construct but is in the context of a browser and as such is not available to your Eventing Functions.

function OnUpdate(doc, meta) {
  var rpc = window.XMLHttpRequest();  // Not allowed - browser extension.
}

Library Imports

Eventing service restricts importing libraries into eventing functions.

Added Language Features

The following constructs have been added:

Basic Keyspace Accessors
Advanced Keyspace Accessors
Logging
SQL++ Statements

Basic Keyspace Accessors

Couchbase buckets, when bound to a Function, appear as a global JavaScript map. The map operations such as get, set and delete are exposed to the Data Service providers get, set, and delete operations respectively.

Note, if the bucket binding has a wild card of * for the scope and/or the collection you can not use a basic keyspace accessors but must use the advanced bucket accessors to access the Data Service.

function OnUpdate(doc, meta) {
  // Assuming 'dest' is a bucket alias or binding to a keyspace
  var val = dest[meta.id];         // this is a bucket GET operation.
  dest[meta.id] = {"status":3};    // this is a bucket SET operation.
  delete dest[meta.id];            // this is a bucket DEL operation.
}

The GET operation (operator [] applied on a bucket binding and used as a value expression)

This fetches the corresponding object from the KV bucket the variable is bound to, and returns the parsed JSON value as a JavaScript object. Fetching a non-existent object from a bucket will return JavaScript undefined value. This operation throws an exception if the underlying bucket GET operation fails with an unexpected error.
The SET operation (operator [] appearing on left of = assignment statement)

This sets the provided JavaScript value into the KV bucket the variable is bound to, replacing any existing value with the specified key. This operation throws an exception if the underlying bucket SET operation fails with an unexpected error.
The DELETE operation (operator [] appearing after JavaScript delete keyword)

This deletes the provided key from the KV bucket the variable is bound to. If the object does not exist, this call is treated as a no-op. This operation throws an exception if the underlying bucket DELETE operation fails with an unexpected error.

Advanced Keyspace Accessors

The following advanced bucket accessors have been added to expose a richer set of options and operators. Unlike the basic bucket accessors these operations have non-trivial argument sets and return values. For complete details refer to Advanced Keyspace Accessors.

Note, if the bucket binding has a wild card of * for the scope and/or the collection you must also provide meta.keyspace in addition to meta.id in the meta object.

The Advanced GET operation: result = couchbase.get(binding, meta)

This operation allows reading a document from the bucket with an ability to specify the CAS value to be matched during the read. For more information see advanced GET operation.

The Advanced INSERT operation: result = couchbase.insert(binding, meta, doc)

This operation allows creating a fresh document in the bucket. This operation will fail if the document with the specified key already exists. It allows specifying an expiration time (or TTL) to be set on the document. For more information see advanced INSERT operation.

The Advanced UPSERT operation: result = couchbase.upsert(binding, meta, doc)

This operation allows updating an existing document in the bucket, or if absent, creating a fresh document with the specified key. The operation does not allow specifying CAS (it will silently ignore it). It also allows specifying an expiration time (or TTL) to be set on the document. For more information see advanced UPSERT operation.

The Advanced REPLACE operation: result = couchbase.replace(binding, meta, doc)

This operation replaces an existing document in the bucket This operation will fail if the document with the specified key does not exist. This operation allows specifying a CAS value that must be matched as a pre-condition before proceeding with the operation. It also allows specifying an expiration time (or TTL) to be set on the document. For more information see advanced REPLACE operation.

The Advanced DELETE operation: result = couchbase.delete(binding, meta)

This operation allows deleting a document in the bucket specified by key. Optionally, a CAS value may be specified which will be matched as a pre-condition to proceed with the operation. For more information see advanced DELETE operation.

The Advanced INCREMENT operation: result = couchbase.incrment(binding, meta)

This operation atomically increments the field "count" in the specified document. For more information see advanced INCREMENT operation.

The document must have the below structure:
```
{"count": 23} // 23 is the current counter value
```
The increment operation returns the post-increment value.

If the specified counter document does not exist, one is created with count value as 0 and the structure noted above. And so, the first returned value will be 1.

Due to limitations in KV engine API, this operation cannot currently manipulate full document counters.

The Advanced DECREMENT operation: result = couchbase.decrement(binding, meta)

This operation atomically decrements the field "count" in the specified document. For more information see advanced DECREMENT operation.

The document must have the below structure:
```
{"count": 23} // 23 is the current counter value
```
The decrement operation returns the post-decrement value.

If the specified counter document does not exist, one is created with count value as 0 and the structure noted above. And so, the first returned value will be -1.

Due to limitations in KV engine API, this operation cannot currently manipulate full document counters.

Logging

An additional function, log() has been introduced to the language, which allows Eventing Functions to log user defined messages. These log() statements will go the specific Eventing Function’s log file also known as the application log. The messages go files located in the Eventing data directory and do not contain any system log messages. The function takes a string to write to the file. If non-string types are passed, a best effort string representation will be logged, but the format of these may change over time. This function does not throw exceptions. For more information see application logs.

function OnUpdate(doc, meta) {
  log("Now processing: " + meta.id);
}

The Eventing Service also creates a system log file named eventing.log common across all Eventing Functions to capture management and lifecycle information, however the end-user cannot write to this file. For more information see system log.

SQL++ Statements

Top level SQL++ keywords, such as SELECT, UPDATE, INSERT and DELETE, are available as inline keywords in Eventing Functions. Operations that return values such as SELECT are accessible through a returned iterable handle. SQL++ Query results, via a SELECT, are streamed in batches to the iterable handle as the iteration progresses through the result set.

SQL++ DML statements cannot manipulate documents in the same bucket as the Eventing Function is listening for mutations on to avoid recursion. Workaround: use the exposed data service KV map in your Eventing function.

JavaScript variables can be referred by SQL++ statements using $<variable> syntax. Such parameters will be substituted with the corresponding JavaScript variable’s runtime value using SQL++ named parameters substitution facility.

When deploying the below Function with a feed boundary of "Everything" the same SQL++ statement will execute 7,303 times. If the feed boundary is configured to "From now" and you then mutate just one (1) document in the keyspace beer-sample._default._default only one (1) query will be executed. Also keep in mind that adding an optimal index can speed up the query performance by 24X.

function OnUpdate(doc, meta) {
    var strong = 70;
    var results =
        SELECT *                               /* SQL++ queries are embedded directly.    */
        FROM `beer-sample`._default._default   /* Token escaping is standard SQL++ style. */
        WHERE abv > $strong;                   // Local variable reference using $ syntax.
    for (var beer of results) {                // Stream results using 'for' iterator.
        log(beer);
        break;
    }
    results.close();                           // End the query and free resources held
}

The embedded SQL++ call starts the query and returns a JavaScript Iterable object representing the result set of the query. The query is streamed in batches as the iteration proceeds. The returned handle can be iterated using any standard JavaScript mechanism including for…of loops.

In multiline SQL++ statements (as above) you cannot use single line // end of line comments like this
prior to the terminating semicolon as it will cause a syntax error in the transpilation of the SQL++ statement, however multiline /* comments like this */ are allowed.

The iterator is an input iterator (elements are read-only). The keyword this cannot be used in the body of the iterator. The variables created inside the iterator are local to the iterator.

The returned handle must be closed using the close() method defined on it, which stops the underlying SQL++ query and releases associated resources.

When an Eventing Function completes for a given mutation and exits all resources will be freed even if you omit the close() statement for your result set(s). However in some complex use cases such as nested SQL++ lookups a failure to explicitly call close() after each result set is no longer needed can tie up an excessive amount of SQL++ resources and lead to poor performance.

All three operations, i.e., the SQL++ statement, iterating over the result set, and closing the Iterable handle can throw exceptions if unexpected error arises from the underlying SQL++ query.

As SQL++ is not syntactically part of the JavaScript language, the Eventing Function code is transpiled to identify valid SQL++ statements which are then converted to a standard JavaScript function call that returns an Iterable object with addition of a close() method.

You must use $<variable>, as per SQL++ specification, to use a JavaScript variable in the query statement. The object expressions for substitution are not supported and therefore you cannot use the meta.id expression in the query statement.

Instead of meta.id expression, you can use var id = meta.id in an SQL++ query.

Invalid SQL++ Statement

DELETE FROM mybucket.myscope.transactions WHERE username = $meta.id;

Valid SQL++ Statement

var id = meta.id;
DELETE FROM mybucket.myscope.transactions WHERE username = $id;

When you use a SQL++ query inside a Eventing Function, remember to use an escaped identifier for keyspaces (bucket.scope.collection) with special characters (`bucket-name`). Escaped identifiers are surrounded by back ticks and support all identifiers in JSON

For example:

If the bucket name is beer-sample and the scope and collection are both _default, then only the bucket in the SQL++ needs to be escaped:
```
SELECT * FROM `beer-sample`._default._default WHERE type ...
```
However if the bucket name was beersample, then the keyspace of the SQL++ query needs no escaping:
```
SELECT * FROM beersample._default._default WHERE type ...
```

Built-in Functions

The following built in functions have been added:

The N1QL() function call
The crc64() function call
The createTimer() function call
The cancelTimer() function call
The curl() function call

The N1QL() Function Call

The N1QL() function call is documented below for reference purposes but should not used directly as doing so would bypass the various semantic and syntactic checks of the transpiler (notably: recursive mutation checks will no longer function, and the statement will need to manual escaping of all SQL++ special sequences and keywords).

In addition the N1qlQuery() is now deprecated and has been replaced with the N1QL() call which has a different parameter format.

statement

This is the identified SQL++ statement. This will be passed to SQL++ via SDK to run as a prepared statement. All referenced JS variables in the statement (using the $var notation) will be treated by SQL++ as named parameters.

params

This can be either a JavaScript array (for positional parameters) or a JavaScript map. When the SQL++ statement utilizes positional parameters (i.e., $1, $2 …), then params is expected to be a JavaScript array corresponding to the values to be bound to these positional parameters. When the SQL++ statement utilizes named parameters (i.e., $name), then params is expected to be a JavaScript map object providing the name-value pairs corresponding to the variables used by the SQL++ statement. Positional and named value parameters cannot be mixed.

Note, adding an optimal index to the travel-sample._default.`_default keyspace for the below query can increase the performance by 57X.

iterator using a positional params array

    // Using `travel-sample`._default._default to demonstrate params.
    // a) Positional param 1 is field 'iata' from the input doc
    // b) Positional param 2 from an Eventing Function variable: max_dist
    // c) Will also prepare the statement for better performance

    if (doc.type !== "airline") return; // only process airline docs

    var max_dist = 120;
    var results = N1QL(
        "SELECT COUNT(*) AS cnt " +
        "FROM `travel-sample`._default._default " +
        "WHERE type = \"route\" " +
        "AND airline = $1 AND distance <= $2",
        [doc.iata,max_dist],
        { 'isPrepared': true }
    );

Example iterator using a named params object

    // Using `travel-sample`._default._default to demonstrate named params.
    // a) Named param 1 '$mytype' is a hardcode
    // b) Named param 2 '$myairline' is field 'iata' from the input doc
    // c) Named param 3 '$mydistance' if from an Eventing Function variable max_dist
    // d) Set the consistency in the options to none

    if (doc.type !== "airline") return; // only process airline docs

    var max_dist = 120;
    var results = N1QL("SELECT COUNT(*) AS cnt " +
        "FROM `travel-sample`._default._default " +
        "WHERE type = $mytype " +
        "AND airline = $myairline AND distance <= $mydistance",
        { '$mytype': 'route', '$mydistance': max_dist, '$myairline': doc.iata },
        { 'consistency': 'none' }
    );

options

This is a JSON object having various query runtime options as keys. Currently, the following settings are recognized:
- isPrepared
  
  This controls if the statement will be prepared. Normally, this defaults to false but can be set on a per statement basis to true for any SQL++ query that needs increased performance.
- consistency
  
  This controls the consistency level for the statement. Normally, this defaults to the consistency level specified in the overall Eventing Function settings but can be set on a per statement basis. The valid values are "none" and "request".
return value (handle)

The call returns a JavaScript Iterable object representing the result set of the query. The query is streamed in batches as the iteration proceeds. The returned handle can be iterated using any standard JavaScript mechanism including for…of loops.
- close() Method on handle object (return value)
  
  This releases the resources held by the SQL++ query. If the query is still streaming results, the query is cancelled.
Exceptions Thrown

The N1QL() function throws an exception if the underlying SQL++ query fails to parse or start executing. The returned Iterable handler throws an exception if the underlying SQL++ query fails after starting. The close() method on the iterable handle can throw an exception if underlying SQL++ query cancellation encounters an unexpected error.

The crc64() Function Call

crc64(): This function calculates the CRC64 hash of an object using the ISO polynomial. The function takes one parameter, the object to checksum, and this can be any JavaScript object that can be encoded to JSON. The hash is returned as a string (because JavaScript numeric types offers only 53-bit precision). Note that the hash is sensitive to ordering of parameters in case of map objects.

function OnUpdate(doc, meta) {
    var crc_str = crc64(doc);
    /// code here ...
}

The crc64 function can be useful in cases like suppressing a duplicate mutation from the Sync Gateway (SG), when both the Sync Gateway & Eventing are leveraging the same bucket. Basically, Sync Gateway updates metadata of the document within the bucket, which in turn generates an event for Eventing to process. Eventing can’t differentiate between events from Sync Gateway and other events (doc updates via SDK, SQL++, and others). A workaround to this double mutation issue is possible via the crc64() function.

function OnUpdate(doc, meta) {
    // Ignore documents created by Sync Gateway
    if(meta.id.startsWith("_sync") == true) return;

    // Ignore documents whose body has not changed since we last saw it
    var prev_crc = checksum_bucket[meta.id];
    var curr_crc = crc64(doc);
    if (prev_crc === curr_crc) return;
    checksum_bucket[meta.id] = curr_crc;

   // Business logic goes in here
}

Note that if multiple Eventing Functions share the same Sync Gateway crc64() checksum documents, real mutations will be suppressed and missed. In this use case make the checksum documents unique to each Eventing Function, i.e. checksum_bucket["evfunc1:" + meta.id], checksum_bucket["evfunc2:" + meta.id], etc.

Timers

Timers are asynchronous compute, which offers Eventing Functions the ability to execute in reference to wall-clock events, refer to the detailed Timers documentation.

The createTimer() Function Call: createTimer(callback, date, reference, context)

To create a timer a callback or JavaScript function will be executed at or close to the desired date. The reference is an identifier for the timer scoped to an Eventing function and callback. The context must be serializable data that is available to the callback when the timer is fired. For more information see createTimer function.

The cancelTimer() Function Call: cancelTimer(callback, reference)

To cancel a timer you can either by invoking createTimer() with the same reference of an existing timer or you can use the _cancelTimer() function. For more information see cancelTimer function.

cURL

The curl() Function Call: response_object = curl(method, binding, [request_object])

The curl() function provides a way of interacting with external entities via a REST endpoint from Eventing Functions using either HTTP or HTTPS. For more information see curl function.

Handler Signatures

The Eventing Service calls the following entry points or JavaScript functions on events (mutations or fired timers).

OnUpdate Handler
OnDelete Handler
Timer Callback Handler

OnUpdate Handler

The OnUpdate handler gets called when a document is created or modified, e.g. Insert/Update. The entry point OnUpdate(doc,meta) listens to mutations (the creation or modification of documents) in the associated source Bucket.

In this handler the following limitations exist, both limitations arise due to KV engine design choices and may be revisited in the future:

If a document is modified several times in a short duration, the calls may be coalesced into a single event due to deduplication.
It is not possible to distinguish between a Create and an Update operation.

A sample OnUpdate handler is displayed below:

function OnUpdate(doc, meta) {
  if (doc.type === 'order' && doc.value > 5000) {
    // ‘phonverify’ is a bucket alias or binding to a keyspace.
    phoneverify[meta.id] = doc.customer;
  }
}

OnDelete Handler

The OnDelete handler gets called when a document is deleted or removed due to an expiry.

The entry point OnDelete(meta,options) listens to mutations (deletions or expirations) in the associated source Bucket. You can determine if the document was deleted or expired via inspecting the optional argument "options" (a JavaScript map object with a boolean property named 'expired').

In this handler the following limitation exists. This limitation arises due to KV engine design choices and may be revisited in the future:

it is not possible to get the value of the document that was just deleted or expired.

A sample OnDelete handler is displayed below:

function OnDelete(meta,options) {
    if (options.expired) {
        log("Document expired", meta.id);
    } else {
        log("Document deleted", meta.id);
    }
    var addr = meta.id;
    var res = SELECT id from mybucket.myscope.orders WHERE shipaddr = $addr;
    for (var id of res) {
        log("Address invalidated for pending order: " + id);
    }
}

Note that the pre-6.6.0 argument syntax, OnDelete(meta), that lacks "options" is still fully supported, but you will not be able to differentiate deletion from expiration.

function OnDelete(meta) {
    log("Document deleted or expired", meta.id);
}

Timer Callback Handler

Timer callbacks are user defined JavaScript functions passed as the callback argument to the built-in createTimer(callback, date, reference, context) function call.

These handlers (JavaScript functions) are the entry points for the event when a timer (created by the specific Eventing Function) matures and fires.

A sample Timer Callback Handler, the user defined JavaScript function DocTimerCallback, is displayed below:

// Timer Callback Handler (user defined entry point)
function DocTimerCallback(context) {
	log("Timer fired running callback 'DocTimerCallback' with context: " + context);
}

// Insert/Update Handler or entry point
function OnUpdate(doc, meta) {
	// filter out docs of no interest.
	if (meta.id != 'make_timer:1') return;
	// Create a Date value 60 seconds from now
	var oneMinuteFromNow = new Date(); // Get current time & add 60 sec. to it.
	oneMinuteFromNow.setSeconds(oneMinuteFromNow.getSeconds() + 60);
	// Create a doc to hold context to pass state to the callback function.
	var context = { docId: meta.id, random_text: "arbitrary text" };
	// Create a timer that will fire an event in the future.
	log("createTimer with callback 'DocTimerCallback'");
	createTimer(DocTimerCallback, oneMinuteFromNow, meta.id, context);
}

For more information see the Timers section above and the detailed Timers documentation.

Reserved Words

Reserved words are words that cannot be used in a Eventing Function as a variable name, function name, or as a property in the Eventing Function’s JavaScript code. The following table lists the reserved words that you must refrain from using as they are used by the transpiler to integrate with Couchbase’s query language, SQL++ with Eventing.

SQL++ Keywords
ALTER	EXECUTE	MERGE	UPDATE
BUILD	EXPLAIN	PREPARE	UPSERT
CREATE	GRANT	RENAME
DELETE	INFER	REVOKE
DROP	INSERT	SELECT

SQL++ Keywords

ALTER

EXECUTE

MERGE

UPDATE

BUILD

EXPLAIN

PREPARE

UPSERT

CREATE

GRANT

RENAME

DELETE

INFER

REVOKE

DROP

INSERT

SELECT

What Happens If You Use a Reserved Word?

Let’s say you try to create a new Eventing Function with JavaScript code using a reserved word for variable names, for function names, and as a property binding value. All three cases generate a deployment error.

Reserved words as a variable name:

function get_numip_first_3_octets(ip) {
    var grant = 0;
    if (ip) {
        var parts = ip.split('.');
    }
}

Reserved words as a function name:

function grant(ip) {
    var return_val = 0;
    if (ip) {
        var parts = ip.split('.');
    }
}

During the Function deployment step, when the system validates the Eventing Function’s JavaScript code, it displays an error message such as the following:

Sample Error Message - Deployment failed: Syntax error (<line and column numbers>)
- grant is a reserved name in N1QLJs

Reserved words as a property bindings value: