A newer version of this documentation is available.

View Latest

Webhooks

      +

      Introducing Sync Gateway events and event handling with Webhooks
      You can configure webhooks to detect document_changed events and post the changed documents to specified URLs.

      Related integration topics: Changes Feed | Prometheus Integration

      Caveats

      Webhooks post your application’s data, which might include user data, to URLs. Consider the security implications.

      Introduction

      Sync Gateway provides the ability to extend the replication process and build responsive services that react to changes in documents, adding value to the end to end process.

      For example, by initiating the sending of notifications, or specialist audit processes, when certain document changes are detected.

      This can be done using either the changes feed or document_changed events — see Table 1 for a comparison of these scenarios.

      Sync Gateway’s webhook event handlers perform both document filtering, and HTTP POST operations, asynchronously.

      In addition to providing the opportunity to integrate with external systems, this minimizes:

      • The performance impact on Sync Gateway’s regular processing

      • The amount of Sync Gateway node CPU resources consumed by slow response times from the HTTP POST operations.

      Behavior

      Webhooks work on the push-cycle of a replication.

      If a webhook event handler is defined:

      • Sync Gateway adds a document_changed event to the event queue [1] whenever it updates a document in a Couchbase Server bucket. These changes [2] can arise from Sync Gateway’s Public REST API and-or Couchbase Lite push replications.

      • Whenever the event queue contains a data_changed event, Sync Gateway spawns a process to:

        • FILTER — decide which changed documents to post.
          The event process executes the webhook’s event handler (the filter) on the associated document. The filter determines which documents need to be POSTed.

          If there is no filter all document changes are passed to POST.

        • POST — send selected changed documents to the URL endpoint.
          HTTP/HTTPS is used to POST the document changes selected by the filter to the defined url.

      Table 1. Changes Feed vs Webhooks
      Scenario Changes feed (pull) Webhooks (push)

      Sequence/Ordered

      Yes

      No

      User Access Control

      Fine Grain

      Limited

      Scalable

      Yes

      No

      Data Stream replay on Failure

      Yes

      No

      Definition

      You can define Webhooks using the Admin Rest API Database Configuration endpoints. For Pre-3.0 Legacy configurations, see the Legacy Pre-3.0 Configuration at the database level.

      Sync Gateway supports the use of Javascript functions to customize the sync process. These functions are referenced from within the Sync Gateway Configuration and may be provided either as:

      • An inline Javascript function

      • An external Javascript file

      • An external HTTP/HTTPS endpoint serving a JS function [3].

      Learn more about this property ($db.event_handlers) in the Configuration Schema Reference — see: database.event_handlers.

      Sync gateway 3.x configuration of Javascript functions is done using the Admin REST API; specifically the Access Control and /{db}/_config/import_filter endpoints.

      Prior to this, configuration was done within the database configuration file — see: Example 1

      • Inline Javascript functions provided within the database configuration must be enclosed by a backtick pair (``).

      • To use an external Javascript function for any of the eligible options, you need to specify the absolute path to the Javascript. The format and content of the external Javascript is the same as that provided inline.

        You must register a CA certificate for the appropriate server if external Javascript functions are hosted on HTTPS endpoints.
        For testing purposes you may use the unsupported configuration option unsupported.remote_config_tls_skip_verify. Setting this true will side-step essential security checks. Do not use in Production deployments.
      Example 1. Configuring a Javascript Sync Function

      This example shows the different ways you might provide a Javascript Sync Function. Although the example uses the Sync Function, the same approach applies wherever a Javascript function is valid (including with Import Filter, Webhook and Custom Conflict Resolver).

      curl -X PUT 'http://localhost:4985/db1/_config' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json' \
      --data-raw '{
               "sync": "/opt/couchbase-sync-gateway/sync.js" (1)
            },
          }
      }'
      
      
        curl -X PUT 'http://localhost:4985/db2/_config' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json' \
      --data-raw '{
               "sync": "https://localhost/sync/func2" (2)
            }
         }
      }
      
      
      curl -X PUT 'http://localhost:4985/db3/_config' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json' \
      --data-raw '{
               "sync": `function(doc,oldDoc, meta){ if (doc.published) { channel("public");} }`
            } (3)
         }
      }
      1 Here we specify an external file sync.js as containing the external function to be provisioned
      2 Here we specify a HTTPS endpoint as resolving to a Javascript function to be provisioned
      3 Here we specify inline Javascript (surrounded by a pair of backticks (``) as the function to be provisioned

      You can define multiple webhook event handlers. For example, you could define webhooks with different filtering criteria to post changed documents to different URLs — see: Example 2.

      Document Change Properties

      Each event handler definition comprises the following properties:

      A Filter

      The filter is a JavaScript function used to determine which documents to post. It accepts the document body as input and returns a boolean value.

      • If the filter function returns true, then Sync Gateway posts the document.

      • If the filter function returns false, then Sync Gateway does not post the document.

      • If no filter function is defined, then Sync Gateway posts all changed documents.

      Filtering only determines which documents to post. It does not extract specific content from documents and post only that.

      An event handler type

      Property name: database.event_handlers.document_changed.handler

      Sets the event handler’s type; currently, this must be webhook.

      A timeout value

      Sets the time (in seconds) to wait for a response to the POST operation. It ensures that slow-running POST operations don’t cause the webhook event queue to back up. When the timeout limit is reached, Sync Gateway stops listening for a response and discards the operation.

      You should not need to adjust the default setting to tune performance.

      URL

      Sets the address to which documents are posted.

      Example 2. Sample Webhook Definitions
      • Simple Webhook

      • Multiple Webhooks

      In this simple example of a webhook event handler we define a single instance with no filter. It simply listens for the document_changed event and immediately sends the changed document to the URL http://someurl.com.

      "event_handlers": {
          "document_changed": [
              {
                  "handler": "webhook",
                  "url": "http://someurl.com"
              }
          ]
      }

      In this example we define two webhook event handlers, both of which use filters to decide how to process the changed document.

      The filter function in the first handler recognizes documents with doc.type equal to A and posts the documents to the URL http://someurl.com/type_A.

      The filter function in the second handler recognizes documents with doc.type equal to B and posts the documents to the URL http://someurl.com/type_B.

      "event_handlers": {
            "document_changed": [
              {"handler": "webhook",
               "url": "http://someurl.com/type_A",
               "filter": `function(doc) {
                    if (doc.type == "A") {
                      return true;
                    }
                    return false;
                  }`
               },
              {"handler": "webhook",
               "url": "http://someurl.com/type_B",
               "filter": `function(doc) {
                    if (doc.type == "B") {
                      return true;
                    }
                    return false;
                  }`
              }
           ]
        }

      Event Processing Properties

      Limited Concurrent Processes

      Sets the maximum number of events that can be processed concurrently. The default value should work well in the majority of cases. You should not need to adjust it to tune performance. However, if you wish to ensure that most webhook posts are sent, you can set it to sufficiently high value.

      Limited Full-Queue Wait Time

      Sets the maximum time (milliseconds) that event processing will wait for a free process, if an event is received whilst the event queue is full. You should not need to adjust it to tune performance.

      To avoid blocking standard Sync Gateway processing, set a zero value. Any events arriving whilst the queue is full are then immediately discarded — see also Logging.

      Logging

      Sync Gateway creates a log whenever an event is discarded, and not added to the event queue.

      You can configure the console logging of events using the configuration file and-or the ADMIN Rest API — see Logging. The log_key you need to include is Event; or Events+ for more verbose output.



      1. An asynchronous event-processing queue
      2. Creations, updates, and deletions
      3. Sync Gateway 3.x