Debugging and Diagnosability

    +
    Debugging and diagnostics in the Eventing Service comprises of debugging functions, functions log, and log redaction.

    Debugging Functions

    Couchbase Server, for its Eventing Service framework, includes a powerful full function online real-time Javascript Debugger. Debug is a special flag on a Function. The Debug option integrates seamlessly with the Google Chrome Debugger engine for running the Javascript code of any handler.

    Port 9140 is the default Eventing debug port. To change the default port settings, see Modifying the Debug Port.

    Debugging Workflow

    • During a debug session, a single mutation received by the Function is considered and sent to the Debugger. This technique ensures that processing of the other data mutations in the cluster does not get blocked.

    • The Worker in the Debugger pauses to trap the next event-instance, opens an ephemeral TCP port, and generates a Chrome dev-tools URL, with a session cookie that controls the debug worker.

    • All other events are unaffected except the trapped event-instance.

    • Using the Debug option, you can place breakpoints in the code and run the Function execution, one line at a time. The step-step execution helps while troubleshooting the deployed Function.

    • If the debugged event-instance completes execution, no further event-instance gets trapped for debugging.

    • If a debug session gets terminated during execution, then the mutation may be abruptly processed.

    • Debugging is a convenience-feature intended to help during Function development: it is highly discouraged for use in production environments.

    Debug mode should be avoided in production environments, as it affects the in-order processing of the document mutations as well as introducing timing related issues.

    Debugging a Function Using the Debug Option

    1. To enable debugging, navigate to Couchbase Web Console > Eventing page, from the top banner, click Settings and check the Enable Debugger option. This setting change is needed to enable the Debug button in the Edit Function page used in a step 3 below.

    2. From the Couchbase Web Console > Eventing page, click on the name of a deployed Function. The deployed Function expands to provide additional options. Click Edit JavaScript.

      debug 1
    3. From the Edit Function page, click Debug. A debug session gets activated. As a result, the next event-instance gets trapped and is forwarded to the Debugger.

      In the below screen, you can notice the message: "Waiting for mutation."

      debug 2
    During a debugging session this window must remain active. When you are presented a debug URL you must copy then paste it into a new browser window (described below). If you are using the UI to create the mutation, you must use a different browser window to create the mutation.
    1. Upon a data mutation, the debug URL in the Debugging pop-up gets populated and the Copy button is enabled.

      debug 3
    2. Click Copy in the pop-up.

    3. Paste the URL from the Debugging pop-up that you copied into a new window (or tab) in your Google Chrome browser’s address bar.

      debug 4
    4. From your Google Chrome browser, you can add breakpoints and run step-step diagnosis to debug and troubleshoot the deployed Function.

    5. From the Debugging pop-up, click Stop Debugging to terminate a debug session.

      The URL you copied is valid only for a single mutation, to debug another subsequent mutation you must click Stop debugging button and repeat from Step 3. above.

    Transpiler and Source Map

    A transpiler accepts source code provided as input from one high-level programming language and produces an equivalent code in another high-level programming language.

    Couchbase Server uses a native transpiler. This transpiler converts the handler syntax into a pure JavaScript representation that the JavaScript engine can understand. If this transpiler was unavailable, then the JavaScript engine would have failed to compile any native N1QL queries or curl() functions.

    When your source code is transformed, debugging becomes a problem because we must know where the original code is. Source maps solve this problem by providing a mapping between the original and the transformed source code.

    A source map, generated by our native transpiler provides a mapping between the transpiled code and the original function handler code. Debugging is easy as the debugger detects the source map and presents the code to the developer in the original handler format.

    Upon source map detection, a text confirmation flag gets displayed in the bottom of your browser’s debug window as a very long comment an example is highlighted below:

    //# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpb truncated

    Modifying the Debug Port

    The Eventing Service Debugger port, eventing_debug_port (9140), is an internal port and is one of the ports that is configured by the ns_server. Note this port is not supported for external access outside of the cluster and should only be used in development environments. To modify this port setting (Linux example):

    1. Install Couchbase Server.

    2. Stop the Couchbase Server service.

    3. Edit the /opt/couchbase/etc/couchbase/static_config file to add the new eventing_debug_port and the new port-number information. For example, to change the Eventing debugging port from 9140 to 9444, you would add the following line (enclosed in braces and terminated by a period):

      {eventing_debug_port, 9444}.
    4. If Couchbase Server was previously configured, you’ll need to delete the /opt/couchbase/var/lib/couchbase/config/config.dat file to remove the old configuration.

    5. Start Couchbase Server.

    For detailed information on the modifying ns_server port mappings, refer to Custom Port Mapping.

    Changing port mappings should only be done at the time of initial node/cluster setup as the required reset and reconfiguration will also purge all data on the node.

    Logging Functions

    The Eventing Service creates two different types of logs:

    • System Log

    • Application Logs

    Couchbase Server creates different application log files depending on the level and severity of the reported problem, as configured during Function definition

    System Log

    For the Eventing Service, Couchbase Server creates a separate system log file named eventing.log (refer to Table 1 for filesystem location by platform). The system log file captures information related to general management and supervision of the Eventing service (not the business logic of the Function). In addition this log file also captures life cycle or housekeeping information of every individual Eventing function depending on the Function’s "System Log Level" setting. An end user cannot write a message (via their JavaScript in a Function handler) to this log file it is intended for debugging for customers that are on support contracts.

    Table 1. Eventing System Log Location in Platform
    Platform Location

    Linux

    /opt/couchbase/var/lib/couchbase/logs/eventing.log

    Windows

    C:\Program Files\Couchbase\Server\var\lib\couchbase\logs\eventing.log
    (Assumes default installation location)

    Mac OS X

    /Users/<user>/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/eventing.log

    The eventing.log contains redactable user data and the log is collected using the cbcollect_info tool. For log rotation, refer to Log File Rotation.

    The available logging levels are: Info, Error, Warning, Debug, and Trace (Info is the default since version 6.0). The level can be altered via the "System Log Level" choice in the Settings dialog of each individual Eventing function and impacts the detail and quantity of information sent to the System Log (but has no effect on a specific Function’s Application Log).

    debug 5 log level

    Application Logs

    Application logs allow you to identify and capture various various business logic related related activities and errors via user defined messages specific to each Eventing function.

    Unlike the System Log, Application logs can be viewed in the UI for any deployed function by clicking on the function’s “Log” hyperlink in the Eventing page.

    Each Eventing function will have its own Application log based on the function name, e.g. the_function_name.log. (refer to Table 2 for filesystem path by platform). The information that goes to these log files is solely dependent the logic of the Function via log(…) statements put inside the individual Function handler’s JavaScript code. Unlike the system log there is currently is no logging level for Application logs. Application logs are primarily used for development and debugging business logic.

    function OnUpdate(doc, meta) {
        log('document', doc);
        try {
            var response = curl('http://localhost:3000/notify', {method : 'POST', data : doc});
            log('curl', response);
        } catch (e) {
            log('error', e);
        }
    }

    Application logs receive user defined messages when a log(…​) statement is encountered in the Function’s JavaScript code (they do not have a Log Level).

    As a best practice the use of log(…​) messages with try catch blocks can greatly assist Eventing Function development and debugging.

    Below a function processes one mutation from the test bucket travel-sample but has an undefined JavaScript variable a without a try catch block

    function OnUpdate(doc, meta) {
        if (meta.id != "airport_1255") return
        log('id', meta.id);
        // undefined variable 'a' stops processing, an error indicator [X] will be
        // displayed via selecting "Edit JavaScript" when the function is deployed.
        // Failure(s) will also be logged in Eventing Stats: _function_name_ in the
        // UI 'Server > Statistics' view for the charts of the _function_name_
        if (a == 1) {
            log('a is 1')
        }
        // never reached
        log('complete')
    }

    Only the first of three log messages is emitted and the processing stops and is marked as failed in the statistics.

    2020-02-09T07:12:17.936-08:00 [INFO] "id" "airport_1255"

    By selecting the "Edit JavaScript" button when the function is deployed a [X] indicator at the exact line in the JavaScript code failed will be displayed. By hovering over the [X] indicator more information is revealed.

    However by adding a try catch block the root cause of the failure is easily apparent, and the function is considered processed (without a failure) because the error is caught and handled in the function.

    function OnUpdate(doc, meta) {
        if (meta.id != "airport_1255") return
        log('id', meta.id);
        try {
            if (a == 1) {
            log('a is 1')
            }
        } catch (e) {
            log('error', e);
        }
        log('complete')
    }

    The output now indicates the exact error that occured via the Function’s Application log as follows:

    2020-02-09T07:12:17.936-08:00 [INFO] "id" "airport_1255"
    2020-02-09T07:12:17.936-08:00 [INFO] "error" "ReferenceError: a is not defined"
    2020-02-09T07:12:17.936-08:00 [INFO] "complete"

    As previously indicated, by selecting the "Edit JavaScript" button when the function is deployed a [X] indicator at the exact line in the JavaScript code failed will be displayed. By hovering over the [X] indicator more information is revealed.

    You can access a Function’s Application log file using the UI by selecting the Function name and clicking on the 'Log' hyperlink/button or by opening a terminal and issuing Linux commands such as cat, more, head, tail, or ‘tail -F’ on a specific Eventing function’s log.

    Couchbase Server creates an individual log file for every Function in the cluster on each Eventing node. Application logs will only contain information for the mutations processed on a given Eventing node.

    By default, the maximum size of a node’s Application log file is 40MB, and the number of log files before rotation is 10. Unlike system logs, the Application logs are user-configurable in number and size.

    The cbcollect_info tool does not collect the Application log files.
    Table 2. Eventing Application Logs Location by Platform
    Platform Location

    Linux

    /opt/couchbase/var/lib/couchbase/data/@eventing/

    Windows

    C:\Program Files\Couchbase\Server\var\lib\couchbase\data\@eventing\
    (Assumes default installation location)

    Mac OS X

    /Users/<user>/Library/Application\ Support/Couchbase/var/lib/couchbase/data/@eventing/

    During Cluster setup, if you have chosen a custom path, then the path for Application logs is same as that of the selected Indexes Path. The @eventing folder in the selected Indexes Path stores the Application logs.

    To configure an Application log, use the REST endpoint settings option. Note you must always specify deployment_status (deployed/undeployed) and processing_status (paused/not-paused) when using this REST endpoint.

    Sample URL: 192.168.1.5:8091/_p/event/api/v1/functions/<Function_name>/settings

    Sample Payload:

    {
      "settings":
        {
          "deployment_status":false,
          "processing_status":false,
          "app_log_max_files": 10,
          "app_log_max_size": 10485760
        }
      }

    The sample payload above illustrates that the system stores 10 application log files and each file records about 10 MB of data.

    At some point in time, old application log files that are no longer necessary need to be deleted to make way for new log records. When an Application log file reaches the set limit, a new log file gets created. All the recorded information from the active log file gets transferred to this newly created file.

    For illustration, consider case_1_enrich_ips from the example Data Enrichment as the name of the Function. A corresponding Application log file, case_1_enrich_ips.log, gets created in the Couchbase cluster. Whenever the case_1_enrich_ips.log reaches 10MB in size, assuming the maximum size of an Application log file is 10MB and the number of log files before rotation is 10, the system automatically generates the case_1_enrich_ips.log.1 file, during its first iteration. The file case_1_enrich_ips.log transfers all the log information to this new log file. For this illustration, since the number of log files is 10, the system stores 10 such files, the currently active log file along with 9 truncated files, at any given instance.

    Log Redaction

    You can use logs for multiple purposes ranging from security, monitoring, and diagnostics. Suppression of sensitive data such as personally identifiable information (PII), hostnames, internal asset information, credit card details, during the logging operation is termed as log redaction. Organizations implement log redaction as part of their legal compliance and security risk mitigations.

    Couchbase Server provides a capability to redact sensitive user data from getting captured in the logs. All sensitive data are scrubbed and gets removed from the log files. Post redaction, log files can be shared for troubleshooting without disregarding any regulatory compliance.

    Log redaction is applicable only for System logs and not for Application logs.

    For details, see Understanding Redaction.