Threshold Logging Tracing through the SDK

Why Tracing?

Tracing is recording details about discrete steps or phases of a request lifecycle, such as request encoding / decoding, or dispatching to server. These phases are timed independently, and contain additional contextual information. A single request is typically made up of multiple tracing points.

Open Tracing

OpenTracing is a standardised API to structure tracing information in a consistent and predictable manner.

Threshold Configuration Settings

The threshold tracer receives completed spans and verifies if an operation has exceeded the given threshold for the operation type. Operations that exceed the threshold are periodically logged with a total count and a sample of the slowest ones. The following are Python SDK configuration properties.

Table 1. Threshold Logging Tracer Properties
Property Name Description Default Value

threshold_logging_tracer_interval

The interval between executions that process the collected operation spans. Expressed in milliseconds.

10,000 (10 seconds)

tracing_threshold_queue_size

The maximum number of items to log per service.

10

tracing_threshold_kv

The KV operation operation threshold. Expressed in microseconds.

500,000 (500 milliseconds)

tracing_threshold_view

The View query operation threshold. Expressed in microseconds.

1,000,000 (1 second)

tracing_threshold_n1ql

The N1QL query operation threshold. Expressed in microseconds.

1,000,000 (1 second)

tracing_threshold_fts

The FTS query operation threshold. Expressed in microseconds.

1,000,000 (1 second)

tracing_threshold_analytics

The Analytics query operation threshold. Expressed in microseconds.

1,000,000 (1 second)

tracing_orphaned_queue_flush_interval

The interval used to flush orphaned response information to the log. Expressed in microseconds.

10,000 (10 seconds)

tracing_orphaned_queue_size

The number of sample orphaned responses whose to log additional information for per execution.

10

Threshold Logging in Python

Threshold tracing is enabled by default. It can be disabled explicitly by adding enable_tracing=false to the connection string/open_bucket options. Here is the code to override the default values of the tracer:

options = dict(hostname="localhost",bucket="default")
bucket = Bucket("couchbase://{hostname}/{bucket}".format(**options))
bucket.tracing_threshold_queue_flush_interval = 5000 # 5 seconds
bucket.tracing_threshold_queue_size = 5
bucket.tracing_threshold_kv = 500000 # 500 ms
bucket.tracing_threshold_n1ql = 1000000 # 1 second
bucket.tracing_threshold_view = 1000000 # 1 second
bucket.tracing_threshold_fts = 1000000 # 1 second
bucket.tracing_threshold_analytics = 1000000 # 1 second

In order to see warnings from the threshold tracer, you will need to enable logging:

import couchbase
import logging
couchbase.enable_logging() # allows us to see warnings for slow/orphaned operations

# set logging to WARNING level so we can see warnings
ch = logging.StreamHandler()
ch.setLevel(logging.WARNING)
logging.getLogger().addHandler(ch)

One can also pass in an OpenTracing tracer in the Bucket options, e.g.:

from opentracing_pyzipkin.tracer import Tracer
import requests

def http_transport(encoded_span):
    # The collector expects a thrift-encoded list of spans.
    requests.post('http://localhost:9411/api/v1/spans',
        data=encoded_span,
        headers={'Content-Type': 'application/x-thrift'})

def jaeger_tracer(service, port = 9414, **kwargs ):
    port = 9411
    tracer= Tracer("My Tracer Name", 100, http_transport, port)
    return tracer

options = dict(hostname="localhost",bucket="default")
bucket = Bucket("couchbase://{hostname}/{bucket}".format(**options), tracer = jaeger_tracer())

This will also log spans to the provided tracer. At present the interface does not populate the 'parent' field in the start_span function, so hierarchical information is lost. This feature will be added in an upcoming Python SDK release.