Logging and Telemetry
Logging
The SDK logs from two places. The native core, which talks to the cluster, has a log of its own. The
Python side of the SDK logs through Python’s logging, under loggers named couchbase:
couchbase.threshold, for Threshold logging;couchbase.metrics, for Metrics logging;couchbase.transactions, for transactions.
The native core logs nothing until it is asked to, in one of two ways, and an application uses one or the other.
From the environment
PYCBC_LOG_LEVEL turns on the native core’s log, and sets the level of the couchbase
logger to match. It needs no change to the code, which makes it the way to investigate a problem in an
application already deployed. Environment Variables describes it and the variables that go
with it.
From the code
couchbase.configure_logging() sends the native core’s log to a Python logger instead, so its
records reach the handlers the application already has. It takes the logger’s name and the level to
log at:
import logging
from couchbase import configure_logging
logging.basicConfig(level=logging.INFO)
# The native core's records, through the logger 'my_app.couchbase'.
configure_logging('couchbase', level=logging.DEBUG, parent_logger=logging.getLogger('my_app'))
# The SDK's own Python loggers are configured like any other.
logging.getLogger('couchbase').setLevel(logging.INFO)
It can be called before or after the first cluster connects, and raises RuntimeError when
PYCBC_LOG_LEVEL is set.
Redaction
A log can hold document keys, query statements, bucket names and hostnames, which an application may
not be free to share. log_redaction=True in ClusterOptions, or
log_redaction=true in the connection string, makes the SDK tag each such value where it writes it,
in the native core’s log and in the couchbase loggers alike:
<ud>...</ud>, user data: document keys and values, query statements, usernames;<md>...</md>, metadata: bucket, scope, collection and index names;<sd>...</sd>, system data: hostnames, IP addresses and ports.
The threshold logging report, for example, is then logged as a whole as system data:
<sd>{"kv":{"total_count":3,"top_requests":[{"operation_name":"upsert","total_duration_us":1500}]}}</sd>
The SDK only tags; it removes nothing, so the log stays as useful for the application’s own
debugging. Redacting happens afterwards, on a copy of the log, with cblogredaction, the tool that
ships with Couchbase Server. It replaces each piece of tagged user data with a salted hash, so a value
logged twice still matches itself, and the copy can be shared.
Redaction is for the whole process: once any cluster connects with it on, it stays on, for every cluster, until the process exits.
Threshold logging
Unless told otherwise, the SDK times every operation, and every 10 seconds logs a report of the slowest
operations that took longer than a threshold for their service: 500 milliseconds for key-value
operations, and a second for the other services. The report is one line of JSON at
INFO, to the couchbase.threshold logger, with up to 10 operations per service:
{"kv":{"total_count":3,"top_requests":[{"operation_name":"upsert","total_duration_us":1500,
"last_dispatch_duration_us":1200}]}}
The thresholds, the number of operations kept and the interval are set by
ClusterTracingOptions:
from datetime import timedelta
from couchbase.options import ClusterOptions, ClusterTracingOptions
tracing = ClusterTracingOptions(tracing_threshold_kv=timedelta(milliseconds=100),
tracing_threshold_queue_flush_interval=timedelta(seconds=30))
options = ClusterOptions(auth, tracing_options=tracing)
enable_tracing=False turns threshold logging off, and an application’s own tracer, given as
tracer in ClusterOptions, replaces it.
Metrics logging
Unless told otherwise, the SDK records the latency of every operation, and every 10 minutes logs the
percentiles for each service and operation. The report is one line of JSON at INFO, to the
couchbase.metrics logger:
{"meta":{"emit_interval_s":600.0},"operations":{"kv":{"upsert":{"total_count":100,
"percentiles_us":{"50.0":150,"90.0":350,"99.0":800,"99.9":1200,"100.0":1500}}}}}
ClusterMetricsOptions sets the interval, as emit_interval, and
enable_metrics=False turns it off. An application’s own meter, given as meter in
ClusterOptions, replaces it.
Both reports are logged at INFO, and Python’s default logging shows only WARNING and above, so
an application that has not configured logging sees neither.
Orphaned responses
A response that arrives after its operation has timed out or been cancelled is an orphan. The native
core collects them, and every 10 seconds logs a sample at WARNING, in its own log.
ClusterOrphanReportingOptions sets the sample size and the interval, and
enable_orphan_reporting=False turns it off.
Turning it off
Threshold logging and metrics logging run on the Python side of the SDK, and each costs every operation a little. An application that needs neither turns both off:
options = ClusterOptions(auth, enable_tracing=False, enable_metrics=False)
With both off, the SDK skips its tracing and metrics work for each operation entirely. With only one
off, that report stops, but every operation still does the work for the other. A tracer or a
meter given in ClusterOptions turns its half back on, whatever its flag
says.
Neither flag affects the report of orphaned responses, which the native core makes.
enable_orphan_reporting=False turns that off.
OpenTelemetry
The SDK can send its spans and its metrics to OpenTelemetry instead of logging them. It needs the
otel extra:
$ python3 -m pip install "couchbase[otel]"
get_otel_tracer() and get_otel_meter()
return a tracer and a meter for the cluster’s options, from the global OpenTelemetry providers or from
the ones given:
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.trace import TracerProvider
from couchbase.observability import get_otel_meter, get_otel_tracer
from couchbase.options import ClusterOptions
tracer_provider = TracerProvider()
meter_provider = MeterProvider()
# Add the application's span processors and metric readers to the providers here.
options = ClusterOptions(auth,
tracer=get_otel_tracer(tracer_provider),
meter=get_otel_meter(meter_provider))
A tracer given this way replaces threshold logging, and a meter replaces metrics logging. To make an
operation’s span the child of one of the application’s own, pass that span as the operation’s
parent_span option.
The SDK’s examples export both to an OTLP collector, with Jaeger and Prometheus behind it.
Over couchbase2
Over couchbase2:// each attempt at an operation is one gRPC call to the gateway, and gets a span
under the operation’s, named for the call as the gateway names its own span for it, such as
couchbase.kv.v1.KvService/Get. With the OpenTelemetry tracer the call carries that span’s trace
context in a traceparent header, so the gateway’s span for the call is a child of it.
opentelemetry-instrumentation-grpc does the same for every gRPC channel opened after it is enabled.
When it was enabled before the cluster was created, the SDK leaves the call’s span and its trace context
to the instrumentation, and makes the operation’s span current while each call starts, so the
instrumentation’s span sits under the operation’s. The instrumentation names its spans with a leading
/. The SDK does not install it.
Another tracing or metrics library
A tracer is a RequestTracer that creates
RequestSpan objects, and a meter is a
Meter that creates
ValueRecorder objects. To send the SDK’s spans or metrics
to a library other than OpenTelemetry, implement those interfaces and pass the tracer or the meter in
ClusterOptions, as above.