Configure Log Forwarding
Configure Couchbase Server logs to be forwarded to standard console output.
Overview
Couchbase Server produces a variety of log files. By default, these logs cannot be collected from the Couchbase Server container’s standard console output. This limits the ability to integrate Couchbase logging with 3rd-party monitoring or collection technologies deployed within a Kubernetes cluster.
However, the Autonomous Operator can optionally enable log forwarding by deploying a third party log processor in a sidecar container on each Couchbase pod, which then reads the log files and forwards them to standard console output. For this purpose, Couchbase supplies a default log processor image based on Fluent Bit.
The sections on this page describe how to enable and configure log forwarding using the Couchbase-supplied log processor image.
The Couchbase-supplied log processor container image is only supported on Kubernetes platforms in conjunction with the Couchbase Autonomous Operator. |
Log forwarding requires that logs be written to a persistent volume (i.e. the Couchbase deployment’s default or logs volumes are backed by persistent storage).
Fully-ephemeral clusters are not supported by this feature.
|
Log forwarding requires that couchbaseclusters.spec.securityContext is set as appropriate for your Kubernetes distribution, as the sidecar requires read access to the shared volume.
|
Enabling Log Forwarding
Log forwarding is fundamentally provided by two components:
-
the log processor image that is used by the Autonomous Operator to deploy the
logging
sidecar container onto each Couchbase Server pod, and -
the log forwarding configuration, stored in a Kubernetes Secret, that gets consumed by the
logging
sidecar container and which controls its behavior.
Enabling log forwarding for a particular Couchbase cluster involves enabling these two components in the CouchbaseCluster
resource specification.
The required configuration parameters for enabling log forwarding are described in the example below. Specified values represent the defaults for their respective fields unless otherwise noted in a callout. (The Autonomous Operator will set the default values for any fields that are not specified by the user.)
CouchbaseCluster
Parameters That Enable Log ForwardingapiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
logging:
server:
enabled: true (1)
sidecar:
image: "couchbase/fluent-bit:1.2.0" (2)
audit:
enabled: false (3)
garbageCollection:
sidecar:
enabled: false
1 | couchbaseclusters.spec.logging.server.enabled : This is technically the only field that must be changed in order to enable log forwarding.
Setting this field to true (defaults to false ) instructs the Autonomous Operator to deploy the logging sidecar container on each pod. |
2 | couchbaseclusters.spec.logging.server.sidecar.image : This field specifies the container image that the Autonomous Operator will use for deploying the logging sidecar container.
This field defaults to couchbase/fluent-bit:1.2.0 , which pulls the Couchbase-supplied log processor image.
You may need to modify this field if your Kubernetes nodes can’t pull from the Docker public registry. |
3 | (Optional) couchbaseclusters.spec.logging.audit.enabled : This field is used to enable Couchbase audit logging, and defaults to false .
Unlike other Couchbase logs, audit logging is not turned on by default.
If you intend to forward audit logs, you must configure audit logging first. |
After deploying the CouchbaseCluster
resource specification, the logs will be available on the standard output of the logging
container on each pod:
$ kubectl logs cb-example-0000 logging
Configuring Log Forwarding
As described in Enabling Log Forwarding, there are two primary components that provide log forwarding: the log processor image and the log forwarding configuration. The Autonomous Operator uses Couchbase-provided defaults for both of these components, which can be used as-is (refer to Using Default Log Forwarding). However, these components can be further customized to better meet the needs of your environment (refer to Using Custom Log Forwarding).
Enable memory buffer limits
The Fluent Bit input plugins have memory buffer limits that can be used to help prevent the container from being terminated for exceeding memory limits. Once an input plugin reaches its memory buffer limit, all new incoming data is instead buffered to file system.
Memory buffer limits are disabled by default and can be enabled by adding the annotation fluentbit.couchbase.com/mem.buf.limits.enabled
to a pod.
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
servers:
- name: default
pod:
metadata:
annotations:
fluentbit.couchbase.com/mem.buf.limits.enabled: "true"
See Fluent Bit Memory Buffer Limits and Fluent Bit Buffering and Storage from more information.
Using Default Log Forwarding
A Couchbase deployment is considered to be using default log forwarding when the Couchbase-supplied defaults for both the log processor image and the log forwarding configuration are being used.
CouchbaseCluster
Parameters Involved with Default Log ForwardingapiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
logging:
server:
enabled: true
manageConfiguration: true (1)
configurationName: "fluent-bit-config" (2)
sidecar:
image: "couchbase/fluent-bit:1.2.0" (3)
configurationMountPath: "/fluent-bit/config/"
resources:
requests:
cpu: 100m
memory: 500Mi
1 | couchbaseclusters.spec.logging.server.manageConfiguration : When this field is set to true (the default), the Autonomous Operator ensures that the logging sidecar container always uses the default log forwarding configuration.
The Autonomous Operator accomplishes this by creating a Secret that contains the default log forwarding configuration, which is then consumed by the logging sidecar container.
The name of the Secret will be whatever name is specified in couchbaseclusters.spec.logging.server.configurationName .
If a Secret already exists with the same name, the Autonomous Operator overwrites it, ensuring that the default log forwarding configuration is maintained. |
||
2 | couchbaseclusters.spec.logging.server.configurationName : This field defines the name of the Secret that contains the log forwarding configuration that will be consumed by the logging sidecar container.
This field defaults to fluent-bit-config and will be the name of the Secret that gets automatically created by the Autonomous Operator when couchbaseclusters.spec.logging.server.manageConfiguration is set to true .
|
||
3 | couchbaseclusters.spec.logging.server.sidecar.image : This field specifies the container image that the Autonomous Operator will use for deploying the logging sidecar container.
This field defaults to couchbase/fluent-bit:1.2.0 , which pulls the Couchbase-supplied log processor image.
You may need to modify this field if your Kubernetes nodes can’t pull from the Docker public registry.
|
By default, this configuration will only output to the standard console output, however, there are other output plugins included that do not match any existing streams.
It is possible to enable matching of streams to output plugins by setting the fluentbit.couchbase.com/<PLUGIN>_MATCH
annotation.
The following is an example for Loki usage.
pod: metadata: annotations: fluentbit.couchbase.com/loki.match: "*" fluentbit.couchbase.com/loki.host: loki.monitoring
Other supported output plugins include Elasticsearch and Splunk. See Configuring Couchbase Fluent Bit for more detailed output configuration options.
Using Custom Log Forwarding
A Couchbase deployment is considered to be using custom log forwarding when either a custom log processor image and/or a custom log forwarding configuration is being used for log forwarding.
CouchbaseCluster
Parameters Involved with Custom Log ForwardingapiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
logging:
server:
enabled: true
manageConfiguration: false (1)
configurationName: "fluent-bit-config" (2)
sidecar:
image: "couchbase/fluent-bit:1.2.0" (3)
configurationMountPath: "/fluent-bit/config/" (4)
resources:
requests:
cpu: 100m
memory: 500Mi
1 | couchbaseclusters.spec.logging.server.manageConfiguration : When this field is set to false (defaults to true ), the Autonomous Operator allows the Secret specified in couchbaseclusters.spec.logging.server.configurationName to contain a non-default log forwarding configuration.
This field must be set to false in order to use a custom log forwarding configuration, otherwise the Autonomous Operator will always reconcile and update the Secret with the default log forwarding configuration.
|
||||
2 | couchbaseclusters.spec.logging.server.configurationName : This field defines the name of the Secret that contains the custom log forwarding configuration that will be consumed by the logging sidecar container.
|
||||
3 | couchbaseclusters.spec.logging.server.sidecar.image : If desired, you can use this field to specify a custom log processing image.
A few reasons for specifying a custom image include adding certificates for self-signed endpoints internally, or using another implementation of either the Fluent Bit image or any other log forwarder/processor. |
||||
4 | couchbaseclusters.spec.logging.server.sidecar.configurationMountPath : This field defines the location to mount the Secret into the logging sidecar container.
This field currently defaults to the usual Fluent Bit mount path (/fluent-bit/config/ ).
This field only needs to be modified if a custom image is specified in couchbaseclusters.spec.logging.server.sidecar.image , and only if that image requires a non-default mount path. |
Creating a Custom Log Forwarding Configuration
Since a log forwarding configuration can contain sensitive information, it is stored in a Kubernetes Secret. The Secret is used to populate the volume mounted into the sidecar so that the actual configuration files get picked up by the log processor.
When using default log forwarding, the Autonomous Operator automatically creates a Secret named fluent-bit-config
that contains the default log forwarding configuration.
This configuration, which can be viewed on GitHub, is used by the Couchbase-supplied log processor image in order to provide features like automatic parsing of a number of supported logs, automatic filtering for adding useful information to the logs (e.g. pod host names), as well as optional log redaction (another type of filtering).
When using custom log forwarding, the Autonomous Operator does not automatically create a log forwarding configuration Secret.
Instead, the administrator is expected to create a custom Secret themselves, and specify that Secret in couchbaseclusters.spec.logging.server.configurationName
.
The rest of this section covers some relatively simple ways in which you can customize the default log forwarding configuration, such as limiting the number of forwarded logs, and enabling log redaction. However, since the Couchbase-supplied log processor image is based on Fluent Bit, you can technically customize the log forwarding configuration with any kind of configuration that is supported by Fluent Bit. Some examples of these types of customizations are covered in the tutorial Configure Log Forwarding.
Only custom configurations that leverage the built-in parsing, redaction, and other standard features of the Couchbase-supplied log processor image are supported. |
Creating a Secret That Contains a Custom Log Forwarding Configuration
The following is an example of a custom log forwarding configuration Secret that is based on the default configuration, but which limits the number of Couchbase logs that are processed and forwarded to just the Couchbase audit log:
apiVersion: v1
kind: Secret
metadata:
name: fluent-bit-config (1)
stringData:
fluent-bit.conf: | (2)
[SERVICE]
flush 1
daemon Off
log_level Warning
parsers_file /fluent-bit/etc/parsers-couchbase.conf (3)
# @include /fluent-bit/etc/couchbase/in-*.conf (4)
@include /fluent-bit/etc/couchbase/in-audit-log.conf (5)
[FILTER]
Name modify
Match couchbase.log.*
Add pod ${HOSTNAME}
Add logshipper couchbase.sidecar.fluentbit
[OUTPUT] (6)
name stdout
match couchbase.log.audit (7)
1 | metadata.name : The Autonomous Operator expects the default name fluent-bit-config .
If you use a different name, then you will need to specify that name in couchbaseclusters.spec.logging.server.configurationName . |
||
2 | stringData.fluent-bit.conf : Log forwarding configuration files are defined here in the stringData field of the Secret (the data field is also potentially an option).
The fluent-bit.conf file is specified with a field that starts with a space and a vertical bar (| ), followed by the contents of the main configuration.
Note that Fluent Bit is very particular about its format and schema — files must all follow the same indentation. |
||
3 | /fluent-bit/etc/parsers-couchbase.conf : This is the path to the built-in file that contains all of the default parser configurations in the Couchbase-supplied log processor image (you can view this file on GitHub).
You should only have to modify this if you’re using a custom log processor image that invokes a parser file from a different location. |
||
4 | @include /fluent-bit/etc/couchbase/in-*.conf : This is the default input configuration that processes most Couchbase logs.
Since the example is meant to only process the Couchbase audit log, this line is commented out.
Couchbase logs are supported by individual input configurations, with those that are included in the default configuration being prefixed with Some logs are not included by default.
The individual input configurations for non-default logs are ones that are not prefixed with
|
||
5 | @include /fluent-bit/etc/couchbase/in-audit-log.conf : This is the path to the built-in file that contains the input configuration that processes just the Couchbase audit log (you can view this file on GitHub).
Since this line is included, and @include /fluent-bit/etc/couchbase/in-\*.conf is commented out, only the Couchbase audit log will be processed. |
||
6 | A log forwarding configuration includes one or more output configurations.
The default output configuration forwards the processed logs to the standard output of the logging container.
A log forwarding configuration can include more than one output configuration. This allows for specific logs to be routed to different, or multiple, destinations. For example, all Couchbase logs could be sent to standard output, with the Couchbase audit log also being sent to AWS S3. |
||
7 | couchbase.log.audit : By default, all parsed Couchbase log events are tagged with couchbase.log.<name-of-log> .
Therefore, specifying the match couchbase.log.audit ensures that only Couchbase audit log events are sent to the specified output destination.
(The default output configuration normally uses the wildcard match couchbase.log.* to forward all tagged Couchbase logs to the output.) |
You can copy the example Secret above and create it in your Kubernetes cluster for use with existing and future Couchbase cluster deployments.
However, please review Using Custom Log Forwarding, as there are a few things that you need to be aware of when creating a custom log forwarding configuration Secret.
In particular, you should make sure that any Couchbase cluster deployment that references the Secret in couchbaseclusters.spec.logging.server.configurationName
also has couchbaseclusters.spec.logging.server.manageConfiguration
set to false
, otherwise the Autonomous Operator will overwrite the custom Secret with the default log forwarding configuration.
Updating an Existing Custom Log Forwarding Configuration
Further updates can be made to a custom log forwarding configuration Secret, even after it has been created in Kubernetes and consumed by the logging
sidecar containers in Couchbase deployments.
Changing the Secret will update the volume in the sidecar, and when the configuration change is detected in the couchbaseclusters.spec.logging.server.sidecar.configurationMountPath
directory, the watcher process restarts Fluent Bit internally to pick it up without having to restart the entire Couchbase Server pod.
The ability to restart Fluent Bit internally is a special characteristic of the Couchbase-supplied log processor image. If you’re using a custom sidecar container image, be aware that Fluent Bit, on its own, does not currently support dynamic reload of its configuration.
The Couchbase-supplied log forwarding implementation does not currently support log buffering during restart. Therefore, log events that occur while Fluent Bit is restarting may be lost. |
Configuring Log Redaction
The Couchbase-supplied log processor image provides optional support for log redaction, whereby sensitive data from Couchbase log events are redacted before the events are forwarded to standard console output or other locations. Log redaction isn’t enabled by default, and therefore must be enabled via the log forwarding configuration Secret.
This section describes how to configure the built-in LUA-based log redaction facility. However, there are simpler, more rudimentary methods of performing log redaction that may be more desirable depending on the use-case. Refer to log forwarding tutorial for examples. |
The following example is the same custom log forwarding configuration Secret from the section Creating a Secret That Contains a Custom Log Forwarding Configuration, except that it is configured to additionally perform redaction of the audit log:
apiVersion: v1
kind: Secret
metadata:
name: fluent-bit-config
stringData:
fluent-bit.conf: |
[SERVICE]
flush 1
daemon Off
log_level Warning
parsers_file /fluent-bit/etc/parsers-couchbase.conf
@include /fluent-bit/etc/couchbase/in-audit-log.conf
[FILTER]
Name lua
Match couchbase.log.audit (1)
script /fluent-bit/etc/redaction/redaction.lua (2)
call cb_sub_message (3)
[FILTER]
Name modify
Match couchbase.log.*
Add pod ${HOSTNAME}
Add logshipper couchbase.sidecar.fluentbit
[OUTPUT]
name stdout
match couchbase.log.audit (4)
redaction.salt: salty (5)
1 | couchbase.log.audit : Log events that match this tag will be processed for redaction.
In this example, only events from the audit log will be processed.
To have all logs processed for redaction, you can specify the wildcard couchbase.log.* . |
||
2 | redaction.lua : This is the path to the Couchbase-supplied LUA script file in the default Couchbase-supplied log processor image.
You should only have to modify this if you’re using a custom log processor image that invokes a LUA script from a different location. |
||
3 | cb_sub_message : This is the function that is called within the Couchbase-supplied LUA script file.
You should only have to modify this if you’re using a custom log processor image and a custom LUA script. |
||
4 | couchbase.log.audit : Specifying couchbase.log.audit as the match ensures that only Couchbase audit log events are forwarded to standard console output.
|
||
5 | redaction.salt : You can optionally define a custom salt for the hashing of tagged data.
The salt is specified in the configuration Secret as a separate file/value (in the example, the salt value is salty ).
If a salt is not specified, the Autonomous Operator will default to using the cluster name as the salt. |
After updating the log forwarding configuration with the example above, you’ll notice output similar to the following, showing the redacted strings as hashes:
[0] couchbase.log.audit: [1615320737.445039000, {"filename":"/fluent-bit/test/logs/audit.log","bucket":"default","description":"Cats are <ud>00b335216f27c1e7d35149b5bbfe19d4eb2d6af1</ud> than dogs, and <ud>888f807d45ff6ce47240c7ed4e884a6f9dc7b4fb</ud>","id":20492,"name":"select bucket","peername":"127.0.0.1:56021","real_userid":{"domain":"local","user":"@ns_server"},"sockname":"127.0.0.1:11209","timestamp":"2021-03-09T20:12:17.445039Z"}]
One thing to note about the above configuration is that the log processor is only outputting a redacted version of the audit log. There may be situations where it is desired to have the log processor output both a redacted and an unredacted version of a particular log.
The following is an example of a custom log forwarding configuration Secret that creates an additional stream of the Couchbase audit log that is then redacted and forwarded to a separate output:
apiVersion: v1
kind: Secret
metadata:
name: fluent-bit-config
stringData:
fluent-bit.conf: |
[SERVICE]
flush 1
daemon Off
log_level Warning
parsers_file /fluent-bit/etc/parsers-couchbase.conf
@include /fluent-bit/etc/couchbase/in-*.conf (1)
[FILTER]
Name modify
Match couchbase.log.*
Add pod ${HOSTNAME}
Add logshipper couchbase.sidecar.fluentbit
[FILTER] (2)
Name rewrite_tag
Match couchbase.log.audit
Rule message .* couchbase.log.audit.redact true
[FILTER] (3)
Name lua
Match couchbase.log.*.redact
script /fluent-bit/etc/redaction/redaction.lua
call cb_sub_message
[OUTPUT] (4)
name stdout
match couchbase.log.*
[OUTPUT] (5)
Name s3
Match couchbase.log.audit.redact
bucket my-bucket
region us-west-2
total_file_size 250M
s3_key_format /$TAG[2]/$TAG[0]/%Y/%m/%d/%H/%M/%S
s3_key_format_tag_delimiters .-
redaction.salt: salty
1 | @include /fluent-bit/etc/couchbase/in-*.conf : specifies the default input configuration that processes most Couchbase logs. |
2 | A rewrite_tag filter is defined in order to duplicate the Couchbase audit log events.
The final result is two streams of the same audit log events: one stream with the tag |
3 | A LUA filter is defined in order to perform redaction on one of the streams of audit log events.
|
4 | An output configuration is defined that forwards all unredacted Couchbase log events to the standard output of the logging container.
|
5 | A second output configuration is defined that forwards redacted Couchbase audit log events to the Amazon S3.
|
About Custom Log Processor Images
A different container image may be used in order to implement your own log processor and forwarding configuration.
couchbaseclusters.spec.logging.server.sidecar.image
supports specifying a different image and/or version for log forwarding support.
For example, this field can be used to specify a custom image with self-signed certificates.
It can also be used to specify an image made available by a public cloud provider, such a AWS for Fluent Bit.
It’s important to note that the built-in parsing, redaction, and other configurations provided by the Couchbase-supplied log processor image are lost if a stock log processor image is used.