Release Notes

CBES-311: The connector now includes the co.elastic.logging:log4j2-ecs-layout library as a convenience for users who want to configure Log4j2 to use this layout.

CBES-314: Upgrade Couchbase Java SDK from 3.6.1 to 3.7.1.

CBES-310: Upgrade Couchbase DCP client from 0.50.0 to 0.51.0. Notably:

CBES-308: Upgrade Couchbase Java SDK from 3.5.2 to 3.6.1.

CBES-305: Upgrade Couchbase Java SDK from 3.4.6 to 3.5.2.

CBES-306: Upgrade Couchbase DCP client from 0.46.0 to 0.48.0.

JDCP-239: Improved reliability of initial startup during a Couchbase Server cluster rebalance, or before a newly-created bucket is ready.

JDCP-240: When a partition is not active on any Couchbase Server node, the checkpoint management "catch up" command now waits for the partition to become active somewhere, instead of clearing the partition’s checkpoint.

JDCP-241: Improved handling of an edge case that could cause the connector to stop streaming for a partition. If a DCP socket connection is lost after receiving a snapshot marker for a partition and before receiving the first item in the snapshot, the connector now recovers instead of failing to resume streaming for that partition.

CBES-297: When resuming from an old checkpoint (where the connector’s sequence number is lower than Couchbase Server’s purge sequence number), the connector no longer rolls back to zero. This behavior requires Couchbase Server 7.2 or later.

CBES-300: Upgrade Couchbase Java SDK from 3.4.4 to 3.4.6.

CBES-301: Upgrade Couchbase DCP client from 0.45.0 to 0.46.0.

CBES-293: The connector now honors the io.enableDnsSrv client setting. Previously, it would always attempt DNS SRV resolution on an eligible Couchbase Server hostname, regardless of whether DNS SRV was disabled via the client setting. This typically has no functional impact, but the old behavior was generating noise in the logs.

CBES-286: Add experimental support for Elastic Cloud.

CBES-285: Add experimental support for OpenSearch. No special configuration is required; the connector automatically detects you’re using OpenSearch instead of Elasticsearch.

CBES-243: Add experimental support for Amazon OpenSearch Service.

CBES-287: When connecting to Elasticsearch/OpenSearch with TLS, the well-known Certificate Authority (CA) certificates from the JVM’s cacerts trust store are now trusted by default, unless you specify different CA certificates to trust.

CBES-288: Add Consul 1.15.1 to the compatibility matrix as a supported version.

CBES-290: Upgrade Couchbase Java SDK from 3.4.3 to 3.4.4.

CBES-291: Upgrade Couchbase DCP client from 0.44.0 to 0.45.0.

CBES-283: Upgrade Couchbase Java SDK from 3.4.1 to 3.4.3.

CBES-284: Upgrade Couchbase DCP client from 0.43.0 to 0.44.0.

CBES-278: Elasticsearch 8.5 and Consul 1.14.3 are now supported.

CBES-276: Upgrade Couchbase DCP client from 0.42.0 to 0.43.0.

CBES-277: Upgrade Couchbase Java SDK from 3.3.4 to 3.4.1.

CBES-281: A regression in version 4.4.0 caused the connector to omit null-valued document fields when writing to Elasticsearch. Null-valued fields are now replicated correctly again.

JDCP-232: Fixed a race condition that sometimes caused the connector to fail on startup with the message: java.lang.IllegalStateException: Tried to add duplicate channel.

JDCP-224: Use SASL mechanism PLAIN when authenticating with Couchbase on a secure connection. PLAIN is the fastest mechanism, and the only one that works with LDAP users.

JDCP-217: Support Couchbase clusters that advertise only TLS ports.

CBES-267: Include date and time zone in log message timestamps.

CBES-268: Increase the DCP connection handshake timeout from 3 seconds to 12 seconds, allowing more time to connect to a remote/overloaded cluster.

CBES-275: Upgrade Couchbase DCP client from 0.41.0 to 0.42.0.

Upgraded commons-text from 1.9 to 1.10.0. Please see our official response to CVE-2022-42889 (also known as Text4Shell). In brief, no version of the Elasticsearch connector is vulnerable.

CBES-240: Autonomous Operations Mode When the connector shuts down gracefully in response to an interrupt signal, it now deregisters its Consul service definition before exiting. For ungraceful shutdowns, Consul automatically deregisters a service definition that remains in "critical" state for 7 days.

--consul <path/to/consul.toml>

CBES-237: Autonomous Operations Mode HashiCorp Consul 1.13, 1.12, and 1.11 are now supported.

CBES-144: Autonomous Operations Mode The connector now recovers from transient Consul errors that sometimes occur during leader election. When Consul returns HTTP status code 500 or 503, the connector now retries the request instead of immediately terminating.

CBES-262: Elasticsearch 8.4 is now supported.

CBES-256: OpenJDK 17 is now supported, and is used by the Docker image. The Dockerfile now refers to the base image by a stable minor version tag (instead of a specific patch version), which should make it easier for us to update the connector image when the base image receives security updates.

CBES-263: Upgraded Couchbase Java SDK from 3.3.2 to 3.3.4.

CBES-266: Upgraded Couchbase DCP client from 0.40.0 to 0.41.0.

The minimum required version of Elasticsearch is now 7.14.0.

OpenSearch and Amazon OpenSearch Service are not supported. If you’re using a previous version of the connector with OpenSearch, please remain on that version for now. We’re investigating restoring support in a future version. UPDATE: Version 4.4.5 (2023-03-21) restores support for OpenSearch and Amazon OpenSearch Service as an experimental feature.

CBES-254: Added support for Elasticsearch 8.

CBES-258: When using secure connections, the connector can now read the Couchbase and Elasticsearch CA certificates from separate PEM files instead of a single Java keystore. The [couchbase] and [elasticsearch] config sections each have a new pathToCaCertificate property that points to the respective PEM file.

CBES-257: When connecting to Couchbase Capella, it’s no longer necessary to configure the CA certificate. Just make sure not to configure the deprecated [truststore] config section, and make sure to leave the pathToCaCertificate property in the [couchbase] config section blank.

CBES-259: Upgraded Couchbase Java SDK from 3.3.1 to 3.3.2.

The [truststore] config section is deprecated, and will be removed in a future version. Instead, please use the new pathToCaCertificate config properties added by CBES-258.

The typeName property in the [elasticsearch.typeDefaults], [[elasticsearch.type]], and [elasticsearch.rejectionLog] config sections is deprecated, and will be removed in a future version. Specifying this property has no effect, since the concept of document types was removed in Elasticsearch 7.

CBES-279: Upgrade the Docker base image to the latest version of ubi8/openjdk-11-runtime. The Dockerfile now uses a floating tag for the base image. This makes it easier to refresh the connector image when the base image is updated.

Upgraded commons-text from 1.9 to 1.10.0. Please see our official response to CVE-2022-42889 (also known as Text4Shell). In brief, no version of the Elasticsearch connector is vulnerable.

CBES-255: Upgrade Couchbase Java SDK from 3.3.0 to 3.3.1.

CBES-253: Upgrade the Docker base image to the latest version of ubi8/openjdk-11-runtime.

CBES-249: Upgrade Couchbase Java SDK from 3.2.6 to 3.3.0.

CBES-252: Upgrade ubi8/openjdk-11-runtime base image from 1.11-2.1648459559 to 1.12-1.1651233103.

CBES-247: The new /info HTTP endpoint reports the connector version and membership in a machine-readable format.

CBES-248: Upgrade Couchbase Java SDK from 3.2.4 to 3.2.6.

CBES-238: Resolved an incompatibility with Consul 1.10.

CBES-241: Upgraded Log4j from 2.17.0 to 2.17.1.

CBES-245: Upgraded Couchbase DCP client from 0.38.0 to 0.39.0:

CBES-242: Upgraded other dependencies to the latest versions.

CBES-241: Upgrade Log4j from 2.17.0 to 2.17.1.

CBES-232: Upgraded Log4j from 2.15.0 to 2.17.0. This prevents vulnerability scanners from flagging Log4j 2.15.0 as a potential security risk.

CBES-200: Added basic Kubernetes integration. See the new documentation page, Deploying in Kubernetes.

CBES-226: Added a LICENSE file to the distribution archive (Apache License Version 2.0, same as before).

CBES-234: Upgraded DCP client from 0.37.0 to 0.38.0 for better error messages if the Couchbase user does not have the required permissions.

CBES-235: Upgraded Couchbase Java client from 3.2.0 to 3.2.4 to pick up the latest dependency versions.

CBES-232: Upgraded Log4j from 2.15.0 to 2.17.0. This prevents vulnerability scanners from flagging Log4j 2.15.0 as a potential security risk.

CBES-230: Upgrade Log4j 2 from 2.14.1 to 2.15.0 to address CVE-2021-44228.

CBES-230: Upgrade Log4j 2 from 2.14.1 to 2.15.0 to address CVE-2021-44228.

CBES-224: The default flow control buffer size is now 16 MB instead of 128 MB. This makes it less likely the connector will run out of memory under heavy load with the default heap size. The documentation now describes how the DCP flowControlBuffer config property affects the connector’s memory requirements.

CBES-223: Upgraded DCP client from 0.36.0 to 0.37.0. This upgrade adds a workaround for MB-48655 so all versions of Couchbase now correctly log DCP diagnostic messages from the connector.

CBES-222: The connector no longer logs the Couchbase Server version. Prior to this change the log message always had a placeholder version of 9999.0.0 which was misleading.

CBES-221: Upgraded the DCP client from 0.35.0 to 0.36.0 for compatibility with Couchbase Server 7.0.2.

The config options for working with Couchbase 7 Scopes and Collections are now part of the "committed" API.

All other "uncommitted" config options are promoted to "committed" status as well.

CBES-215: The connector now requires Java 11 (or later).

CBES-212: Elasticsearch 5 is no longer supported. It still works (at least for now), but we’re no longer testing it. Please upgrade to a more recent version of Elasticsearch.

The deprecated cbes.backfill* metrics have been removed. As a replacement, please use the cbes.backlog metric which gives ongoing insight into the connector’s performance.

CBES-213: When logging permanent indexing failures, the connector now sanitizes the contents of the failure message to ensure sensitive information is redacted.

CBES-209: Upgraded the Couchbase Java SDK from 3.1.3 to 3.2.0.

CBES-217: Upgraded the DCP client from 0.34.0 to 0.35.0.

CBES-211: Upgraded various dependencies to the latest versions.

CBES-206: Integer and boolean config properties can now be specified as strings. For example: 7 and '7' are now both valid for an integer property, and true and 'true' are now both valid for a boolean property. This lets you use environment variable placeholders for integer and boolean properties without invalidating the TOML syntax — just enclose them in quotes, like: '${SOME_INTEGER}'. The connector resolves placeholders before parsing the TOML, so it doesn’t care about the invalid syntax; this change just makes it easier for other tools to parse the connector config file.

CBES-202: The connector now waits until it has connected to both Couchbase and Elasticsearch before starting the HTTP server for the /metrics endpoint. This makes it more useful as a "readiness" probe.

CBES-204: Added a new [couchbase.env] config section for tuning Couchbase Java SDK settings.

CBES-203: Upgraded the Couchbase Java SDK from 3.1.2 to 3.1.3.

CBES-205: Upgraded the DCP client from 0.33.0 to 0.34.0.

CBES-192: When the new logDocumentLifecycle config property is set to true in the logging section, the connector writes detailed log entries as each document flows through the connector.

CBES-198: Log redaction is now configurable via the new redactionLevel config property in the logging section.

CBES-199: Upgraded the DCP client from 0.32.0 to 0.33.0.

CBES-193: If two documents in different collections had the same ID, and both were processed by the connector in the same batch, one would be incorrectly flagged as a duplicate and dropped, even if it had a different destination index. With this fix, documents are never considered duplicates if they have different destination indexes. This issue only affected users experimenting with the Scopes & Collections feature.

CBES-183: When secure connections are enabled, it is now possible to authenticate with Couchbase and/or Elasticsearch using an X.509 certificate instead of a username & password. See the Client Certificates documentation for details. (This feature is added as "uncommitted", meaning it may change without notice.)

CBES-187: Errors during an early phase of connector startup were written to the console instead of being logged. Now these errors will appear in the log as well.

CBES-189: Upgraded the Couchbase Java SDK from 3.1.0 to 3.1.2.

CBES-188: Upgraded the DCP client from 0.31.0 to 0.32.0.

JDCP-188: Previously, TLS hostname verification was done only for the Couchbase Java client connection; now the DCP client connection is verified as well.

JDCP-183: If an invalid stream offset is detected, the connector will now fail fast instead of potentially corrupting the saved checkpoint.

JDCP-184: Resolved an issue that could cause a flow control deadlock when streaming from a subset of a bucket’s collections or scopes.

CBES-185: The connector no longer fails to start when using Consul version 1.8.4 and later. The list of tested and supported Consul versions now includes Consul 1.9.1.

CBES-184: The connector now exposes Prometheus metrics at /metrics/prometheus. Prometheus metrics are "Uncommitted API" and subject to change between releases without notice. The original Dropwizard JSON metrics are still available at /metrics, and can now be accessed at the alternative path /metrics/dropwizard.

CBES-181: Added a new config property, hostnameVerification (default: true). Couchbase TLS certificate hostname verification was silently enabled by default in version 4.2.3. If this caused problems for your deployment, and you are unable to issue certificates matching the Couchbase server nodes, you can now disable hostname verification by setting this new config property to false.

CBES-182: Upgraded the Couchbase Java SDK from 3.0.9 to 3.1.0.

CBES-121: Added new cbes.backlog metric which estimates the number of Couchbase document changes yet to be processed. This is a general indication of how well the connector is keeping up with changes in Couchbase. Note that the count only includes changes in the Couchbase partitions handled by the connector instance reporting the metric. Unlike cbes.backfill, this value is dynamic; it goes up when changes happen in Couchbase, and goes down as the changes are processed by the connector.

CBES-178: Upgraded the DCP client from 0.30.0 to 0.31.0.

CBES-179: Fixed an issue that prevented the connector from establishing a secure connection to Elasticsearch if a custom port was specified.

CBES-170: Bootstrap performance is improved when specifying custom ports.

CBES-175: Upgraded the Couchbase Java SDK from 3.0.6 to 3.0.9. The connector no longer logs spurious warnings about being unable to fetch collections manifests.

CBES-177: Upgraded the DCP client from 0.28.0 to 0.30.0.

CBES-173: Fixed a regression in version 4.2.3 that broke alternate address resolution. The connector now handles DNS SRV and alternate addresses correctly, and can connect to Couchbase Cloud or other network environments that use alternate addresses.

CBES-172: Removed duplicate command line scripts from the ZIP archive. You can now unzip the archive without being prompted about overwriting the duplicate files.

CBES-163: Type definitions now have a matchOnQualifiedKey property that lets a rule match against the qualified document name, which includes the scope and collection. This enables type definition rules that write to an Elasticsearch index whose name matches the Couchbase collection name.

CBES-164: The [couchbase] config section now has optional scope and collection properties that limit the replication to a single scope or to a set of collections.

CBES-165: The [couchbase] config section now has optional metadataCollection property that controls which collection is used to store metadata like replication checkpoints.

CBES-170: If you specify a custom port for a Couchbase host, it can take a long while to connect, and the connector will log lots of warnings about being unable to connect to the KV service.

CBES-155 Couchbase client ignores 'couchbase.network' config setting

JDCP-163 Force DNS lookups on reconnect

JDCP-156 Enable Netty native transports by default

JDCP-82 Decompress with org.iq80.snappy instead of Netty

CBES-158 Upgrade Couchbase SDK from 2.7.11 to 2.7.15

CBES-154 Upgrade DCP client from 0.25.0 to 0.28.0

JDCP-146 Upgrade Netty from 4.0.56 to 4.1.48

CBES-147 Stuck shutdown hook can prevent/delay JVM termination

CBES-149 Bulk request timeout is capped at 30 seconds

CBES-143 Report DCP metrics

CBES-148 Report CPU usage metrics

CBES-150 Upgrade Couchbase client to 2.7.11 and DCP client to 0.25.0

CBES-129 Support direct connections to Amazon Elasticsearch Service

CBES-140 Support ACL Token Authentication when communicating with Consul

CBES-141 Extend support coverage to Elasticsearch 7.4

CBES-65 Autonomous Operations Mode with Consul

CBES-135 Expose multi-network config options

CBES-138 Upgrade to Couchbase client 2.7.9 and DCP client 0.24.0

CBES-122 Add support for OpenJDK

CBES-123 Support Elasticsearch 6.7 & 7.0 docker images

CBES-125 Suppress "types removal" warnings from Elasticsearch 7.0

CBES-128 Couchbase Server fails to rebalance if Elasticsearch connector is running

CBES-90 Ability to use environment variables inside config

CBES-107 Misleading error message when can’t connect to Elasticsearch

CBES-110 Need document routing to support join

CBES-114 Allow saving checkpoints in a different bucket

CBES-117 Connector exits on values that fail to parse

The connector is now a standalone process instead of an Elasticsearch plug-in.

Compatible with Elasticsearch versions 5 and 6.

Support for secure connections to Couchbase and Elasticsearch.

Tools for managing replication checkpoints.

A "rejection log" for documents Elasticsearch permanently refuses to index.

Configurable document structure (omit metadata if you don’t need it).

The connector now listens for document changes using the high performance Couchbase Database Change Protocol (DCP).