Release Notes

      +
      Release notes, installation instructions, and download archive for the Couchbase Elasticsearch Connector.

      Installation

      Scroll down to the version you want, then click the "Download" link to get the full connector distribution. Refer to Getting Started for detailed installation instructions.

      Version 4.4.1 (2022-09-20)

      Adds support for recent versions of HashiCorp Consul, and improves the lifecycle of the connector’s Consul service definitions.

      Elasticsearch 8.4 joins the list of supported versions.

      Behavioral Changes

      • CBES-240: Autonomous Operations Mode When the connector shuts down gracefully in response to an interrupt signal, it now deregisters its Consul service definition before exiting. For ungraceful shutdowns, Consul automatically deregisters a service definition that remains in "critical" state for 7 days.

        You can customize these behaviors in the connector’s Consul-specific configuration, specified with the command-line option:
        --consul <path/to/consul.toml>

      Enhancements

      • CBES-237: Autonomous Operations Mode HashiCorp Consul 1.13, 1.12, and 1.11 are now supported.

      • CBES-144: Autonomous Operations Mode The connector now recovers from transient Consul errors that sometimes occur during leader election. When Consul returns HTTP status code 500 or 503, the connector now retries the request instead of immediately terminating.

      • CBES-262: Elasticsearch 8.4 is now supported.

      • CBES-256: OpenJDK 17 is now supported, and is used by the Docker image. The Dockerfile now refers to the base image by a stable minor version tag (instead of a specific patch version), which should make it easier for us to update the connector image when the base image receives security updates.

      • CBES-263: Upgraded Couchbase Java SDK from 3.3.2 to 3.3.4.

      • CBES-266: Upgraded Couchbase DCP client from 0.40.0 to 0.41.0.

      Version 4.4.0 (2022-08-01)

      The connector now supports Elasticsearch 8, and it’s easier to configure the trusted Certificate Authority (CA) certificates.

      Breaking Changes

      • The minimum required version of Elasticsearch is now 7.14.0.

      • Amazon OpenSearch is not supported. If you’re using a previous version of the connector with OpenSearch, please remain on that version for now. We’re investigating restoring support in a future version.

      Enhancements

      • CBES-254: Added support for Elasticsearch 8.

      • CBES-258: When using secure connections, the connector can now read the Couchbase and Elasticsearch CA certificates from separate PEM files instead of a single Java keystore. The [couchbase] and [elasticsearch] config sections each have a new pathToCaCertificate property that points to the respective PEM file.

      • CBES-257: When connecting to Couchbase Capella, it’s no longer necessary to configure the CA certificate. Just make sure not to configure the deprecated [truststore] config section, and make sure to leave the pathToCaCertificate property in the [couchbase] config section blank.

      • CBES-259: Upgraded Couchbase Java SDK from 3.3.1 to 3.3.2.

      Deprecations

      • The [truststore] config section is deprecated, and will be removed in a future version. Instead, please use the new pathToCaCertificate config properties added by CBES-258.

      • The typeName property in the [elasticsearch.typeDefaults], [[elasticsearch.type]], and [elasticsearch.rejectionLog] config sections is deprecated, and will be removed in a future version. Specifying this property has no effect, since the concept of document types was removed in Elasticsearch 7.

      Version 4.3.8 (2022-06-21)

      This release bumps various dependency versions. There are no new features or bug fixes.

      Enhancements

      • CBES-255: Upgrade Couchbase Java SDK from 3.3.0 to 3.3.1.

      • CBES-253: Upgrade the Docker base image to the latest version of ubi8/openjdk-11-runtime.

      Version 4.3.7 (2022-05-17)

      This release bumps various dependency versions. There are no new features or bug fixes.

      Enhancements

      • CBES-249: Upgrade Couchbase Java SDK from 3.2.6 to 3.3.0.

      • CBES-252: Upgrade ubi8/openjdk-11-runtime base image from 1.11-2.1648459559 to 1.12-1.1651233103.

      Version 4.3.6 (2022-04-19)

      This release adds a new /info HTTP endpoint that displays information about the connector.

      Enhancements

      • CBES-247: The new /info HTTP endpoint reports the connector version and membership in a machine-readable format.

      • CBES-248: Upgrade Couchbase Java SDK from 3.2.4 to 3.2.6.

      Version 4.3.5 (2022-01-18)

      This is a maintenance released focused on upgrading dependencies to the latest versions.

      Enhancements

      • CBES-238: Resolved an incompatibility with Consul 1.10.

      • CBES-241: Upgraded Log4j from 2.17.0 to 2.17.1.

      Regarding CVE-2021-44832: The connector’s out-of-the-box logging configuration does not use Log4j’s JDBC appender. You may still wish to upgrade to avoid false positives from vulnerability scanners.
      • CBES-245: Upgraded Couchbase DCP client from 0.38.0 to 0.39.0:

        • JDCP-210 Authentication no longer fails when credentials have non-ASCII characters and the system default encoding is not UTF-8.

      • CBES-242: Upgraded other dependencies to the latest versions.

      Version 4.2.15 (2022-01-18)

      This release upgrades Log4j again (sigh).

      Regarding CVE-2021-44832: The connector’s out-of-the-box logging configuration does not use Log4j’s JDBC appender. You may still wish to upgrade to avoid false positives from vulnerability scanners.

      Enhancements

      • CBES-241: Upgrade Log4j from 2.17.0 to 2.17.1.

      Version 4.3.4 (2021-12-21)

      This release makes it easier to deploy the connector in Kubernetes, and upgrades Log4j from 2.15.0 to 2.17.0.

      Enhancements

      • CBES-232: Upgraded Log4j from 2.15.0 to 2.17.0. This prevents vulnerability scanners from flagging Log4j 2.15.0 as a potential security risk.

      All versions of the connector are immune to CVE-2021-45046 and CVE-2021-45105 because the connector does not use the Thread Context Map / Mapped Diagnostic Context (MDC) feature of Log4j.
      • CBES-200: Added basic Kubernetes integration. See the new documentation page, Deploying in Kubernetes.

      • CBES-226: Added a LICENSE file to the distribution archive (Apache License Version 2.0, same as before).

      • CBES-234: Upgraded DCP client from 0.37.0 to 0.38.0 for better error messages if the Couchbase user does not have the required permissions.

      • CBES-235: Upgraded Couchbase Java client from 3.2.0 to 3.2.4 to pick up the latest dependency versions.

      Version 4.2.14 (2021-12-21)

      This release upgrades Log4j from 2.15.0 to 2.17.0.

      Enhancements

      • CBES-232: Upgraded Log4j from 2.15.0 to 2.17.0. This prevents vulnerability scanners from flagging Log4j 2.15.0 as a potential security risk.

      All versions of the connector are immune to CVE-2021-45046 and CVE-2021-45105 because the connector does not use the Thread Context Map / Mapped Diagnostic Context (MDC) feature of Log4j.

      Version 4.3.3 (2021-12-10)

      This release fixes a high severity vulnerability related to Log4j 2. All users should upgrade the connector to 4.3.3 (or 4.2.13) as soon as possible.

      Bug Fixes

      • CBES-230: Upgrade Log4j 2 from 2.14.1 to 2.15.0 to address CVE-2021-44228.

      Version 4.2.13 (2021-12-10)

      This release fixes a high severity vulnerability related to Log4j 2. All users should upgrade the connector to 4.2.13 (or 4.3.3 and later) as soon as possible.

      Bug Fixes

      • CBES-230: Upgrade Log4j 2 from 2.14.1 to 2.15.0 to address CVE-2021-44228.

      Version 4.3.2 (2021-10-19)

      This version reduces the default flow control buffer size to a more reasonable value and improves DCP diagnostics.

      Enhancements

      • CBES-224: The default flow control buffer size is now 16 MB instead of 128 MB. This makes it less likely the connector will run out of memory under heavy load with the default heap size. The documentation now describes how the DCP flowControlBuffer config property affects the connector’s memory requirements.

      • CBES-223: Upgraded DCP client from 0.36.0 to 0.37.0. This upgrade adds a workaround for MB-48655 so all versions of Couchbase now correctly log DCP diagnostic messages from the connector.

      • CBES-222: The connector no longer logs the Couchbase Server version. Prior to this change the log message always had a placeholder version of 9999.0.0 which was misleading.

      Version 4.3.1 (2021-08-20)

      This release restores compatibility with Couchbase Server 7.0.2.

      If you are currently using a connector version between 4.2.2 and 4.3.0 inclusive, please upgrade to 4.3.1 or later before upgrading Couchbase Server beyond 7.0.1.

      Enhancements

      • CBES-221: Upgraded the DCP client from 0.35.0 to 0.36.0 for compatibility with Couchbase Server 7.0.2.

      Version 4.3.0 (2021-07-20)

      This release stabilizes the configuration options for working with Couchbase 7 Scopes and Collections. All previously "uncommitted" options are promoted to "committed" status.

      Enhancements

      • The config options for working with Couchbase 7 Scopes and Collections are now part of the "committed" API.

      • All other "uncommitted" config options are promoted to "committed" status as well.

      Breaking Changes

      • CBES-215: The connector now requires Java 11 (or later).

      • CBES-212: Elasticsearch 5 is no longer supported. It still works (at least for now), but we’re no longer testing it. Please upgrade to a more recent version of Elasticsearch.

      • The deprecated cbes.backfill* metrics have been removed. As a replacement, please use the cbes.backlog metric which gives ongoing insight into the connector’s performance.

      Version 4.2.12 (2021-07-20)

      This release upgrades various dependencies to the latest versions and improves how permanent indexing failures are logged.

      Enhancements

      • CBES-213: When logging permanent indexing failures, the connector now sanitizes the contents of the failure message to ensure sensitive information is redacted.

      • CBES-209: Upgraded the Couchbase Java SDK from 3.1.3 to 3.2.0.

      • CBES-217: Upgraded the DCP client from 0.34.0 to 0.35.0.

      • CBES-211: Upgraded various dependencies to the latest versions.

      Version 4.2.11 (2021-05-18)

      This release makes it easier for custom tools to parse config files that have environment variable placeholders.

      Enhancements

      • CBES-206: Integer and boolean config properties can now be specified as strings. For example: 7 and '7' are now both valid for an integer property, and true and 'true' are now both valid for a boolean property. This lets you use environment variable placeholders for integer and boolean properties without invalidating the TOML syntax — just enclose them in quotes, like: '${SOME_INTEGER}'. The connector resolves placeholders before parsing the TOML, so it doesn’t care about the invalid syntax; this change just makes it easier for other tools to parse the connector config file.

      Version 4.2.10 (2021-04-20)

      This release brings minor improvements to monitoring and configurability.

      Enhancements

      • CBES-202: The connector now waits until it has connected to both Couchbase and Elasticsearch before starting the HTTP server for the /metrics endpoint. This makes it more useful as a "readiness" probe.

      • CBES-204: Added a new [couchbase.env] config section for tuning Couchbase Java SDK settings.

      • CBES-203: Upgraded the Couchbase Java SDK from 3.1.2 to 3.1.3.

      • CBES-205: Upgraded the DCP client from 0.33.0 to 0.34.0.

      Version 4.2.9 (2021-03-16)

      This release improves diagnostic logging and fixes an issue with scopes & collections.

      Enhancements

      • CBES-192: When the new logDocumentLifecycle config property is set to true in the logging section, the connector writes detailed log entries as each document flows through the connector.

      • CBES-198: Log redaction is now configurable via the new redactionLevel config property in the logging section.

      • CBES-199: Upgraded the DCP client from 0.32.0 to 0.33.0.

      Bug Fixes

      • CBES-193: If two documents in different collections had the same ID, and both were processed by the connector in the same batch, one would be incorrectly flagged as a duplicate and dropped, even if it had a different destination index. With this fix, documents are never considered duplicates if they have different destination indexes. This issue only affected users experimenting with the Scopes & Collections feature.

      Version 4.2.8 (2021-02-16)

      This release adds uncommitted support for client certificate authentication (mTLS), adds hostname verification for secure DCP connections, and improves the stability of the connector.

      Enhancements

      • CBES-183: When secure connections are enabled, it is now possible to authenticate with Couchbase and/or Elasticsearch using an X.509 certificate instead of a username & password. See the Client Certificates documentation for details. (This feature is added as "uncommitted", meaning it may change without notice.)

      • CBES-187: Errors during an early phase of connector startup were written to the console instead of being logged. Now these errors will appear in the log as well.

      • CBES-189: Upgraded the Couchbase Java SDK from 3.1.0 to 3.1.2.

      • CBES-188: Upgraded the DCP client from 0.31.0 to 0.32.0.

      • JDCP-188: Previously, TLS hostname verification was done only for the Couchbase Java client connection; now the DCP client connection is verified as well.

      Bug Fixes

      • JDCP-183: If an invalid stream offset is detected, the connector will now fail fast instead of potentially corrupting the saved checkpoint.

      • JDCP-184: Resolved an issue that could cause a flow control deadlock when streaming from a subset of a bucket’s collections or scopes.

      Version 4.2.7 (2021-01-19)

      For those running the connector in Autonomous Operations mode, this release improves compatibility with recent Consul versions.

      Enhancements

      • CBES-185: The connector no longer fails to start when using Consul version 1.8.4 and later. The list of tested and supported Consul versions now includes Consul 1.9.1.

      Version 4.2.6 (2020-12-15)

      This release brings improvements to monitoring and adds an option for disabling TLS certificate hostname verification.

      Enhancements

      • CBES-184: The connector now exposes Prometheus metrics at /metrics/prometheus. Prometheus metrics are "Uncommitted API" and subject to change between releases without notice. The original Dropwizard JSON metrics are still available at /metrics, and can now be accessed at the alternative path /metrics/dropwizard.

      • CBES-181: Added a new config property, hostnameVerification (default: true). Couchbase TLS certificate hostname verification was silently enabled by default in version 4.2.3. If this caused problems for your deployment, and you are unable to issue certificates matching the Couchbase server nodes, you can now disable hostname verification by setting this new config property to false.

      • CBES-182: Upgraded the Couchbase Java SDK from 3.0.9 to 3.1.0.

      Version 4.2.5 (2020-11-17)

      This release fixes an issue with configuring secure connections to Elasticsearch using custom ports. It also adds a new metric for monitoring the replication backlog.

      Enhancements

      • CBES-121: Added new cbes.backlog metric which estimates the number of Couchbase document changes yet to be processed. This is a general indication of how well the connector is keeping up with changes in Couchbase. Note that the count only includes changes in the Couchbase partitions handled by the connector instance reporting the metric. Unlike cbes.backfill, this value is dynamic; it goes up when changes happen in Couchbase, and goes down as the changes are processed by the connector.

      • CBES-178: Upgraded the DCP client from 0.30.0 to 0.31.0.

      Bug Fixes

      • CBES-179: Fixed an issue that prevented the connector from establishing a secure connection to Elasticsearch if a custom port was specified.

      Version 4.2.4 (2020-10-20)

      This release improves compatibility with Couchbase Cloud, and fixes a few minor issues.

      Enhancements

      • CBES-170: Bootstrap performance is improved when specifying custom ports.

      • CBES-175: Upgraded the Couchbase Java SDK from 3.0.6 to 3.0.9. The connector no longer logs spurious warnings about being unable to fetch collections manifests.

      • CBES-177: Upgraded the DCP client from 0.28.0 to 0.30.0.

      Bug Fixes

      • CBES-173: Fixed a regression in version 4.2.3 that broke alternate address resolution. The connector now handles DNS SRV and alternate addresses correctly, and can connect to Couchbase Cloud or other network environments that use alternate addresses.

      • CBES-172: Removed duplicate command line scripts from the ZIP archive. You can now unzip the archive without being prompted about overwriting the duplicate files.

      Version 4.2.3 (2020-07-21)

      This release adds "uncommitted" support for Collections and Scopes, a new feature planned for Couchbase Server 7.0.

      Enhancements

      • CBES-163: Type definitions now have a matchOnQualifiedKey property that lets a rule match against the qualified document name, which includes the scope and collection. This enables type definition rules that write to an Elasticsearch index whose name matches the Couchbase collection name.

      • CBES-164: The [couchbase] config section now has optional scope and collection properties that limit the replication to a single scope or to a set of collections.

      • CBES-165: The [couchbase] config section now has optional metadataCollection property that controls which collection is used to store metadata like replication checkpoints.

      Known Issues

      • CBES-170: If you specify a custom port for a Couchbase host, it can take a long while to connect, and the connector will log lots of warnings about being unable to connect to the KV service.

      The workaround is to bootstrap using a KV port (default 11210) instead of a manager port (default 8091), and to explicitly tag the port as belonging to the KV service, like this:

      hosts = ['example.com:12345=kv']

      Version 4.2.2 (2020-05-19)

      The connector now behaves better in environments where DNS entries are highly dynamic. Instead of caching resolved hostnames, it now resolves hostnames prior to every connection attempt.

      This release also improves decompression performance, activates Netty native transports, and fixes a bug that prevented the 'couchbase.network' config setting from being honored.

      The compatibility matrix is updated to add support for Elasticsearch 7.6 and 7.7. Elasticsearch versions prior to 6.6 are dropped due to end of life, with the exception of 5.6.16.

      Bug Fixes

      • CBES-155 Couchbase client ignores 'couchbase.network' config setting

      Enhancements

      • JDCP-163 Force DNS lookups on reconnect

      • JDCP-156 Enable Netty native transports by default

      • JDCP-82 Decompress with org.iq80.snappy instead of Netty

      • CBES-158 Upgrade Couchbase SDK from 2.7.11 to 2.7.15

      • CBES-154 Upgrade DCP client from 0.25.0 to 0.28.0

      • JDCP-146 Upgrade Netty from 4.0.56 to 4.1.48

      Version 4.2.1 (2020-01-21)

      This maintenance release addresses an issue with shutdown hooks that could prevent the connector from terminating in some circumstances.

      Also fixed in this release, bulk request timeouts longer than 30 seconds are now honored instead of being reduced to 30 seconds.

      Metrics from the Couchbase DCP client are now included in the metrics report, along with gauges for CPU load.

      Bug Fixes

      • CBES-147 Stuck shutdown hook can prevent/delay JVM termination

      • CBES-149 Bulk request timeout is capped at 30 seconds

      Enhancements

      • CBES-143 Report DCP metrics

      • CBES-148 Report CPU usage metrics

      • CBES-150 Upgrade Couchbase client to 2.7.11 and DCP client to 0.25.0

      Version 4.2.0 (2019-10-15)

      Hot on the heels of 4.1, we’re releasing 4.2 with support for connecting directly to an Amazon Elasticsearch Service instance. There’s a new [elasticsearch.aws] config section for specifying the AWS region of the service. Amazon credentials are obtained from the Default Credential Provider Chain.

      Also new in 4.2, the cbes-consul command now takes an optional --consul-config argument which points to a separate config file where you can specify a Consul ACL token.

      On the version compatibility front, we’ve added support for Elasticsearch 7.4 and removed support for Elasticsearch 5.4 (which reached EOL on 2018-11-04).

      Enhancements

      • CBES-129 Support direct connections to Amazon Elasticsearch Service

      • CBES-140 Support ACL Token Authentication when communicating with Consul

      • CBES-141 Extend support coverage to Elasticsearch 7.4

      Version 4.1.0 (2019-09-05)

      We are excited to unveil the new Autonomous Operations (AO) mode with major improvements to the availability and manageability of the connector. When the connector is deployed in AO mode, worker processes use your HashiCorp Consul cluster to communicate with each other and automatically distribute the replication workload. You can add or remove worker processes at any time without having to manually stop and reconfigure all of the workers. Any worker that fails a health check is automatically removed, and its workload is redistributed among remaining workers.

      The new cbes-consul command line tool is used to start a worker in AO mode. It also provides streamlined checkpoint management and the ability to reconfigure or pause/resume all of the workers in an AO group at once.

      Also new in this release is support for multi-network configuration. This feature allows the connector to talk to Couchbase Server nodes that have been configured to advertise alternate network addresses for connecting to the node from outside a container/cloud networking environment. The new network property in the [couchbase] section of the configuration gives you control over network selection (although the default value of auto is appropriate for most cases).

      Finally, the range of supported Elasticsearch versions is extended to include 7.1, 7.2. and 7.3.

      Enhancements

      • CBES-65 Autonomous Operations Mode with Consul

      • CBES-135 Expose multi-network config options

      • CBES-138 Upgrade to Couchbase client 2.7.9 and DCP client 0.24.0

      Version 4.0.2 (2019-05-21)

      This maintenance release fixes a bug that prevented some versions of Couchbase Server from rebalancing when the connector is running.

      It also adds compatibility with the official Docker images for Elasticsearch 6.7.x and 7.0.x, and is the first version tested against OpenJDK 8 and OpenJDK 11.

      Enhancements

      • CBES-122 Add support for OpenJDK

      • CBES-123 Support Elasticsearch 6.7 & 7.0 docker images

      • CBES-125 Suppress "types removal" warnings from Elasticsearch 7.0

      Bug Fixes

      • CBES-128 Couchbase Server fails to rebalance if Elasticsearch connector is running

      Version 4.0.1 (2019-04-15)

      This maintenance release improves the stability of the connector and adds new configuration options.

      Enhancements

      • CBES-90 Ability to use environment variables inside config

      • CBES-107 Misleading error message when can’t connect to Elasticsearch

      • CBES-110 Need document routing to support join

      • CBES-114 Allow saving checkpoints in a different bucket

      Bug Fixes

      • CBES-117 Connector exits on values that fail to parse

      Version 4.0.0 (2018-10-12)

      New in this version

      • The connector is now a standalone process instead of an Elasticsearch plug-in.

      • Compatible with Elasticsearch versions 5 and 6.

      • Support for secure connections to Couchbase and Elasticsearch.

      • Tools for managing replication checkpoints.

      • A "rejection log" for documents Elasticsearch permanently refuses to index.

      • Configurable document structure (omit metadata if you don’t need it).

      • The connector now listens for document changes using the high performance Couchbase Database Change Protocol (DCP).

      Things to be aware of

      This is a major version upgrade. Because the plug-in and the standalone connector are so different, there is no online upgrade process. See the Migration documentation for details.
      • Parent-child relationships are no longer supported, as this feature was removed in ES 6.

      • Routing documents to specific Elasticsearch shards is not implemented. Please let us know if this feature is still relevant for your deployment.

      Older Releases

      Although no longer supported, documentation for older releases continues to be available in our docs archive.