Release Notes

    +
    Release notes, installation instructions, and download archive for the Couchbase Kafka Connector.

    This page covers version 4.x of the Couchbase Kafka Connector. For earlier versions, please see the 3.x Release Notes.

    Installation

    Scroll down to the version you want, then click the "Download" link to get the full connector distribution. Refer to the Quickstart Guide for detailed installation instructions.

    Java API Maven Coordinates

    You can optionally extend the connector’s functionality by writing custom components. When developing a custom component, include the connector’s Java API in your Maven project by adding this dependency to your POM:

    <dependency>
        <groupId>com.couchbase.client</groupId>
        <artifactId>kafka-connect-couchbase</artifactId>
        <version><!-- connector version --></version>
    </dependency>

    Version 4.1.0 (22 July 2021)

    This release stabilizes the configuration options for working with Couchbase 7 Scopes and Collections. All previously "uncommitted" options are promoted to "committed" status.

    There’s also a new SinkHandler extension point, and bug fixes for the Subdocument and N1QL modes of operation.

    Breaking Changes

    • Apache Kafka 2.4.0 is now the minimum required version. For Confluent Platform users, this corresponds to Confluent version 5.4.0.

    Enhancements

    • The config options for working with Couchbase 7 Scopes and Collections are now part of the "committed" API.

    • All other "uncommitted" config options are promoted to "committed" status as well.

    • KAFKAC-257: Sink The connector’s behavior is now completely customizable by implementing the SinkHandler interface. The new couchbase.sink.handler config option specifies the class to use. See the custom extensions example code to see how to implement your own handler.

    The couchbase.document.mode config option is now DEPRECATED. Instead, please use couchbase.sink.handler to specify one of the built-in handler classes.
    • KAFKAC-263: Source Upgraded DCP client from 0.34.0 to 0.35.0.

    • KAFKAC-258: Sink Upgraded Couchbase Java SDK from 3.1.3 to 3.2.0.

    Bug Fixes

    • KAFKAC-261: Sink A connector configured to use SUBDOCUMENT mode with an operation of ARRAY_PREPEND or ARRAY_PREPEND could mistakenly ignore updates if Kafka records targeting the same document arrive in rapid succession. This is now fixed.

    • KAFKAC-262: Sink A connector configured to use N1QL mode could mistakenly ignore updates if Kafka records targeting the same document(s) arrive in rapid succession. This is now fixed.

    Version 4.0.6 (20 April 2021)

    This release adds new Sink configuration options, and addresses a long-standing issue that caused the Source connector to fail when a rollback occurred in Couchbase.

    Enhancements

    Bug Fixes

    • KAFKAC-211: Source A rollback in Couchbase no longer causes the connector to fail.

    Version 4.0.5 (16 March 2021)

    This release improves diagnostic logging and simplifies Couchbase Cloud configuration.

    Enhancements

    • KAFKAC-234: The Couchbase root CA certificate can now be read directly from a PEM file; it’s no longer necessary to add it to a Java keystore first. The new couchbase.trust.certificate.path config property points to the PEM file.

    • KAFKAC-242: Source When the new couchbase.log.document.lifecycle config property is set to true, the connector writes detailed log entries as each document flows through the connector.

    • KAFKAC-245: Sink Upgraded Couchbase client from 3.1.2 to 3.1.3.

    • KAFKAC-246: Source Upgraded DCP client from 0.32.0 to 0.33.0. OBSERVE_SEQNO events are now logged at TRACE level instead of DEBUG.

    Version 4.0.4 (17 February 2021)

    This release adds uncommitted support for client certificate authentication (mTLS), adds hostname verification for secure DCP connections, and improves the stability of the connector.

    Enhancements

    • KAFKAC-241: When secure connections are enabled, it is now possible to authenticate with Couchbase using an X.509 certificate instead of a username & password. See couchbase.client.certificate.path and couchbase.client.certificate.password for details. (This feature is added as "uncommitted", meaning it may change without notice.)

    • KAFKAC-238: Sink Upgraded Couchbase client from 3.0.9 to 3.1.2.

    • KAFKAC-239: Source Upgraded DCP client from 0.31.0 to 0.32.0. Previously, TLS hostname verification was done only for the Couchbase Java client connection; now the DCP client connection is verified as well. If for some reason you need to disable TLS hostname verification, this is now possible by setting the couchbase.enable.hostname.verification config property to false.

    Bug Fixes

    • JDCP-183: If an invalid stream offset is detected, the connector will now fail fast instead of potentially corrupting the saved checkpoint.

    • JDCP-184: Resolved an issue that could cause a flow control deadlock when streaming from a subset of a bucket’s collections or scopes.

    Version 4.0.3

    Not released.

    Version 4.0.2 (17 November 2020)

    In this release, the connector publishes metrics via JMX to facilitate monitoring.

    Enhancements

    • KAFKAC-152: Documentation - Added a "Monitoring" page which refers users to the Kafka Connect framework documentation for monitoring connectors via JMX.

    • KAFKAC-232: Source - Metrics specific to the Couchbase source connector are now exposed via JMX under the kafka.connect.couchbase domain.

    • KAFKAC-110: Source - Added a gauge that reports the Couchbase connection status.

    • KAFKAC-231: Source - Upgrade DCP client from 0.30.0 to 0.31.0

    Version 4.0.1 (20 October 2020)

    This release improves compatibility with Couchbase Cloud, and exposes extended attributes (XATTRS) to custom components.

    Enhancements

    • KAFKAC-227: Source - Added config property couchbase.xattrs (boolean, defaults to false). If set to true, a custom Filter or SourceHandler may inspect a document’s extended attributes by calling DocumentEvent.xattrs().

    • KAFKAC-226: Renamed the ZIP archive from couchbaseinc-kafka-connect-couchbase to couchbase-kafka-connect-couchbase (removed the "inc" from "couchbaseinc"). This made it easier to publish the connector on Confluent Hub.

    • KAFKAC-228: Upgraded the Couchbase Java SDK from 3.0.6 to 3.0.9. Bootstrap performance is improved when specifying custom ports, and the connector no longer logs spurious warnings about being unable to fetch collections manifests.

    • KAFKAC-229: Upgraded the DCP client from 0.29.0 to 0.30.0, adding support for XATTRs.

    Bug Fixes

    • KAFKAC-225: Fixed a regression in version 4.0.0 that broke alternate address resolution. The connector now handles DNS SRV and alternate addresses correctly, and can connect to Couchbase Cloud or other network environments that use alternate addresses.

    Version 4.0.0 (18 August 2020)

    Version 4.0 is a major update that changes how you configure and extend the connector. If you are upgrading from a previous version of the connector, be sure to read the Migration Guide which will help with the upgrade process.

    The leap to 4.0 brings many new features, most notably:

    • Enhanced durability options (requires Couchbase 6.5)

    • Better workload distribution

    • More flexible API for extensions

    • Option to omit document contents

    • Support for Couchbase collections and scopes (planned for Couchbase 7.0)

    The notes below describe these features, and more.

    The new configuration properties related to Couchbase scopes and collections are "uncommitted" and may change without notice in a patch release.

    Enhancements

    Complete list of changes since version 3.4.8
    • KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

    • KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

    • KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

    • KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

    • KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

    • KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

    • KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

    • KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

    • KAFKAC-194: Source - A custom SourceHandler can now set headers on the Kafka record.

    • KAFKAC-223: Source - The connector is now more responsive to "pause" requests.

    • KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

    • KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

    • KAFKAC-196: Source - Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

    • KAFKAC-195: Source - Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

    • KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

    • KAFKAC-175: APIs deprecated in version 3.x have been removed.

    • Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

    • Upgraded Couchbase client from 2.7.13 to 3.0.6.

    • Upgraded DCP client from 0.26.0 to 0.29.0.

    Bug Fixes

    Complete list of changes since version 3.4.8
    • KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

    Changes since 4.0.0-dp.3

    • KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

    • KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

    Version 4.0.0-dp.3 (21 July 2020)

    In this developer preview, both the Sink and Source connector now support Couchbase collections. This preview also brings a handful of fixes and new features, including support for enhanced durability, and optionally omitting document contents.

    The new features in this pre-release version should be considered "volatile" and may change before the 4.0.0 GA release.

    Enhancements

    • KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

    • KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

    • KAFKAC-206: Source - Config property couchbase.connector.name.in.offsets now defaults to false again. This property doesn’t do anything useful, and should only be set to true if you previously had compat.connector_name_in_offsets set to true.

    • KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

    • KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

    • KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

    • KAFKAC-205: Removed the unused couchbase.force.ipv4 config property.

    Bug Fixes

    • KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

    • KAFKAC-214: Sink - The Couchbase Java SDK has been updated from 3.0.5 to 3.0.6. As a result, setting couchbase.document.expiration to longer than 30 days now works correctly instead of causing immediate expiration. (This was a regression in 4.0.0-dp.1.)

    • KAFKAC-203: Source - The 3.x → 4.0 migration script now properly converts the old couchbase.flow_control_buffer property to the new name: couchbase.flow.control.buffer.size.

    • KAFKAC-204: Source - Fixed the invalid value for couchbase.bootstrap.timeout in the quickstart-couchbase-source.json example config file.

    Version 4.0.0-dp.1 (17 June 2020)

    This developer preview version offers a sneak peek at some features coming in version 4.0.0 of the Couchbase Kafka connector, including support for Couchbase Collections and Scopes.

    Version 4.0 is a major update that changes how you configure and extend the connector. If you are upgrading from a previous version of the connector, be sure to read the Migration Guide which will help you with the upgrade process.

    The new features in this pre-release version should be considered "volatile" and may change before the 4.0.0 GA release.

    Enhancements

    • KAFKAC-182: Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

    • KAFKAC-188: Upgraded Couchbase client from 2.7.13 to 3.0.5.

    • KAFKAC-189: Upgraded DCP client from 0.26.0 to 0.28.0.

    • KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

    • KAFKAC-196: Source: Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

    • KAFKAC-195: Source: Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

    • KAFKAC-194: Source: A custom SourceHandler can now set headers on the Kafka record.

    • KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

    • KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

    • KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

    • KAFKAC-173: The couchbase.connector.name.in.offsets config property (previously called compat.connector_name_in_offsets) now defaults to true.

    • KAFKAC-175: APIs deprecated in version 3.x have been removed.

    Older Releases

    Although no longer supported, documentation for older releases continues to be available in our docs archive.

    Parent topic: Kafka Connector