Release Notes

Release notes for the Kafka Connector.

Couchbase Kafka Connector 3.4.4 (2019-05-21)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.4</version>
</dependency>

This maintenance release fixes a bug that could prevent some versions of Couchbase Server from rebalancing when the connector is running.

Bug Fixes

  • KAFKAC-148 Couchbase Server fails to rebalance if Kafka connector is running

Couchbase Kafka Connector 3.4.3 (2019-03-19)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.3</version>
</dependency>

This patch release adds out-of-the-box support for publishing Couchbase documents to Kafka as raw JSON using RawJsonSourceHandler.

The quickstart guide has been updated with an example of how to apply Single Message Transforms to modify documents before the Sink connector writes them to Couchbase.

The included examples/custom-extensions Maven project now demonstrates how to implement a custom transform.

The new source handlers and transforms introduced in this patch version are "uncommitted" features, meaning they may change without advance notice in a future patch version.

Enhancements

  • KAFKAC-141 Add example of a custom Single Message Transform

  • KAFKAC-142 Support publishing raw JSON without having to build example project

  • KAFKAC-143 Upgrade DCP client to 0.22.0

Couchbase Kafka Connector 3.4.2 (2019-02-19)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.2</version>
</dependency>

This maintenance release improves the behavior of persistence polling and connection keep-alive.

When connecting to a multi-node Couchbase cluster, connections not used for streaming will no longer be incorrectly flagged as dead.

Persistence polling no longer repeatedly closes connections to servers not hosting any partitions during a rebalance operation.

In this release the Couchbase client dependency has been upgraded to version 2.7.2 for security updates in the repackaged Jackson databind library. Because Jackson versions compatible with Java 1.6 are no longer maintained, the connector is no longer compatible with Java 1.6. The connector now requires Java 1.8 or later. Couchbase prefers to make such platform changes in minor releases, but is applying the change here given the nature of the issue.

Enhancements

  • KAFKAC-136 Require Java 1.8 or later

  • KAFKAC-137 Fix Maven issues with Java 11

  • KAFKAC-138 Upgrade Kafka API dependencies to 1.0.2

  • KAFKAC-139 Upgrade Couchbase client to 2.7.2 and DCP client to 0.21.0

Bug Fixes

  • JDCP-116 BucketConfigHelper.getHostedPartitions() fails when node hosts no partitions

  • JDCP-117 Dead connection detection gives false positives when no streams are open

Couchbase Kafka Connector 3.4.1 (2019-01-15)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.1</version>
</dependency>

This maintenance release fixes memory leaks that can occur in distributed mode or under certain error conditions.

Also, the connector now responds to more error conditions by reporting them to the Kafka Connect framework instead of failing silently.

Issues resolved

  • KAFKAC-133: [BUGFIX] Buffers not released on all code paths

Couchbase Kafka Connector 3.4.0 (2018-10-16)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0</version>
</dependency>

New features

Version 3.4 introduces rollback mitigation in the form of persistence polling. Messages are published to Kafka only after the database events have been saved to disk on all Couchbase replicas. Rollback mitigation is enabled by default. It may be disabled by setting the persistence polling interval to 0.

The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

New configuration properties

  • couchbase.flow_control_buffer=128m

  • couchbase.persistence_polling_interval=100ms

Issues resolved

  • KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation

  • KAFKAC-127: [BUGFIX] Source connector can’t reconnect to Couchbase hosted on Docker Container

  • KAFKAC-124: [BUGFIX] Rollback mitigation causes data loss when opening DCP stream

Couchbase Kafka Connector 3.4.0-beta.1 (2018-07-06)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0-beta.1</version>
</dependency>
Version 3.4.0-beta.1 has a bug that can cause data loss. When rollback mitigation is enabled, some events might not be published to the Kafka topic. See KAFKAC-124 for details. This issue is resolved in the 3.4.0 release.

When database changes are rolled back due to failover of a Couchbase Server node, the connector will now by default prevent the rolled-back changes from being published to the Kafka topic. This is done by buffering changes until they have been persisted to all replicas in the Couchbase cluster (and are unlikely to be rolled back). The connector polls the database at a configurable interval to determine which changes have been persisted. This feature is enabled by default, and may be disabled by setting the polling interval to 0.

The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

New configuration properties:

  • couchbase.flow_control_buffer=128m

  • couchbase.persistence_polling_interval=100ms

Issues resolved in this release:

  • KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation

Couchbase Kafka Connector 3.3.2 GA (2018-07-02)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.2</version>
</dependency>

The source connector now resumes streaming after a Couchbase Server rollback.

Issues resolved in this release:

  • KAFKAC-72: [BUGFIX] Connector stops streaming after rollback

  • KAFKAC-115: [ENHANCEMENT] Upgrade to DCP client 0.18.0

Couchbase Kafka Connector 3.3.1 GA (2018-06-19)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.1</version>
</dependency>

The sink connector can now set expiration times for documents, and can assign document IDs by composing values from multiple fields.

The source connector now continues streaming after a graceful failover.

Issues resolved in this release:

  • KAFKAC-104: [ENHANCEMENT] Support for setting TTL on items being written to Couchbase

  • KAFKAC-106: [ENHANCEMENT] Support assigning document ID from multiple fields

  • KAFKAC-109: [ENHANCEMENT] KAFKAC-109 Upgrade to Java client 2.5.9 and DCP client 0.17.0

  • JDCP-85: [BUGFIX] Client stops receiving some events after failover

  • JDCP-94: [BUGFIX] Should never pass null ChannelFlowController to event handlers

Couchbase Kafka Connector 3.3.0 GA (2018-05-15)

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.0</version>
</dependency>

Compression is now enabled by default (requires Couchbase Server 5.5 or later).

IPv6 is now enabled by default. To disable IPv6 in a dual-stack environment, set the new couchbase.forceIPv4 config property to true.

Resolved a regression that caused the connector to need a restart after a Couchbase Server restart.

Couchbase password and SSL keystore password may be specified using via KAFKA_COUCHBASE_PASSWORD and KAFKA_COUCHBASE_SSL_KEYSTORE_PASSWORD environment variables.

The sink connector can now modify documents using the sub-document API (many thanks to community member Didier Caron).

Issues resolved in this release:

  • KAFKAC-97: [BUGFIX] Kafka Connector needs a restart after Couchbase Server restart

  • KAFKAC-102: [ENHANCEMENT] Enable compression by default

  • KAFKAC-100: [ENHANCEMENT] Enable IPv6 by default, add first-class config option

  • KAFKAC-96: [ENHANCEMENT] Support setting password via environment variable

  • KAFKAC-98: [ENHANCEMENT] Sink: Support sub-document access (Didier Caron)

  • KAFKAC-103: [ENHANCEMENT] Upgrade Couchbase dependencies

Couchbase Kafka Connector 3.2.3 GA (2018-02-20)

Keepalive is now enabled on the Couchbase Server connection. This prevents the server from dropping idle connections, and enables dead connection detection. Thanks to community member Patrik Nordebo (patrikn).

A new config key "couchbase.log_redaction" controls whether sensitive info in connector log messages is tagged for redaction. Options are "NONE", "PARTIAL", and "FULL".

A new source config key "couchbase.compression" can be used to enable compression when reading from Couchbase Server 4.5 and later. Options are "DISABLED", "ENABLED", and "FORCED".

Issues resolved in this release:

  • KAFKAC-89: [ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo)

  • KAFKAC-82: [FEATURE] Implement log redaction for Kafka Connector

  • KAFKAC-90: [FEATURE] Source: Add config settings to enable compression

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.3</version>
</dependency>

Couchbase Kafka Connector 3.2.2 GA (19 December 2017)

The source connector now does a better job of reporting abnormal termination. Thanks to community member p_mx (tiny1990).

A new config key "couchbase.stream_from" lets you tell the source connector when in Couchbase history to start streaming from. Options are "BEGINNING", "NOW", "SAVED_OFFSET_OR_BEGINNING", and "SAVED_OFFSET_OR_NOW".

When the sink connector receives a Kafka message with a null value, it now deletes the Couchbase document whose ID matches the Kafka message key. (Previous versions would terminate when a null value was encountered.)

You can now specify durability requirements for the sink connector’s write operations via two new config keys: "couchbase.durability.persist_to" and "couchbase.durability.replicate_to".

Issues resolved in this release:

  • KAFKAC-84: [FEATURE] Sink: Allow setting durability requirements for Couchbase writes

  • KAFKAC-85: [FEATURE] Sink: Support deletion

  • KAFKAC-86: [FEATURE] Source: Restart from a given state / offset

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.2</version>
</dependency>

Couchbase Kafka Connector 3.2.1 GA (8 November 2017)

Fixes a regression in 3.2.0 that prevented the sink connector from working.

Issues resolved in this release:

  • KAFKAC-83: [BUGFIX] Sink connector fails due to UnsupportedOperationException in JsonBinaryTranscoder.newDocument.

  • KAFKAC-81: [ENHANCEMENT] Document how to customize source handler

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.1</version>
</dependency>

Couchbase Kafka Connector 3.2.0 GA (17 October 2017)

The sink connector is now officially supported.

Multiple source connector instances reading from the same Couchbase bucket can now manage their state independently. To enable this feature, assign a unique name to each connector and set the new config property compat.connector_name_in_offsets to true.

The source connector has a new, more flexible SourceHandler extension point intended to replace the Converter interface. By providing a custom SourceHandler, a developer can filter events, route messages to a topic other than the configured default topic, and control the format of the Kafka message.

The default Kafka message format now includes bucket and vBucketUuid fields, which may be used along with the partition (vBucketID) and bySeqno fields to construct a Couchbase MutationToken.

The sink connector now allows the Couchbase document ID to come from a field of the document. This feature is controlled by two new config properties, couchbase.document.id and couchbase.remove.document.id.

The Couchbase client libraries are upgraded to the latest versions, with performance enhancements and other improvements for Couchbase Server 5.0.

The following classes are scheduled for removal in version 4.0.0 of the connector:

  • com.couchbase.connect.kafka.converter.Converter - Deprecated in favor of SourceHandler.

  • com.couchbase.connect.kafka.converter.SchemaConverter - Deprecated in favor of DefaultSchemaSourceHandler.

Issues resolved in this release:

  • KAFKAC-69: [FEATURE] Allow Source connector to split DCP stream and write into separate topics.

  • KAFKAC-70: [FEATURE] Allow using connector name in offset storage namespace

  • KAFKAC-77: [FEATURE] Allow setting document ID from message field.

  • KAFKAC-78: [ENHANCEMENT] Upgrade Couchbase java-client to version 2.5.1, dcp-client to version 0.12.0

  • KAFKAC-79: [ENHANCEMENT] Use custom doc transcoder to reduce memory copies.

  • KAFKAC-80: [FEATURE] MutationToken Enabled in Kafka Connector to N1QL at plus.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.0</version>
</dependency>

Couchbase Kafka Connector 3.1.3 GA (31 May 2017)

Version 3.1.3 is maintenance release.

  • KAFKAC-71: Support for RBAC credentials on Couchbase Server 5+. Java DCP Client updated to 0.10.0.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.1.3</version>
</dependency>

Couchbase Kafka Connector 3.1.2 GA (14 March 2017)

Version 3.1.2 is maintenance release.

  • KAFKAC-66: On backfilling from large bucket, it is possible to get OOM when internal queue is not drained quickly enough to relay the data into Kafka.

Couchbase Kafka Connector 3.1.1 GA (21 February 2017)

Version 3.1.1 is maintenance release. It contains fixes for resuming DCP streams after restart.

  • KAFKAC-56: Session state might be left partially initialized, which leads to rolling back to sequence number zero (0) and starting from the beginning (duplicating events in Kafka topic).

Couchbase Kafka Connector 3.1.0 GA (03 January 2017)

Version 3.1.0 is GA release.

Couchbase Kafka Connector 3.0.0 GA (14 December 2016)

Version 3.0.0 is GA release. It brings documentation update.

Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)

Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings documentation update, feature enhancements and bug fixes

  • KAFKAC-52: Support for SSL connections

  • Update dependencies: dcp-client to 0.7.0, and confluent libraries up to versions shipped with 3.1.1

  • Cleanup various configuration workarounds for platform 3.0

Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)

Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.

  • KAFKAC-54: Create example of using in Kafka Stream to process events from Couchbase

  • Rename internal classes, and make configuration more consistent with other connectors (e.g. instead of timeout_ms, use timeout.ms)

  • Allow to override internal convertor into SourceRecord, and allow to inject Filter class to skip events before writing into Kafka

Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)

Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It implements new features and also includes bug fixes to previous release.

  • KAFKAC-50: Allow to buffer DCP snapshots for consistent writes.

  • KAFKAC-51: Specify key for SourceRecord. Allows to use multiple Kafka partitions.

  • KAFKAC-53: Node-aware distribution of partitions for Tasks. Reduces amount of resources allocated on the server.

Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)

Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. It improves configuration. And now can maintain replication state, which allow to resume transmission.

Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)

Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.

Parent topic: Kafka Connector