A newer version of this documentation is available.

View Latest

Release Notes

Release notes for the Kafka Connector.

Couchbase Kafka Connector 3.2.3 GA (2018-02-20)

Keepalive is now enabled on the Couchbase Server connection. This prevents the server from dropping idle connections, and enables dead connection detection. Thanks to community member Patrik Nordebo (patrikn).

A new config key "couchbase.log_redaction" controls whether sensitive info in connector log messages is tagged for redaction. Options are "NONE", "PARTIAL", and "FULL".

A new source config key "couchbase.compression" can be used to enable compression when reading from Couchbase Server 4.5 and later. Options are "DISABLED", "ENABLED", and "FORCED".

Issues resolved in this release:

  • KAFKAC-89: [ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo)

  • KAFKAC-82: [FEATURE] Implement log redaction for Kafka Connector

  • KAFKAC-90: [FEATURE] Source: Add config settings to enable compression

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.3</version>
</dependency>

Couchbase Kafka Connector 3.2.2 GA (19 December 2017)

The source connector now does a better job of reporting abnormal termination. Thanks to community member p_mx (tiny1990).

A new config key "couchbase.stream_from" lets you tell the source connector when in Couchbase history to start streaming from. Options are "BEGINNING", "NOW", "SAVED_OFFSET_OR_BEGINNING", and "SAVED_OFFSET_OR_NOW".

When the sink connector receives a Kafka message with a null value, it now deletes the Couchbase document whose ID matches the Kafka message key. (Previous versions would terminate when a null value was encountered.)

You can now specify durability requirements for the sink connector’s write operations via two new config keys: "couchbase.durability.persist_to" and "couchbase.durability.replicate_to".

Issues resolved in this release:

  • KAFKAC-84: [FEATURE] Sink: Allow setting durability requirements for Couchbase writes

  • KAFKAC-85: [FEATURE] Sink: Support deletion

  • KAFKAC-86: [FEATURE] Source: Restart from a given state / offset

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.2</version>
</dependency>

Couchbase Kafka Connector 3.2.1 GA (8 November 2017)

Fixes a regression in 3.2.0 that prevented the sink connector from working.

Issues resolved in this release:

  • KAFKAC-83: [BUGFIX] Sink connector fails due to UnsupportedOperationException in JsonBinaryTranscoder.newDocument.

  • KAFKAC-81: [ENHANCEMENT] Document how to customize source handler

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.1</version>
</dependency>

Couchbase Kafka Connector 3.2.0 GA (17 October 2017)

The sink connector is now officially supported.

Multiple source connector instances reading from the same Couchbase bucket can now manage their state independently. To enable this feature, assign a unique name to each connector and set the new config property compat.connector_name_in_offsets to true.

The source connector has a new, more flexible SourceHandler extension point intended to replace the Converter interface. By providing a custom SourceHandler, a developer can filter events, route messages to a topic other than the configured default topic, and control the format of the Kafka message.

The default Kafka message format now includes bucket and vBucketUuid fields, which may be used along with the partition (vBucketID) and bySeqno fields to construct a Couchbase MutationToken.

The sink connector now allows the Couchbase document ID to come from a field of the document. This feature is controlled by two new config properties, couchbase.document.id and couchbase.remove.document.id.

The Couchbase client libraries are upgraded to the latest versions, with performance enhancements and other improvements for Couchbase Server 5.0.

The following classes are scheduled for removal in version 4.0.0 of the connector:

  • com.couchbase.connect.kafka.converter.Converter - Deprecated in favor of SourceHandler.

  • com.couchbase.connect.kafka.converter.SchemaConverter - Deprecated in favor of DefaultSchemaSourceHandler.

Issues resolved in this release:

  • KAFKAC-69: [FEATURE] Allow Source connector to split DCP stream and write into separate topics.

  • KAFKAC-70: [FEATURE] Allow using connector name in offset storage namespace

  • KAFKAC-77: [FEATURE] Allow setting document ID from message field.

  • KAFKAC-78: [ENHANCEMENT] Upgrade Couchbase java-client to version 2.5.1, dcp-client to version 0.12.0

  • KAFKAC-79: [ENHANCEMENT] Use custom doc transcoder to reduce memory copies.

  • KAFKAC-80: [FEATURE] MutationToken Enabled in Kafka Connector to N1QL at plus.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.0</version>
</dependency>

Couchbase Kafka Connector 3.1.3 GA (31 May 2017)

Version 3.1.3 is maintenance release.

  • KAFKAC-71: Support for RBAC credentials on Couchbase Server 5+.

  • Java DCP Client updated to 0.10.0.

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.1.3</version>
</dependency>

Couchbase Kafka Connector 3.1.2 GA (14 March 2017)

Version 3.1.2 is maintenance release..

  • KAFKAC-66: On backfilling from large bucket, it is possible to get OOM when internal queue is not drained quickly enough to relay the data into Kafka.

Couchbase Kafka Connector 3.1.1 GA (21 February 2017)

Version 3.1.1 is maintenance release. It contains fixes for resuming DCP streams after restart.

  • KAFKAC-56: Session state might be left partially initialized, which leads to rolling back to sequence number zero (0) and starting from the beginning (duplicating events in Kafka topic).

Couchbase Kafka Connector 3.1.0 GA (03 January 2017)

Version 3.1.0 is GA release.

Couchbase Kafka Connector 3.0.0 GA (14 December 2016)

Version 3.0.0 is GA release. It brings documentation update.

Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)

Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings documentation update, feature enhancements and bug fixes

  • KAFKAC-52: Support for SSL connections

  • Update dependencies: dcp-client to 0.7.0, and confluent libraries up to versions shipped with 3.1.1

  • Cleanup various configuration workarounds for platform 3.0

Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)

Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.

  • KAFKAC-54: Create example of using in Kafka Stream to process events from Couchbase

  • Rename internal classes, and make configuration more consistent with other connectors (e.g. instead of timeout_ms, use timeout.ms)

  • Allow to override internal convertor into SourceRecord, and allow to inject Filter class to skip events before writing into Kafka

Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)

Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It implements new features and also includes bug fixes to previous release.

  • KAFKAC-50: Allow to buffer DCP snapshots for consistent writes.

  • KAFKAC-51: Specify key for SourceRecord. Allows to use multiple Kafka partitions.

  • KAFKAC-53: Node-aware distribution of partitions for Tasks. Reduces amount of resources allocated on the server.

Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)

Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. It improves configuration. And now can maintain replication state, which allow to resume transmission.

Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)

Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.