Release Notes

[SPARKC-212](https://couchbasecloud.atlassian.net/browse/SPARKC-212): Better distribute query read partitions so they are all approximately the same size, resulting in a performance improvement.

[SPARKC-209](https://couchbasecloud.atlassian.net/browse/SPARKC-209): Limits are now pushed down in DataFrame query reads, for a performance improvement.

[SPARKC-208](https://couchbasecloud.atlassian.net/browse/SPARKC-208): Move to Spark 3.5.3.

[SPARKC-207](https://couchbasecloud.atlassian.net/browse/SPARKC-207): Many improvements to logging, with the most useful lines now logged at INFO rather than DEBUG.

[SPARKC-210](https://couchbasecloud.atlassian.net/browse/SPARKC-210): Fixes a 3.3.4 regression that caused each read query to stream significantly slower.

[SPARKC-211](https://couchbasecloud.atlassian.net/browse/SPARKC-211): Fixes a 3.3.4 regression that caused each read query to get executed twice.

[SPARKC-206](https://couchbasecloud.atlassian.net/browse/SPARKC-206): Fixes query read partitioning not working with aggregation operations such as groupBy and count.

SPARKC-196: Support authentication with client certificates. See Client Certificates for more information.

SPARKC-184: Support Spark 3.5.

SPARKC-185: Support Scala 2.13.

SPARKC-203: Support Capella Columnar.

SPARKC-194: Partitioning options added for DataFrame query reads.

SPARKC-181: When performing a DataFrame read (spark.read().format("couchbase.query")), rows will now be streamed and backpressured, rather than buffered in-memory.

SPARKC-181: Allow structured streaming to work against clusters that do not have the KV service running on all nodes.

SPARKC-178: Fix issues with connectionIdentifier.

SPARKC-177: If alternate addresses are configured on the cluster (for instance, for Kubernetes deployments), these will now be used automatically when doing structured streaming.

SPARKC-166: Improved handling of aggregate result datatypes.

SPARKC-176: Bump DCP client version.

SPARKC-174: Adjust default DCP bootstrap timeout to improve connection capability in high-latency environments.

SPARKC-167, SPARKC-168: Support connecting to multiple Clusters.

SPARKC-175, SPARKC-165: Support Spark 3.3.0.

SPARKC-160: Shade Reactor dependency into 'fatjar' build to better support the Databricks environment.

SPARKC-157: Add support for "ignore" variants for KeyValue insert, replace and remove operations. This has been available in the Spark Connector 2.x series and has not been ported forward until this point.

SPARKC-159: TLS configuration is now more flexible. TLS can be enabled trough couchbases:// in the connection string, as well as being more robust in accepting certail security config parameters.

SPARKC-160: To avoid classpath issues in the Databricks notebook environment, the Reactor library is now shaded in the assembly jar that is available for download. The regular jar still has the unshaded version of Reactor.

SPARKC-158: Open the implicitBucket if set for cluster-level operations. This makes sure that cluster-level operations like N1QL queries can be performed against Couchbase Server clusters pre 6.5.

Updated the underlying SDK to 1.3.0, which (along with general enhancements and fixes) bundles the Capella certificate, so it doesn’t have to be added manually.

SPARKC-133: Added support for spark structured streaming. For more details, see the documentation section for streaming.

Support for Apache Spark 3.2.x

SPARKC-146: Added support for scope-level Query and Analytics RDDs (through Keyspace)

SPARKC-148: Added support for aggregate pushdown for Query DataFrames.

SPARKC-149: Added support for aggregate pushdown for Analytics DataFrames.

SPARKC-143: Allow passing timeouts to KV, Query and Analytics DataFrame as option.

SPARKC-151: Fix ScanConsistency not being applied for Analytics and Query DataFrames.

SPARKC-144, SPARKC-145: Move Jackson JSON handling into the connector. This solves issues in the databricks notebook environment.

SPARKC-153: Properly escape fields for Analytics and Query DataFrames.

Support for Apache Spark 3.1.x

Support for Apache Spark 3.0.x

Built on top of the new Scala SDK 1.2.x

Support for Couchbase Server 7.0 and later

SPARKC-110: Allow passthrough of timestampFormat and dateFormat for inferring schema

Updated the Java SDK to 2.7.18

Allows enabling DNS SRV via a configuration property

SPARKC-104: Stopping and restarting of Spark-Couchbase-Streaming-job throws ClassCastException

Properly propagate the expiry into the DefaultSource

SPARKC-93: Support for Apache Spark 2.3.0

SPARKC-89: Support Analytics

SPARKC-88: Allow N1QL queries to run on Spark node(s) co-located with query service

SPARKC-96: Provide more fault-tolerant batch mutations. saveToCouchbase now takes a maxConcurrent parameter, giving the application control over the size of batches that will be written, from each executor.

SPARCK-85: Raise N1QL errors as exceptions rather than logging them.

SPARKC-82: When running a N1QLQuery, if multiple buckets have been specified, then the bucket to use must now be explicitly chosen. E.g. sc.couchbaseQuery(query, bucketName = "default"). This is safer than choosing an arbitrary bucket.

SPARKC-95: Fix to get streaming source working with Spark 2.3

Support for Apache Spark 2.2.0

SPARKC-80: Support for Couchbase Server 5.0 and Role-Based Access Control

SPARKC-77: Global and per-operation timeout configuration is now possible

SPARKC-44: Support for Subdocument Mutations has been added.

SPARKC-79: Support for easier SSL/TLS configuration via spark config.

SPARKC-77: per-operation timeout configuration is now possible