Release Notes for Couchbase Server 7.2

MB-60813

This release adds the median function to the builtin analytics functions. See Builtin Functions.

MB-61310

MB-60972

Previously, an index scan would continue reading data until it had returned at least one row of data, even if it had been cancelled or had timed out. These continued reads resulted in wasted CPU resources and potentially blocking other scans from starting.

Couchbase Server now stops an index scan from reading data after it has timed out or a client cancels it even if it has not produced a row of data.

MB-61043

A rebalance followed by a failover could result in partitions not being evenly distributed across all nodes by a later rebalance.

The rebalance process now detects the partition skew and fixes it.

MB-61435

An optimization that skips key-value lookups in some cases when inserting new keys could be incorrectly enabled for all indexes in buckets containing empty indexes. This issue could result in duplicate rows being added to indexes.

The optimization is not longer incorrectly applied to indexes and duplicate rows are no longer added to indexes.

MB-60568

When running a VM on a XEN hypervisor, metrics were computed using the number of processors configured for the entire system.

MB-58694

DCP messages are now processed once, regardless of the configured dcpIdleQueueFlushThresholdMillis.
High CPU usage will not trigger unnecessary flushes of DCP state.

MB-58723

Frames that contain only DCP data were unnecessarily processed by the complete ingestion pipeline, resulting in performance degradation.

The complete ingestion pipeline no longer processes frames that only contain DCP data.
Instead, they are processed specifically by the ingestion operators that update DCP metadata, resulting in better performance.

MB-58450

An issue occurred with a node failover.
If a user did not bring in a replacement node before the failover occurred, then lost active or lost replica search indexes that had partitions on the replaced node were not rebuilt. As a result, search requests returned partial/incomplete results. In addition, existing indexes with defined replica(s) became vulnerable to potential node failures.

This issue is only pertinent to the 7.2.1 release, and does not affect older builds.

During rebalancing, a check is now made for missing partitions.
If missing partitions are found, they are rebuilt before the rebalance is complete.

MB-56957

External collections could not be created using Azure Managed Identity.

Azure dependencies have been updated to correct this issue.

MB-57588

Query results could be unnecessarily converted twice to JSON when documents were large.

The Query result is now converted to JSON once for all documents.

MB-57615

When the Prometheus stats returned from Analytics exceeded four kilobytes, the status code was inadvertently set to 500 (Internal Error), and this resulted in a large number of warnings in the Analytics warning log. Couchbase Server discarded these statistics.

This has been fixed to properly return a 200 (OK) status code when the size of Prometheus stats exceeds 4KiB, allowing these stats to be recorded properly. The warning is not displayed.

MB-39344

The last item in a replica checkpoint was not expelled. In scenarios such as large average item size, high numbers of replicas or low Bucket quota could result in a data-node entering an unrecoverable Out-of-Memory state.

ItemExpel has been enhanced to release all the items in a checkpoint when memory conditions allow.

MB-56084

A rollback loop affected legacy clients when collections were used and a tombstone newer than the last mutation in the default collection was purged.

The lastReadSeqno is now Incremented when the client is not collection-aware.

MB-56644

In rare cases, after a failover or memcached restart, a replica rollback while under memory pressure might have caused a crash in the Data Service.

Memory pressure recovery logic (Item expelling) is now skipped when replica rollback is in progress.

MB-56970

XDCR or restore from backup entered an endless loop if attempting to overwrite a document which was deleted or expired some time ago with a deleteWithMeta operation. This was due to a specific unanticipated state in memory which increased CPU usage, and connection became unusable for further operations.

deleteWithMeta is now resilient to temporary non-existent values with xattr datatype.

MB-57002

When using .NET SDK on Windows 10 client and client certs were enabled on CB Server, the Data-Service did not establish a connection and client bootstrap failed with a OpenSSL “session id context uninitialized" error.

Data-Service has been updated to disable TLS session resume.

MB-57064

GET_META requests for deleted items fetched metadata in memory which was not evicted in value-eviction buckets.

Metadata items are now cleaned when the expiry pager runs.

MB-57106

DCP clients streamed in out-of-sequence-order [OSO] backfill snapshots under Magma observed duplicate documents received in the disk snapshot. This happened where the stream was paused and resumed when the resume point was wrongly set to a key already processed in the stream.

OSO backfill in Magma now sets the correct resume point after a pause.

MB-57304

Data Service rebalance duration was significantly impacted if other DCP clients created a large number of Streams, if those streams needed to be read from disk, due to the lack of prioritizing between rebalance and other DCP clients.

The number of backfills each DCP client can perform concurrently has been limited to allow fairer allocation of resources.

MB-57400

The computation count for the items remaining DCP/Checkpoint stats exposed to Prometheus was the O(N) function. Where N is the number of items in a checkpoint. This caused various performance issues including Prometheus stats timeouts when checkpoints accumulated a high number of items.

The computation count has been optimized and now is O(1).

MB-57609

A spurious auto-failover could happen when Magma compaction visited a TTL’d document that was already deleted.

Document not found does not now increment the number of read failures.

MB-56339

During scaling, an GSI indexer rebalance froze and did not successfully complete. This was because an index snapshot was not correctly deleted and recreated.

A flag now handles snapshots to ensure they are correctly deleted or recreated when indexes are updated during rebalancing.

MB-57021

When alter index updated the replica count, new replicas were not built immediately when the original definition was {defer_build: true}. Existing replicas were built and new replicas were built in the next processing iteration.

New replicas are now built when the replica count is updated for deferred indexes. The status of existing index instances is checked, and if ready, a new build of the instance is triggered.

MB-57777

When the indexer was unable to keep up with KV mutations, and there was a queue of mutations within the indexer, there was a large memory overhead from the bookkeeping of queued up mutations.

Indexer has been improved to optimize memory usage so that the bookkeeping overhead is reduced for queued up mutations.

MB-56533

Due to how nested dependencies were handled, a sudden rise in memory utilization of the query service on a node caused a memory alert issue. The node did not recover correctly following a restart.

Nested dependencies are now handled appropriately in the ADVISE statement.

MB-56563

A query with multiple filters on an index key, one of which was a parameter, could produce incorrect results. This was caused by incorrectly composing the exact index spans to support the query.

The way in which exact spans are set has been modified to correct this issue.

MB-56579

Covering FLATTEN_KEYS() on an array index generated incorrect results. This was because a modified version of the ANY clause was applied after the index which meant false positives were retained and Distinct scan rows were eliminated.

The ANY filter is now applied on an index scan itself when covering an index scan with flatten keys.

MB-56683

Inter-service read timeout errors were not detected or handled accordingly. User requests consequently failed with timeout errors without retrying with a new connection.

The error handling and retry mechanism has been modified to handle these types of timeout issues and errors.

MB-56727

Under certain circumstances, a query with UNNEST used a covering index scan and incorrect results were returned. Reference to the UNNEST expression should have prevented the covering index from being used for the query as the index did not contain the entire array.

The logic to determine covering UNNEST scans has been changed to not use a covering index scan for such queries.

MB-56937

When an index scan had multiple spans, index selectivity was incorrectly calculated.

Index selectivity for multiple index spans is now correctly calculated.

MB-57024

Incorrect results were returned for a non-IndexScan on a constant false condition. This was due to incorrect handling of a FALSE WHERE clause.

The FALSE WHERE clause is now correctly handled.

MB-57029

Querying system:functions_cache in a multi query node cluster returned incomplete results with warnings. The query result included entries in the local query node, but none from remote query nodes. This was due to a typographical error.

The typographical error has been corrected.

MB-57080

A panic in go_json.stateInString under parsed value functions caused by incorrect concurrent access resulted in the state being freed whilst still in use.

The concurrent access issue has been resolved.

MB-57251

A Prepared statement might have resulted in an incorrect result in a multi-node environment. For example, a database with two query nodes.

Correlated subqueries from an encoded plan are now detected and marked. This ensures correct results are provided.

MB-57279

When a WITH clause (common table expression, or CTE) was used inside a subquery, and the WITH clause definition referenced the parent query, and was correlated, the query engine did not properly detect the correlation. This produced an incorrect result from the WITH clause evaluation because the result was not cached correctly.

Correlations inside WITH clause definitions are now properly detected.

MB-57316

cbq required a client authentication key file whenever a certificate authority file was used.

cbq now accepts a certificate authority file without a client key file enabling use with username and password credentials.

MB-57680

When appropriate optimizer statistics were used in Cost-Based Optimizer (CBO), for a query with ORDER BY, if there were multiple indexes available for the query, CBO unconditionally favored an index that provided ordering. Such indexes were not always the best ones to use.

CBO now allows cost-based comparison of indexes.

MB-57838

An ADVISE statement with multiple levels of UNNEST caused a syntax error in the CREATE INDEX statement from the Index Advisor.

ADVISE has been improved when there are queries with multiple levels of UNNEST.

MB-57484

A Cluster Manager process crash meant the Delete Bucket memcached command was not always called before bucket files were deleted later in rebalance. This caused the memcached process to crash repeatedly causing data service downtime.

The Delete Bucket command is now called on memcached before a file is deleted during rebalance. This ensures mencached doesn’t attempt to read the files.

MB-56601

Data streamed from the Data Service over XDCR should always be streamed in order by mutation id. However, in some scenarios, for efficiency, the Data Service streamed records that were not ordered by mutation id. In certain situations, this out-of-sequence-order [OSO] caused performance issues.

OSO mode is now available as a global override to be switched off for any currently deployed replications to avoid performance issues.

MB-56711

XDCR did not process documents with a JSON array and Extended Attributes (XATTRs). When a document contained XATTRs, XDCR checked for XATTRs in transactions, transaction filters were enabled, and XATTRs were not checked.

When documents contain arrays, XATTRs are now checked in the transaction XATTRs, and the document is not prevented from being parsed in an array.

MB-56741

Binary documents were replicated when an Advanced Filtering Expression was present.

A filter has been added which can be turned on to prevent all binary documents from being replicated.

MB-56773

It appeared that XDCR had stalled and an explanation was not provided. For rebalances on the source or target, the XDCR pipeline should be restarted, and data movement should continue. Before the pipeline was restated, there might have been fewer data movements as the rebalanced VBs were no longer streaming.

An ETA is now provided in the Server UI to show when the pipeline is due to be restarted.

MB-56799

Checkpoint Manager created checkpoint records out-of-sequence when many target nodes ran slowly.

Checkpoint Manager now creates checkpoints in sequence when target nodes are slow.

MB-56848

The bucket topology service sent a concurrent map iteration and map write panic to XDCR which caused a fatal error.

Validation has been improved to prevent the panic from happening.

MB-56938

Prometheus stats did not include a pipeline’s status.

The pipeline status is now provided as part of a prometheus stat.

MB-57130

A Checkpoint Manager Initialization error caused two memory leak types. These were a backfill pipeline and a main pipeline memory leak.

The Pipeline Manager and backfill pipeline have both been modified to prevent the memory leaks.

MB-57183

XDCR Checkpoint Manager instances were not cleaned up under certain circumstances due to timing and networking issues when contacting target, or when an invalid backfill task was fed in as input.

Checkpoint Manager instances are now cleaned up. A flag has been added to check for invalid backfill tasks.

MB-57234

When a replication spec change was made to a non-Data Service node, delete replication hung and caused the node to return an incorrect replication configuration.

XDCR now checks that the node is running the Data Service and handles it correctly.

MB-57277

XDCR could fail due to multiple connection issues. For example, DNS issues or firewalls. For a number of databases, it was a difficult task to manually check every node to determine where the connection issue was. For multiple nodes in a database and in the target database, debugging the issue required many connection checks.

A connection pre-check feature has been added to XDCR which ensures all connections from source nodes to target nodes are valid. Credentials are now also checked.

MB-57412

Running ipv6 only mode + non-encrypted remote resulted in invalid IP addresses being returned, leading to connection issues.

A valid IP address is now returned.

MB-57462

StatsMgr stopping could hang due to watching for notifications resulting in stranded go-routines.

Go-routines are now stopped correctly.

MB-57562

When ipv4 only mode was used, and full encryption only had an alternate address configured where the internal address was unresolvable, XDCR resulted in an error when it contacted the target data nodes.

The specific scenario has been fixed so that replication can now proceed.

MB-57743

The Prometheus endpoint did not expose any XDCR error metrics.

XDCR error metrics are now exposed via Prometheus.

MB-57787

A legacy race condition where metadata store could cause a conflict was exposed as part of the binary filter improvements.

Legacy race conditions have all been resolved.

MB-58203

Under certain circumstances, when rebalancing, the target cluster could return an EACCESS error code that caused source XDCR to pause the pipeline.

This has been reversed. Instead of pausing the pipeline when rebalancing, XDCR now retries when an EACCESS error is encountered in XmemNozzle. XDCR counts and prints this activity in the log.

MB-58545

Checkpoint Manager could be stuck when stopping if it had not been started yet, resulting in memory leak.

Checkpoint Manager can now be stopped correctly even when it hasn’t been started.

MB-56464

An issue occurred where the Cluster Manager instructed Prometheus to reload the configuration and the reload timeout impacted other requests.

The Cluster Manager has been improved to handle timeouts when instructing Prometheus to reload the configuration.

MB-56517

The Cluster Manager’s computed utilization stats were inaccurate due to time interval discrepancies in components where data was collected.

The Cluster Manager now reports raw stats as Prometheus counters.

MB-57157

Inconsistencies were observed where a single Magma bucket in a database took a long time to warm up.

The seq index scan has been optimized for tombstones of zero value size. Optimization is for look up by key, sequence iteration, and key iteration. Docs of 0 value size are placed in both key index and seq index.

MB-57714

Disk backfills were hanging permanently due to high memory consumption when large documents were streamed over many DCP streams concurrently.

Memory for a document read by a DCP stream is now released before switching to another stream.

MB-58450

An issue occurs with node failover. If a user does not bring in a replacement node before the failover occurs, then lost active or lost replica search indexes that have partitions on the replaced node are not rebuilt.
As a result, search requests return partial/incomplete results. In addition, existing indexes with defined replica(s) become vulnerable to another node failure.

This issue is only pertinent to the 7.2.1 release, and does not affect older builds.

This situation can be prevented if a replacement node was brought into the cluster in place of the failed over node, before starting the rebalance operation. This problem should not affect on-line or off-line rebalances.

If lost indexes are encountered, then, a manual update to the affected search index definition(s) will trigger a rebuild of the affected indexes.
(The advice here is to toggle the replica count so that unaffected index partitions are re-used, and only the lost partitions are rebuilt.)

MB-54396

Reporting wrong fragmentation and data size stats.

MB-55769

Buckets page should load even if the browser machine is slow or bandwidth is low.

MB-53065

Prolonged TMPFAIL or ENOMEM causes memory bloat.

MB-54332

Inter Cluster XDCR failing in Server 7.1.2 and Capella.

MB-55060

XDCR on non-KV node freezes when replication settings changed several times.

MB-54034

Display the number of uses for prepared statements accurately.

MB-54040

[SQL++] insert does not trigger a memory quota exceeded.

MB-54043

Potential for request stall if stream operator fails to notify request that it has terminated.

MB-54044

Optimizer hints are not displayed in EXPLAIN statements for subqueries.

MB-54326

Disable impersonate if KV node does not support collections to prevent Query service errors when upgrading from 6.6.5.

MB-54514

ORDER BY after UNION requires explicitly aliased terms.

MB-54596

Covered FTS SEARCH() with memory_quota fails.

MB-54705

Active requests and queued requests in SQL++ metrics are gauges, not counters.

MB-55864

Memory exceeded quota error with ARRAY_AGG.

MB-57215

OBJECT_ functions may return incorrect results.

MB-54348

When an index drop is immediately followed by bucket delete, the indexer can deadlock when a rare race condition occurs.

MB-55195

A scanning issue occurs when an index is on a node and the replica index is on a different node.

MB-55216

Index build hangs in mixed mode when the projector skips transaction records.

MB-55217

Index build stuck on "Check pending stream" during shard rebalance testing.

MB-55228

Report aggregated node level statistics information using prometheus.

MB-55218

Use streamId instead of index.Stream to determine stream catchup pending.

MB-55243

cbindex did not execute the build index and the performance test is stuck.

MB-55246

Change log level when using watchers to connect to indexer services in a cluster.

MB-55248

"FlushTs Not Snapshot Aligned" message incorrectly displayed in Log multiple times.

MB-55312

Nil value in the Node table causes a panic issue in the logs.

MB-55512

Orphaned watcher background thread in logs following a Server upgrade.

MB-55517

RedistributeIndexes flag should consider partitioned and non-partitioned indexes.

MB-55426

Address plasma RP Version 16 bit overflow, and recovery and data logs.

MB-56016

Ensure rows that were previously compacted do not return when a crash and recovery occurs in the Magma storage engine.

MB-53270

Use "_txn:" so the Eventing service detects and rejects transaction documents.

MB-53393

Restoring Eventing Functions to new scope in the same bucket upgrades and overwrites the admin or global Function Scope.

MB-54315

Running "advancedGetOpWithCache" returns an incorrect meta.id on the second call when accessing a document twice.

MB-55193

FunctionOverload parser results in false positives when it incorrectly flags function names with reserved function names as their prefix.

MB-56748

Eventing Bucket Backed Cache in the Advanced Accessor couchbase.get() not efficient for very large documents. The final implementation returns small documents with a speed up is 20-25X while large documents are returned with a speed up of 400-500X when using {"cache": true}.

MB-56749

Eventing writes to the wrong keyspace (_default._default) if the collection name is long (over 30 characters).

MB-56750

As of 7.2, the curl() call no longer performs URL encoding. Older functions with a language compatibility of 6.6.2 still work. In some cases, an eventing function created in 7.1 might require a parameter added to the curl() call "url_encode_version": "7.1.0" as the 7.1 release changed the 6.6.2 URL encoding method.

MB-54549

Backup and restore fails when a bucket name contains a "." character.

MB-53646

View engine queries should filter out transaction documents inserted into buckets.

MB-54172

Ensure all data is returned when DCP stream starts in disk-only mode.

MB-54173

Handle json and raw compressed XATTR document types in the View engine.

MB-56321

7.1.0 couchbase-cli may authenticate using either a username or password, or a mTLS (client certificate). The CLI argument validation does not handle the case where no authentication is required. For example, node-init. This results in a false positive where couchbase-cli requires authentication flags to be provided.

Before a node is initialized, use placeholders for the username and password. After the node is initialized, a username and password must be supplied.

MB-56226

Intermittent crashes and errors happen on Full Text Search at query time in term dictionary and postings list. This happens when attempting to access invalid addresses or out-of-bounds data. For example, when term dictionary might be an empty byte slice.

For the errors, a retry logic on the application layer might help. However, there is no workaround for the intermittent crashes which Couchbase are investigating.

MB-56058

The first version of the cost-based optimizer (CBO) focuses on optimizing SPJ (select-project-join) queries, or multi-inner-join subgraphs of such queries. For more complex queries, e.g. queries involving outer joins or complex correlated subqueries, some parts of the query will be handled by CBO and other parts will not be.

As a result, query plans currently displayed for such queries will be missing CBO-provided cost and cardinality estimates, instead showing those values as 0. Such queries will still have valid plans and will execute correctly, but their plans may not be as optimal as desired.

There is no workaround for this display issue.