Release Notes for Couchbase Server 7.6

MB-64353

Hard failover of ephemeral buckets could lead to the rollback of DCP clients back to zero.

Fixed in version 7.6.3.

Upgrading to 7.6.5 will enable the fix.

MB-64742

The system now marks only the latest recovery point that exists in both the recovery log and data log. This change effectively limits the recovery point history list to a single entry in the recovery log. The plasma tracking statistics have been fixed to correctly identify older recovery points as stale data in the recovery log. These improvements allow the log cleaner to run efficiently even at low mutation rates.

MB-64684

Issue resolved

Issue

Description

Resolution

MB-64652

The issue has been resolved in version 7.6.5. Our recommendation is to avoid the issue by moving directly to this version.

MB-64764

If an index was created during rebalance, and the rebalance was stopped before plans were computed for that index, then no plans or partitions were created for the index until a subsequent rebalance was completed.

In 7.6.4 we introduced a change to compute the plans for indexes created during a rebalance even when the rebalance stops midway. This inadvertently caused a regression where a subsequent rebalance could get stuck and fail continuously on timeouts.

In 7.6.5 we reverted the change: if the user aborts an ongoing rebalance, then plans for an index introduced during the operation may not be deployed until a subsequent rebalance operation is triggered and completed.

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-62219

The cluster manager was unable to rotate the root certificate when a new root uses the same private key as the old root.

Issue resolved.

MB-62413

When a username is extracted from a client certificate, the candidate username is checked for its existence in couchbase-server. If such a user doesn’t exist, the algorithm now doesn’t stop but tries to extract another candidate username.

Issue resolved.

MB-63871

The /prometheus_sd_config endpoint provides a new option clusterLabels which species the cluster name and cluster UUID be returned. Prometheus will use them as labels in time series data.

This provides a method to guarantee uniqueness for stats with the same name gathered from multiple clusters.

MB-63261

An issue occurred caused by a race condition in the index recovery code path, which may result in item count mismatch and wrong query results.
Prior to Release 7.6.0, this issue may occur during an Indexer restart. However, as part of the file-based rebalance process introduced in 7.6.0, a recovery of the index is performed after the index is moved, which increases the likelihood that the race condition might be reached.

The race condition has been addressed and the issue is resolved.

MB-62383

In rare situations, a document with a very large CAS could be inserted into a bucket. It was possible for XDCR to replicate this document with malformed CAS to other clusters.

XDCR now has built-in guardrails to prevent documents with malformed CAS beyond a certain drift threshold from being replicated and then to notify the user if this situation occurs via UI and logs.

MB-62410

Connection Pre-check did support all inputs normally accepted by XDCR

XDCR connection pre-check now supports alternate addresses, DNS SRV, and works identically to XDCR remote cluster references.

MB-63738

The XDCR dashboard on Source cluster now shows a UI alert when a pipeline is stuck due to EACCESS errors while writing to Target cluster.

Issue resolved.

MB-62932

The execution of prepared statements containing (*) expressions in the projection can return incorrect results in scenarios where the encoded plans of such statements need to be decoded before execution.

Issue resolved.

MB-63016

If an index is defined with a CASE statement as an index key, and the CASE statement contains a not-equal expression, and the same CASE statement is used in the projection clause of a query, then the covering index scan is not performed properly.

The issue is fixed, and we now use covering index scan appropriately.

MB-63069

In versions 7.6.2, and 7.6.3, queries with a FROM clause containing an UNNEST and ORDER BY with a LIMIT or LIMIT + OFFSET clauses, then in some cases incorrect results were observed when the ORDER could not be pushed down to the indexer.

Issue resolved.

MB-63414

If memory_quota is set to a value and Query uses a non-covered array index, it can result in Request has exceeded memory quota error.

Issue resolved.

MB-64155

From version 7.6.4+, a statement containing text such as: … 1AND … will now correctly generate a syntax error indicating it is an invalid number.
Correct such instances by inserting a space between the numeric constant and the following token, e.g. … 1 AND ….

MB-63518

Issue resolved.

MB-63420

Issue resolved.

MB-63014

In earlier versions of Couchbase, during eventing function deployment or resumption, KV service disruptions (like network partitions) could leave LCB instances in an unrecoverable, unhealthy state. This caused the eventing function to become permanently stuck, with subsequent operations failing to return control, rendering the function non-functional.

This issue has been resolved. Couchbase Eventing now actively monitors LCB instance statuses and implements a "lazy" retry mechanism for bootstrapping unhealthy instances. The retry process continues until the operation times out, which is dynamically determined by the script timeout. This ensures that JavaScript code remains responsive, with mutations timing out gracefully if bootstrap attempts fail, instead of causing a complete function lockup.

MB-61387

To speed up the initial process of building the index, the index service has an optimization that skips checking for existing entries and directly adds new ones. This optimization is crucial for the initial build but should not be used for updates to existing indexes. Unfortunately, in a rare sequence of events, all indexes might be enabled for this optimization, leading to duplicate entries in the storage layer, leading to incorrect results.

The optimization is now only enabled for those indexes that are undergoing an initial build process.

MB-62220

We have an optimization in place which avoids unnecessary index movements when a swap rebalance is performed. A minor bug in this optimization interfered with rebuilding user-dropped replicas/lost replicas during such a rebalance where we didn’t consider new nodes coming in when placing lost indexes.

Adds a sorted pseudo-random order where we give higher priority to new incoming nodes followed by old nodes staying in the cluster when we try to place lost indexes. Hence, we can repair lost replicas.

MB-62919

After enabling GSI shard-based rebalance from the Web UI and saving the settings, it was not possible to disable the same setting by unticking the box. The box is greyed out and cannot be interacted with.

Issue resolved.

MB-63276

An issue in the rebalancer caused several partitions to miss streaming mutations when partitioned indexes were moved during file-based rebalances. This issue resulted in corrupt indexes as not all data was processed.
The issue occurred only with partitioned indexes with file-based rebalances enabled.

Issue resolved.

MB-62427

In summary, version 7.6.4 now supports upgrades of 6.x index definitions. However, Couchbase recommends upgrading indexes to 15+ segment version format for improvements in indexing footprint and performance.
Issue resolved.

MB-61013

The Backup service incorrectly allowed backups to be created in the data directory because the server could delete the backup files.

The server will not allow archives to be created in the data directory.

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-63070

An issue occurred where cbcollect_info did not process some <ud>…</ud> tags, especially when they were present at the end of the file or inside binary data. As a result, some logs that should have been redacted were not redacted.

Issue resolved.

MB-62984

When a Couchbase Data Service experienced a hard failover or crashed with data loss, DCP clients (like those used with Elasticsearch and Kafka) sometimes received incomplete or partial snapshots of the data. This incomplete data sometimes leads to rare situations where the Data Service crashed repeatedly.

Issue resolved.

MB-63261

An issue caused by a race condition in the index recovery code path sometimes resulted in an item count mismatch and wrong query results. Prior to Release 7.6.0, this issue sometimes occurred during Indexer restart. However, as part of the file-based rebalance process introduced in 7.6.0, a recovery of the index is performed after the index is moved, which increases the likelihood that this race condition might be reached.

Issue resolved.

MB-63024

Issue resolved.

MB-63060

An issue occurred when migrating nested UDFs to any 7.6.x release from a previous version. In cases when an inline UDF called another UDF, query nodes became unresponsive.

Issue resolved.

MB-63078

Issue resolved.

MB-63147

An issue occurred with user-defined functions (UDFs) that contained queries defined with a common-table expression (CTE). If the CTE contained references to the UDF arguments, the execution of the UDF returned an error indicating “correlation reference <with_alias> is not allowed.”

Issue resolved.

MB-63193

An issue in the rebalancer code caused several partitions to miss streaming mutations when partitioned indexes were moved during file-based rebalance. This issue resulted in corrupt indexes, as not all data will be processed. The issue occurred only with partitioned indexes when file-based rebalance was enabled.

Issue resolved.

MB-62923

An issue with the HTTP client lifecycle resulted in leaked TCP connections between the Analytics service processes during internal credentials rotation, eventually leading to ephemeral port exhaustion.

Issue resolved.

MB-62949

When a query encounters a failure, the details of the exception can get masked by subsequent exceptions, leading to the loss of the root cause of the query failure.

Issue resolved.

MB-63414

If memory_quota is set to a value and Query uses a non-covered array index, it can result in Request has exceeded memory quota error.

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-62220

Dropped replicas are not rebuilt during swap rebalance

Drop and then recreate the indexes.

MB-60621

The sys_cpu_cgroup_seconds_total{mode="usage"} stat also included the amount of time for sys_cpu_cgroup_seconds_total{mode="user"} and sys_cpu_cgroup_seconds_total{mode="sys"}. As a result, using certain functions e.g. sum(sys_cgroup_seconds_total) would double the user count and sys time.

To prevent this, the sys_cpu_cgroup_seconds_total{mode="usage"} is no longer returned and is replaced with sys_cpu_cgroup_usage_seconds_total.

MB-60883

The metric now shows the ratio between the number of read operations which failed due to the key not being present, and all read operations:

kv_ops{op="get", result="miss"} / kv_ops{op="get"}.

MB-60879

HashTable resizing can be avoided by setting the minimum size to be large enough, but there was no metric to indicate what that should be.

Added kv_vb_ht_avg_size and kv_vb_ht_max_size which summarizes the current HT sizes across vBuckets.

MB-61525

With the removal of the dedicated DCP consumer buffer in 7.6.0 in a temporary out-of-memory scenario, a consumer node could continue to consume DCP mutations and add to the checkpoint beyond the allocated checkpoint memory quota. This is limited by the dcp consumer buffer quota.

A new metric has been added to track the sum of the checkpoint quota and the DCP consumer buffer quota. This is used to make sure the memcached memory allocation on a consumer node isn’t exceeding the allocated memory quotas:
checkpoint_mem-usage limit = checkpoint_memory_ratio + dcp_consumer_buffer_ratio.

MB-62547

A critical issue occurs when performing an off-line-upgrade, graceful failover, or delta-recovery upgrade methods on index service nodes from Couchbase Server versions 7.1 through 7.2 to versions 7.6.0 or 7.6.1. The upgrade corrupted existing indexes requiring you to drop and rebuild them.

Issue resolved.

MB-61014

The planner may not pick a covering index if the index is defined with a complex WHERE clause (e.g. AND or OR clauses involving !=, NOT or IN predicates).

Issue resolved.

MB-61171

In rare circumstances with high-load Query, requests can fail to complete, blocking rebalance attempts.
Such requests don’t affect other regular processing beyond occupying a servicer, but a rebalance can’t complete until they do. By the time it is an issue, it is likely that the client that submitted such a request has already terminated, and the request is just stuck in Query processing. Deleting such a request from system:active_requests using the REST API may release it.
Forcibly restarting the Query service will clear the issue.

Issue resolved.

MB-61564

Incorrect return code when attempting to create an index that already exists. The system returns the error: 200: index already exists.

Issue resolved: the system will now return: 409: index already exists.

MB-61764

In rare cases, Couchbase Server could report an IndexOutOfBoundsException error if a SQL++ query contained a subquery with an IN clause.

Issue resolved.

MB-61488

When performing a Sub-Document operation using the Eventing couchbase.mutateIn Sub-Document API, the operation sometimes caused exceptions in the JavaScript code and failed with the error code LCB_ERR_SUBDOC_XATTR_UNKNOWN_MACRO.

Issue resolved.

MB-50944

Eventing did not properly support read-write bindings to collections with Sync Gateway attached at the source. Adding a new Eventing function failed if the bucket alias was set to read-write.

Issue resolved.

MB-61387

To speed up the initial process of building the index, the index service has an optimization that skips checking for existing entries and directly adds new ones. This optimization is crucial for the initial build but should not be used for updates to existing indexes. Unfortunately, in a rare sequence of events, all indexes might be accidentally enabled for this optimization, leading to duplicate entries in the storage layer and causing incorrect results.

Optimization is only enabled for those indexes that are undergoing the initial build process.

MB-61793

The indexes made using the plasma storage engine have both in-memory and on-disk components. There are some components which are always present in memory and are never evicted to disk, so they consume the same memory even at varying resident ratios.
For any rebalance or index planning calculation, the memory usage of all indexes at the recommended resident ratio is estimated and used. During these estimations, the memory taken by the fixed in-memory component was also scaled up with the respective resident ratio, which caused overestimation. This overestimation is only evident at very low resident ratios and could sometimes cause rebalance failure.

Now, a more accurate calculation is made to avoid overestimating the memory of the indexes.

MB-62199

During restore, the index planning operation adds replicas for lost replicas of indexes in the plan.

If multiple indexes exist with the same name, there are lost replicas, and there are not enough indexer nodes to hold all the index replicas in the plan, then extra replicas will not be removed from the plan, and can remain on the old node.
This causes restore operation failures.

Issue resolved.

MB-60719

Running a query with score:none results in response containing score:0. The score is incorrectly added to the response.

If a user runs a query with score:none then the query response will no longer contain score:0.

MB-61043

In scenarios where a rebalance is followed by a failover, the partitions are not evenly distributed across all nodes, causing a skewness.

Skewness has been resolved.

MB-61310

During rebalance, when moving partitions around, we track the progress of movement and then check the seq numbers the partition has caught up relative to the view of the partition on source node and also the KV’s view.
This progress monitoring procedure was only for active partitions

You can now optionally monitor the replicas as well

MB-61654

Prometheus fails to scrape the new xattrs fields)

Problem caused by the use of *num_vectors which uses Prometheus-reserved character; num_vectors will no longer show up in the stats.

MB-60630

Moving a cloud backup archive in a normal GCP bucket to a locked GCP bucket (which allows creating new files but prohibits modifying or deleting pre-existing objects) and then performing a restore from that bucket.
The restore didn’t fail, which would be the expected behavior; instead, the restore hangs.

The problem occurs because cbbackupmgr always retries on 403s responses when using a GCP client, since it considered them intermittent.

cbbackupmgr no longer considers 403s as temporary errors, and will not always retry when receiving them.

MB-61630

Previously it was not possible to import an encrypted backup repository into the backup service as we did not accept the KMS parameters.

Both the UI and REST API now allow users to specify the KMS and its authentication parameters so an encrypted repository can successfully be imported.

MB-61631

Previously, passing a relative path to cbbackupmgr as --obj-staging-dir, the backup or restore would fail with an empty object name.

Issue resolved.

MB-63414

If memory_quota is set to a value and Query uses a non-covered array index, it can result in Request has exceeded memory quota error.

Disable memory quota or contact support for alternatives.

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-62220

Dropped replicas are not rebuilt during swap rebalance

Drop and then recreate the indexes.

MB-61154

When bucket data exceeded 4 TB and Magma was used as the storage engine, rebalance sometimes hung and failed to run to completion.

Issue resolved.

MB-61076

Scheduled merges – that is merges which are done due to a task in a plan – will always fail.
The system will not perform the merge, and it will leave the backups that should have been merged in place, ensuring no data is lost.

You can perform manual merges through the UI, or using the API.

MB-62547

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-60719

Older SDKs might have failed operations when you access the Search Service with the disableScoring option set to false. This is a breaking change due to a change in the response payload.

Set the disableScoring option in SDKs to true.

MB-60568

CPU utilization rate may be incorrect in a VM.
When running a VM on a XEN hypervisor, stats were computed using the number of processors configured for the entire system.

MB-58088

When the client sends an unlock request for a document that is not locked, the server returns a "temp fail" error code. This causes the client to retry until the operation times out.

The server returns a new "not locked" error code.

MB-59060

When the client sends an unlock request for a document that is not cached in Full Eviction mode, the server returns a "temp fail" error code, even if the document is stored on disk. This causes the client to retry until the operation times out.

The metadata of locked documents are kept in the hashtable. The server returns a new "not locked" error code if the document exists, "not found" otherwise.

MB-58671

Erroneous network conditions could lead to checkpoint manager slow at stopping.

Ensure the checkpoint manager stops, even if it has trouble starting due to network issues.

MB-59233

In low-priority replications, it is possible for a Data Service stream to end temporarily and for XDCR to not handle it correctly. This will lead to a hung replication.

Now, XDCR restarts the pipeline if a data service stream ends.
This ensures replication continues.

MB-59320

Race condition in starting and stopping XDCR source nozzle could lead to a memory leak

Fixed the race condition

MB-59416

If the Bandwidth throttler is used, race condition may occur during pipeline shutdown where the Out nozzle is unable to exit

Fix race condition during shutdown to ensure out nozzle closes properly.

MB-59499

In a slow running backfill replication, XDCR could be too aggressive in restarting pipelines.

Ensure XDCR does not restart backfill pipelines if some progress is observed periodically.

MB-59669

If a replication is idle without mutations, XDCR is unable to detect that a target bucket failover occurred.

Ensure target-side failover detection takes place, even if the source has no incoming mutations

MB-59745

When a target document is locked, and a non-optimistic LWW replication is taking place, XDCR will retrieve a "locked CAS" of maxUint. This will cause the source mutation to lose, and lead to scenarios where the mutation is not replicated, even if it should have won conflict resolution.

XDCR will retry conflict resolution for the duration that the document is locked in pessimistic replication. This will ensure that a valid CAS is used for source-side conflict resolution.

MB-39484

SQL++ will not support operations on legacy buckets without password specification.

Users must authenticate first to connect to the query service and execute any SQL statement.

MB-58648

Observed a memory leak with multiple executions of the same UDF function.

Fix the condition whereby UDF functions are being re-loaded into cache from storage every time the function is executed.

MB-59501

The system catalog allows users to see items without RBAC authentication or authorization.

Valid RBAC permissions are required to query the system catalog, and to view items stored in the catalog.

MB-59138

The system did not support nested flattened array indexes when an entry was missing in nested arrays.

The system now correctly expands null or missing entries for nested arrays.

MB-57657

This restriction has been lifted in 7.6.0.
The user will now be able to run queries via SQL++ without having to run the keyword analyzer.

MB-59858

When a Search index name is too long, the index silently fails to ingest documents.

The UI will now flag instances where the chosen index name is too long.

MB-60718

Index alias queries not returning cumulative (duplicate) results from its targets.

The fix prevents cyclic lockups within aliased index targets (aliases with the same targets pointing to each other to an infinite depth).
The service also de-duplicates index targets.

MB-57988

cli should allow modifying existing collection’s maxTTL

The Couchbase CLI has been extended to allow the maxTTL (maximum time-to-live) to be modified for a collection.

NCBC-3724

Versions of the .NET SDK earlier than 3.5.1 have compatibility issues with Couchbase Server 7.6.

Use version 3.5.1 or later of the .NET SDK with Couchbase Server 7.6.

MB-59352

When the Load Metadata from File option is selected, the Couchbase Server UI does not disable two options that can’t be used: Validate metadata using trusted fingerprints and Verify Remote Peer. These two options are irrelevant during metadata upload. Selecting them will have no impact on the process.

NA

MB-60062

When the auto-failover timeout setting is set to fewer than 5 seconds (the recommended minimum), you can no longer modify any cluster settings using the Couchbase Server UI.

Modify settings using the Nodes and Clusters REST API. For more information on the auto-failover settings, see the documentation.

MB-61076

Scheduled merges (i.e. merges that are performed in a task in a plan) do not run. Note that backups scheduled for the merge are left in place so no data is lost.

Merge backups manually using the UI or using the API.

MB-61154

In situations where bucket data exceeds 4 TB and Magma is being used as the storage engine, it is possible for rebalance to hang and fail to run to completion.

NA

MB-62547

MB-64966

Scope level user-defined functions (UDFs), sequences, and histogram data are stored in the bucket/_system/_query collection.

FLUSHing the bucket will delete the scoped UDF entries. When the UDFs are executed, it will work until the UDFs are evicted from the cache. After that, the execution will always result in an error.

Sequences metadata is also stored in the bucket/_system/_query collection. FLUSHing the bucket will delete the entries. When executed, sequences will return errors once evicted from the cache.

FLUSHing the bucket will also delete statistics data stored in the bucket/_system/_query collection. By using stale statistics stored in the cache, a non-optimal plan could be generated.

MB-60719

Older SDKs might have failed operations when you access the Search Service with the disableScoring option set to false. This is a breaking change due to a change in the response payload.

Set the disableScoring option in SDKs to true.