Release Notes for Couchbase Server 6.0

MB-36177

Summary: Fixed an issue so that data files are recreated on ALL keep nodes when failing over datasets and not just the intersection of keep nodes and nodes where the datasets were previously located.

MB-35940

Summary: Added the ability to filter datasets based on document keys and not just fields.

MB-35688

Summary: A cancelled rebalance-in left an Analytics cluster without any metadata replicas. Upon attempt to failover and remove the remaining node, the cluster became permanently unusable unless you removed all Analytics nodes from the cluster and re-added the service, which resulted in an uninitialized Analytics cluster. This has been fixed so that metadata replica addition is not skipped on a cancelled rebalance-in.

MB-34798

Summary: During node startup, the system latest checkpoint files are read to determine the system state and complete the startup based on that state. If all the checkpoint files found were corrupted, then the node was bootstrapped as a new node and as a result, all data was deleted. This has been fixed to distinguish between a real bootstrap scenario (where there are no checkpoint files) and a scenario where the checkpoint files could not be read successfully, and in the latter case, force recovery from the start of the available transaction logs.

MB-35748

Summary: Fixed a race condition around trap_exit.

MB-35807

Summary: XDCR has been enhanced to be aware of target nodes' setups and only restart pipelines if the target and remote cluster reference are both set up for mixed mode encryption, and the target node’s mode really changed.

MB-34298

Summary: Fixed an issue so that the topology change detector does not delete the replication spec when a target bucket is not found.

MB-36101

Summary: Fixed an issue where DeleteWithMetadata against an XATTR document could result in a mutation.

MB-36087

Summary: Fixed an issue when DeleteWithMeta against a value evicted XATTR document crashed kv-engine.

MB-35702

Summary: SSL write failures may not reset the connection in the write phase. This has been fixed by propagating SSL write errors from sendmsg.

MB-35195

Summary: Fixed an issue where the audit config reload was stuck due to the drive (or path) being unavailable.

MB-34879

Summary: An error was thrown when performing a GET on a document that is both compressed and has extended attributes(XATTRs) if the client supported compression and the uncompressed document is sent to the client with the compressed flag still set. This has been fixed.

MB-35734

Summary: In rebalance_service_manager, the default value of the isBalanced flag is set to false. If the index service restarts after rebalance is complete, this flag will be set to false and the rebalance button is activated assuming that the index service requires a rebalance. This is not expected behaviour. This has been fixed by setting the default value of isBalanced to true.

MB-35700

Summary: The collatejson package’s JoinArray has been optimized to mitigate high memory usage.

MB-35477

Summary: Fixed a rare issue where the Plasma storage log grew indefinitely.

MB-35473

Summary: Fixed an issue that caused a Write Bucket Failure and corrupted the vBucket file.

MB-35323

Summary: Fixed an issue in ForestDB where renaming a compacted file led to the file getting deleted.

MB-34428

Summary: When querying on a ddoc that is deleted, the views query incorrectly returned HTTP error code 500 suggesting an internal sever error. This has been fixed and the views query now returns an error message suggesting that the ddoc has been deleted.

MB-34158

Summary: Fixed a memory allocation failure in Forestdb that caused a system crash under pressure.

MB-32341

Summary: Added the ability to trace operations on a file.

MB-35811

Summary: Fixed a connection pool leak observed when querying system:indexes.

MB-35670

Summary: When scanning through all n1ql nodes to perform a distributed command, the query engine memory grew until it crashed. This has been fixed.

MB-35516

Summary: COUNT on an index FullScan returned incorrect results when it included DESC order. This has been fixed by updating the code to handle Inclusion getting flipped due to the DESC order.

MB-35392

Summary: The query service returned incorrect results as the implict covering of ANY predicate did not handle any part that was not covered. This has been fixed.

MB-34750, MB-34665

Summary: Avoid IntersectScan when the index is a subset.

MB-34698

Summary: Improved performance for count aggregate on full index scan by using CountTotal.

MB-34667

Summary: Better identifier handling to determine whether an identifier should be covered or not.

MB-34660

Summary: Fixed an issue where the implicit cover of nested array did not work as expected with arbitrary variables.

MB-34616

Summary: When schema inferencing is run, document field names are no longer included in the log file.

MB-34544

Summary: The query service now generates a special error for subquery build failures and sets the HTTP error code appropriately.

MB-34387

Summary: Fixed an issue where the query engine returned incorrect results when using a partition index and the ORDER BY clause.

MB-35840

Summary: Fixed the UI to display View index build progress status on the Views page.

MB-35130

Summary: The couchbase-cli user-manage help has been corrected to note the valid role query_manage_index.

MB-34224

Summary: The couchbase-cli user-manage help has been updated with the correct information on adding RBAC roles for multiple buckets.

MB-31517

Summary: Fixed an issue where a cbbackup accumulative backup always performed a full backup after the first accumulative backup.

MB-33696

Summary: The analytics service on one node failed to startup repeatedly due to a failure while performing a rollback and threw an exception. This has been fixed.

MB-33595

Summary: Variables bound by a QuantifiedExpression were incorrectly included when computing free variables for that expression. This has been fixed and these variables will now be excluded from the free variable set.

MB-33564

Summary: Fixed an issue where the predicate in the WHERE clause was incorrectly pushed through the running aggregate operator (AT subclause), which then lead to wrong results.

MB-33338

Summary: It was observed that a rebalance in of new analytics nodes hangs indefinitely on a firewall misconfiguration where newly added nodes were unable to contact the CC. This has been fixed so that rebalance fails if the Keep-Nodes do not join within a reasonable timeframe (2 minutes).

MB-33025

Summary: Fixed an issue where the analytics HTTP server incorrectly assumed that all request payloads are UTF-8 encoded. The analytics HTTP responses are encoded using system-default encoding and the service will now attempt to honor any requested encodings (Accept-Charset), or default to UTF-8.

MB-33016

Summary: Fixed an internal compiler error caused by index selection when the search expression was not constant.

MB-32777

Summary: Fixed an issue where array functions, most notably the array_intersect() function, caused an out of memory exception due to object creations, especially for large datasets.

MB-32775

Summary: Fixed an issue to catch all unexpected exceptions on network threads and close the connection in such cases to avoid attempting to process the message causing the exception forever.

MB-32745

Summary: The analytics service now uses the Couchbase temp directory, which is set to /var/lib/couchbase/tmp by default, and no longer requires access to the operating system’s temp directory.

MB-33970

Summary: By default, XDCR will prevent replications from being created on a source ephemeral bucket with NRU policy. This specific release of Couchbase Server includes a special XDCR internal settings flag, AllowSourceNRUCreation, that allows replications to be created on a source ephemeral bucket with NRU policy.

---- curl -X POST -u Administrator:password http://127.0.0.1:9998/xdcr/internalSettings -d AllowSourceNRUCreation="true" ----

WARNING: Changing the internal replication setting will cause all XDCR processes to restart. This causes temporary disruption as all existing replications will have to restart.

MB-33671

Summary: Following an issue on a remote cluster, XDCR failed to recover and threw an exception.

MB-33582

Summary: After upgrading to 6.0.x, it was observed that the compression type field in the UI was not populated automatically. This has been fixed.

MB-32814

Summary: Fixed REST endpoint error handling for settings errors and nil value types.

MB-33750

Summary: Fixed a race condition that was exposed when a supervisor that is shutting down normally, is for some reason brutally shutdown leaving an orphan child process. This rogue lease acquirer interfered with the operation of the orchestrator causing, for instance, rebalance to fail.

MB-33321

Summary: Fixed an issue to prevent map corruption due to a change in num_replicas after an interrupted rebalance operation.

MB-34507

Summary: Fixed an issue where documents with a TTL, containing system XATTRs, and which are compressed, may have their datatype metadata incorrectly set. This prevents any subsequent modification of the XATTRs on the deleted document, until the document is re-created.

MB-34262

Summary: Follow-on to MB-33919. Fixed issue where erroneous tombstone delete times could get used when performing a DCP backfill on ephemeral buckets.

MB-34173

Summary: Fixed an issue where recovery failed due to invalid snapshot start (or end) on disk.

MB-33919

Summary: The tombstone delete-time could be in the future or in the far past, causing the tombstone to be purged immediately or to remain on the disk for potentially long time. This has been fixed and the tombstone delete-time is now set to ep_real_time at the point of deletion.

MB-33918

Summary: Fixed excessive triggering cursor dropping issue which caused Couchbase Server to attempt to recover more memory than required.

MB-33906

Summary: Fixed an issue where the response to a getReplica command could be significantly delayed.

MB-33854

Summary: Fixed an issue where the KV-Engine may incorrectly return ENGINE NOT_STORED to the client after trying to perform arithmetic operations if another client concurrently modified the same key.

MB-33773

Summary: Fixed an issue where the KV-Engine may crash due to a race between DCP Consumer message processing and stream shutdown.

MB-33351

Summary: Fixed an issue where the breakpad did not catch the memcached segfault.

MB-32840

Summary: Fixed an issue where the audit daemon asserted when rotating an empty audit log causing a crash.

MB-32687

Summary: Fixed an issue where signalling a connection could result in deadlock.

MB-32685

Summary: Introduced a delay in updating the RBAC database revision number to avoid returning AUTH_STALE during a refresh of the RBAC database.

MB-15009

Summary: Improve the effectiveness of the active defragmenter to also defragment Document metadata (StoredValue objects) in addition to document values. This should reduce memory fragmentation for workloads which have multiple document metadata sizes (i.e. a range of key lengths).

MB-33085

Summary: The cbevent tool provides a way to recover the eventing service when eventing specific metakv contents and metadata bucket contents were corrupted or lost. This tool clears all the Functions (deployed and undeployed) and restores Eventing service to a clean state. To use the tool, run the following command: $ cbevent -<host> -<username> -<password> -flush

MB-33158

Summary: Fixed an erratic crash issue observed with the use of queries like facets or sort on a custom field.

MB-34430

Summary: Fixed a race condition on feed cleanup that caused the projector to panic.

MB-34405

Summary: Fixed a memory leak caused by snapshot metadata objects linked list references.

MB-34385

Summary: Fixed an issue where the admin console incorrectly displayed the node UUID instead of name in index definition after upgrade.

MB-34168

Summary: Fixed a race condition while handling an error during move index.

MB-34166

Summary: Fixed an issue where new index builds remained stuck if indexes on MAINT_STREAM were dropped before indexes on INIT_STREAM during rebalance.

MB-34164

Summary: Fixed an issue where the index node crashed with too many open files.

MB-34003

Summary: Index created failed as the index service was unable to create index replicas. This has been fixed.

MB-33945

Summary: Fixed an issue where the Views engine returns a server error when bucket data is compressed and doc is not used in the view definition.

MB-33666

Summary: It was possible to create an index with fewer replicas than the default num_replicas if you used nodes_list in the WITH clause with fewer nodes than the default number of replicas.

MB-33654

Summary: Fixed an issue where frequent processing of old create tokens caused high CPU usage on index nodes.

MB-33649

Summary: Fixed an issue where the projector took a long time to process admin requests if data flow was blocked.

MB-33640

Summary: Fixed an issue to terminate all watcher goroutines upon close of MetadataProvider.

MB-33497

Summary: The index service crashed during index creation if unicode special characters were used. This has been fixed.

MB-32932

Summary: For non-partitioned indexes, the planner did not always use all the indexer nodes in cluster for planning. This has been fixed.

MB-32824

Summary: Fixed an issue where the index planner could end up assigning the same instance id to lost replicas when more than 1 replica was lost.

MB-32748

Summary: Sometimes, the Windows uninstaller left behind registry entries and subsequently this blocked upgrades on these machines. This has been fixed.

MB-34136

Summary: Fixed an issue where the query engine returned incorrect results when using a partition index and the ORDER BY clause.

MB-33827

Summary: Fixed an issue so that partition UNNEST scans do not use DISTINCT index aggregation.

MB-33757

Summary: Queries with multiple array indexes caused inconsistent query plans, which then led to inconsistent and sometimes incorrect results. This has been fixed.

MB-33208, MB-33185

Summary: Fixed an issue with connection pool leak that caused queries to fail.

MB-33764

Summary: Fixed an issue where changing cluster CA and node certificates and incorrectly needed separate RBAC roles.

MB-33040

Summary: The builtin Administrator user can now be authenticated via a client certificate.

MB-34126

Summary: The couchbase-cli tool no longer enforces the 24 character password limit.

MB-33429

Summary: The couchbase-cli setting-alert man page has been updated to include the alert-communication-issue flag.

MB-33204

Summary: The cbbackupmgr backup --resume command incorrectly required full administrator privileges. This has been fixed so that any user with back and restore permissions can also resume a backup.

MB-32671

Summary: The cbimport key generator did not handle escaped hash(#) or percent(%) characters correctly. This has been fixed.

MB-32466

Summary: Index creation on meta fields is not allowed and trying to do so will result in an error.

MB-32455

Summary: Fixed an issue where rebalance failed to complete with a badmatch error.

MB-32435

Summary: Fixed an issue where a rebalance operation could fail due to the reuse of dataset IDs internally.

MB-32079

Summary: When a dataverse was dropped, bucket listeners were not unregistered. Consequently, when the dataverse was re-created, the CONNECT LINK statement failed as it picked up the old bucket listeners. This has been fixed.

MB-32073

Summary: Fixed an issue where the AnalyticsReader role, a read-only role, was incorrectly able to change the metadata and drop dataverses.

MB-31814

Summary: Starting this release, the OpenJDK version 11 is bundled with the service. See Additional Requirements for details.

MB-32299

Summary: XDCR failed to replicate files larger than the network usage limit. This has been fixed so the bandwidth throttler can handle large documents with sizes greater than the usage limit.

MB-32221

Summary: Uncompressed replication from 6.0 nodes to 5.0 nodes failed due to partial information being returned from the parts. This has been fixed to enable accurate detection of compression error code.

MB-32220

Summary: Fixed an issue where replication stops and throws an exception due to a nil error.

MB-32219

Summary: Fixed an issue where the XDCR log level setting was not retained when upgrading Couchbase Server from version 4.x to newer version.

MB-32042

Summary: Fixed an issue where encrypted XDCR was incorrectly using port 8091.

MB-31857

Summary: An error was thrown when DNS alternate addresses were used. This has been fixed so utilities return the hostname even if they can’t find the port numbers.

MB-31764

Summary: The unit of interval for XmemSelfMonitorInterval was incorrectly set to millisecond instead of second, causing an increase in the frequency of Xmem self monitor and the spurious Xmem is stuck error. This has been fixed.

MB-32669

Summary: Fixed an issue where a chain of events, active compression of an extended attribute value followed by eviction and then expiry, triggered an exception.

MB-32364

Summary: Sub-document API single-path mutation ignored expiry if doc_flags included extras.

MB-32181

Summary: If one or more Ephemeral buckets were present, then nonIO background tasks could be incorrectly scheduled, potentially resulting in DCP connection instability (premature disconnection). This has been fixed.

MB-32136

Summary: When data corruption is detected on a data service node, relevant information that can help troubleshoot is now being logged.

MB-32135

Summary: In some cases, the memory allocation failures were not handled causing "Write Commit Failure" errors or corruption in vBucket files.

MB-32078

Summary: If an append/prepend request contended with another request at the Server, instead of automatically retrying on the Server (as expected), it returned EEXISTs back to the application.

MB-31968

Summary: Audit logging has been updated to capture the user name when documents are created, updated, or deleted from the web console.

MB-31967

Summary: Fixed an issue where the datatype field when using the DCP_OPEN flags incorrectly reflected the contents of the body payload and not the original datatype.

MB-31410

Summary: If bucket memory usage exceeded the high watermark and reached 99% of the bucket quota, a race condition in KV-Engine DCP consumer was exposed. This could result in DCP messages being processed out-of-order.

MB-31924

Summary: Reduced excessive logging by Eventing service when the debugger is turned on.

MB-31923

Summary: Following the failover of a vBucket, the Eventing service restarted streams with seqno from the latest failover log entry instead of using the entry where seqnoInFailoverLog ⇐ lastProcessedSeqno. This resulted in missed events.

MB-31922

Summary: Fixed an issue where some events were missing after a source bucket flush operation.

MB-32717

Summary: Fixed an issue where the index service exceeded the memory quota even though only the number of items that were indexed was low.

MB-32507

Summary: In a cluster with duplicate index names across buckets, replica repair failed after a node was added back to the cluster. This has been fixed.

MB-32433

Summary: Plasma memory tuner will now increment indexer memory quota only if the indexer process' RSS is below the memory quota.

MB-32239

Summary: The DCP noop interval has been lowered to 20sec to improve the chances of the projector receiving the noop message early and thus avoiding frequent roll backs.

MB-32228

Summary: During warmup, one of the indexes with low number of items triggered an issue and caused the internal garbage collector to run forever. Thus causing the entire index service to stall during warmup. This issue was observed when using Plasma storage engine.

MB-31989

Summary: Improved the DCP rollback handling mechanism. Instead of ignoring the disk snapshots before rolling back, the index service will now exhaust trying DCP stream requests with all the disk snapshots before rolling back to 0.

MB-31744

Summary: An error is seen during a SUM aggregate pushdown when the entry value is greater than MaxInt64.

MB-31724

Summary: Fixed an issue to enable creating secondary indexes on binary documents on meta.id(), meta.cas(), and meta.expiration().

MB-31587

Summary: The index service would crash under some conditions if the data bucket was flushed while the index was being built.

MB-32366

Summary: Starting this release, you can use the UNNEST scan to generate index spans on non-leading keys (non-array-index keys) if appropriate predicates exist on those keys.

MB-32306

Summary: Index creation using the IN clause did not work as expected. This has been fixed.

However, when using the IN clause, ensure that your queries use the same IN terms in the same order as in the index.

MB-32301

Summary: Fixed an issue so that when an on connection error is encountered, certain operations are retried before returning an error.

MB-32118

Summary: Fixed an issue so that connections are properly discarded when an Out of Bound data error is encountered.

MB-32142

Summary: Fixed an issue where the Views UI was unable to process the "undefined" value in a binary JSON document.

MB-31831

Summary: The cbbackupmgr utility would hang and not return an error when trying to backup a cluster in which at least one node was down and had not failed over.

MB-31558

Summary: All requests originating from the UI were authenticated using token based authentication, even when client certificate authentication is enabled on the Server and a client certificate is configured in the browser. This has been fixed to use client certificates for authentication in such scenarios.

MB-23074

Summary: Performance issues may be observed when running Couchbase Server on CentOS 7.3 with kernel 3.10.0-514.6.

MB-30637

Summary: Backup and restore for the Analytics service artifacts using the cbbackupmgr utility is currently not supported.

MB-30007

Summary: It is possible for a CONNECT BUCKET statement to fail temporarily with the error message "The vbucket belongs to another server(0x7)" when rebalancing Data nodes.

Workaround: Retry the operation.

MB-29724

Summary: Currently, the Analytics service supports index creation only when a dataset is disconnected.

MB-28544

Summary: The UNION ALL operation does not work for all query types. For example, a query of the form shown below is not parsed correctly.

---- (SELECT …​ FROM …​ WHERE …​) UNION ALL (SELECT …​ FROM …​ WHERE …​) ----

Workaround: Remove the parentheses around the first subquery block.

---- SELECT …​ FROM …​ WHERE …​ UNION ALL (SELECT …​ FROM …​ WHERE …​) ----

MB-31410

Summary: If bucket memory usage exceeds the high watermark and reaches 99% of the bucket quota, a race condition in KV-Engine DCP consumer is exposed. This may result in DCP messages being processed out-of-order.

MB-30074

Summary: If the memcached process crashes while indexes are being rebalanced, the memcached process can fail to be brought online and the rebalance can get stuck. Manually stopping rebalance will bring the memcached process online.

MB-29809

Summary: The last_modified attribute can be incorrect if a bucket was upgraded and contains old documents.

MB-30772

Summary: Killing an Eventing process externally may result in a loss of timers that have been dequeued but not run to completion.

MB-29360

Summary: When Sync Gateway is deployed against a bucket, a single document update can be seen over DCP as multiple mutations, some modifying body and others modifying XATTRs. The Eventing service invokes OnUpdate for each such mutation as it does not distinguish at the DCP level if an update modified the body of a document, the XATTRs, or both.

Workaround: When both Eventing service and Sync Gateway are deployed against the same bucket, you must ensure that the Eventing code written is idempotent.

MB-31660

Summary: The Search index file deletions take longer time than expected especially when the index is very large and is in a building-up phase while deleting. This delays the expected freeing up of the actual disk space right after an index deletion. No file leaks have been observed yet.

Workaround: If you wait long enough, all the index files will get cleaned up eventually. Hence in a tight disk space situation, you should be careful of this fact while performing subsequent index deletions and creations. With enough disk space configured, everything ought to work without much perceivable problems.

MB-31405

Summary: This problem becomes a concern when the system is under heavy DGM (Disk Greater than Memory) condition. In such cases, the Search service ends up having too many files in the system and due to concurrent file compactions, the overall indexing and querying becomes very slow.

Workaround: While there is no proper workaround for this issue, lowering the FTS memory quota (< 10GB) seems to help maintain a reasonably high average indexing throughput.

MB-27429

Summary: Scorch indexes were found to contain duplicate pindexes.

MB-31039

Summary: A disk usage spike is observed during the initial index build. The disk usage comes down once the log cleaner catches up.

MB-30011

Summary: The rebalance progress (in percentage) during a GSI swap rebalance does not always increase linearly. Note that this issue does not persist and the percentage displayed returns to normal quickly.

MB-19869

Summary: Rebalance fails when taking out failed over nodes running views, in certain circumstances.

MB-31393

Summary: In some instances, the Windows Uninstaller can leave behind registry entries and subsequently block upgrades on these machines.

Workaround: Remove the registry entries manually before installing newer versions of Couchbase Server.

MB-17571

Summary: On an undersized node, the default memory quota assigned to all the selected services by the server might result in a failure.

Workaround: Manually adjust the memory allocations appropriately.

MB-29391, MB-29393

Summary: Large numeric values are reported incorrectly. For example, the modulo operation or a scan for min int64 value returns an incorrect result in case of int64.

MB-31558

Summary: All requests originating from the UI are currently authenticated using token based authentication, even when client certificate authentication is enabled on the Server and a client certificate is configured in the browser.

MB-26421

Summary: The Security > Users tab in the UI does not list the default "administrator" user.

MB-31680

Summary: Unable to log in to the Couchbase Server Web Console (UI) using Internet Explorer 11 or Edge browsers.

MB-31619

Summary: The Memcached logs always use Z (UTC) for timezone when logging instead of the current system timezone offset.

MB-31495

Summary: Fetching a random key from the Data service may hang if the bucket contains zero documents.

MB-31481

Summary: The data service engine may not send STREAM_END message to consumers if non-infinity end sequence number and cursor dropping occurs. This could cause clients such as cbbackupmgr to hang indefinitely.

MB-31175

Summary: Ephemeral buckets can have tombstones purged before the configured metadata purge interval.

Ephemeral buckets can prematurely purge (discard) tombstones when under high memory and op/s load. As a consequence, DCP consumers fed by KV-Engine (such as replication, secondary indexing, views) may have tombstones not sent to them, meaning their dataset diverges from the Data Service - documents are not deleted from the DCP consumer.

MB-30920

Authenticated users with full administrative privileges were able to access the diagnostic endpoint, /diag/eval remotely, allowing these users to run arbitrary code on the server host. This issue is addressed by only allowing full administrators to access this endpoint from localhost. This mitigates the issue as users that have access to the server host are already in a position to run arbitrary code.

Note that the diag/eval endpoint is authenticated and requires the highest privileges in the system (Full Admin). It is the case that a user with access to this endpoint can already delete, corrupt or coy all the data in the system. Customers can mitigate this issue by tightly controlling access to full administrative privileges.

MB-30498

Summary: Fixed an issue where documents were missing from the Scorch index but were intact in an upside_down index.

MB-28847

Summary: Fixed an issue where the memory usage during search index build was found to overshoot the RAM quota.

MB-27429

Summary: Scorch indexes were found to contain duplicate pindexes.

MB-25714

Summary: On Windows platform, a very high memory/CPU usage was observed when search service was indexing using the fts index. This issue is not seen with the new default 'Scorch' index.

MB-31315

Summary: Fixed an issue where the index service threw an error if an index was dropped when index mutation was ongoing.