Perform Graceful Failover
Graceful failover allows a node to be removed from a cluster proactively, when the cluster is healthy, and all data is available.
Graceful failover allows a Data Service node to be removed from the cluster proactively, in an orderly and controlled fashion (say, for the purposes of system-maintenance). It is manually initiated when the entire cluster is in a healthy state, and all active and replica vBuckets on all nodes are available.
A complete conceptual description of failover and its variants (including graceful) is provided in Failover.
A two-node cluster already exists; as at the conclusion of Join a Cluster and Rebalance.
The cluster has the Full Administrator username of
Administrator, and password of
Proceed as follows:
Access the Couchbase Web Console Servers screen, on node
10.142.181.101, by left-clicking on the Servers tab in the left-hand navigation bar. The display is as follows:
To see further details of each node, left-click on the row for the node. The row expands vertically, as follows:
To initiate failover, left-click on the Failover button, at the lower right of the row for
The Confirm Failover Dialog now appears:
Two radio buttons are provided, to allow selection of either Graceful or Hard failover. Graceful is selected by default.
Confirm graceful failover by left-clicking on the Failover Node button.
Graceful failover is now initiated, and a rebalance occurs as part of the procedure. A progress dialog appears, summarizing overall progress:
For server-level details of the graceful failover process, see the conceptual overview provided in Graceful Failover.
When the process ends, the display is as follows:
This indicates the graceful failover has successfully completed, but an additional rebalance is required to complete the reduction of the cluster to one node.
Left-click the Rebalance button, at the upper right, to initiate a further rebalance. When the process is complete, the Server screen appears as follows:
10.142.181.102has successfully been removed.
To fail a node over gracefully, use the
failover command, as follows:
couchbase-cli failover -c 10.142.181.101:8091 \ --username Administrator \ --password password \ --server-failover 10.142.181.102:8091
--server-failover flag specifies the name and port number of the node to be gracefully failed over.
Progress is displayed as console output:
Gracefully failing over Bucket: 00/00 () 0 docs remaining [====================== ] 17.77
When the progress completes successfully, the following output is displayed:
SUCCESS: Server failed over
The cluster can now be rebalanced with the following command, to remove the failed-over node:
couchbase-cli rebalance -c 10.142.181.101:8091 \ --username Administrator \ --password password \ --server-remove 10.142.181.102:8091
If successful, the operation gives the following output:
SUCCESS: Rebalance complete
To fail a node over gracefully with the REST API, use the
/controller/startGracefulFailover URI, specifying the node to be failed over, as follows:
curl -v -X POST -u Administrator:password \ http://10.142.181.101:8091/controller/startGracefulFailover \ -d 'otpNodeemail@example.com'
Subsequently, the cluster can be rebalanced, and the failed-over node removed, with the
curl -u Administrator:password -v -X POST \ http://10.142.181.101:8091/controller/rebalance \ -d 'knownNodesfirstname.lastname@example.org,email@example.com&ejectedNodesfirstname.lastname@example.org'
A hard failover can be used when a node is unresponsive. See Hard Failover.