A newer version of this documentation is available.

View Latest

Recover partitions from a remote cluster

Data recovery from remote clusters requires XDCR and enough memory and disk space.

If more nodes in a cluster catastrophically fail than the number of replicas, data partitions in that cluster will no longer be available. For example, if you have a four-node cluster with one replica and two nodes fail and are unrecoverable, some data shards will no longer be available.

Data recovery from remote clusters requires an XDCR environment and adequate amount of memory and disk space to support the workload and recovered data.

To recover the missing data partition in case of node failure and when vBuckets are unavailable, follow these steps.

  1. For each failed node, click on Server Nodes  Fail Over.

    You will see whether data is unavailable and which vBuckets are unavailable. If you do not have enough replicas for the number of the failed-over nodes, some vBuckets will no longer be available:

    xdcr recover partition 1
  2. Add new functioning nodes to replace the failed nodes.

    Do not rebalance after you add new nodes to a cluster. Typically, you will rebalance after adding nodes to the cluster, but in this scenario the rebalance will destroy the information about the missing vBuckets and you cannot recover it.

    xdcr recover partition 2

    In this example, two nodes failed in a three-node cluster and a new node at 10.3.3.61 was added. If you are certain that your cluster can easily handle the workload and recovered data, you may choose to skip this step.

  3. Run cbrecovery to recover data from your backup cluster. In the Server Panel, the Stop Recovery button appears.

    xdcr recover partition 3

    After the recovery completes, this button disappears.

  4. Rebalance your cluster.

    Once the recovery is completed, you can rebalance your cluster. Rebalancing will recreate replica vBuckets and redistribute them evenly across the cluster.

    xdcr recover partition 4