A newer version of this documentation is available.

View Latest

Delta node recovery failure scenarios

There are conditions where delta node recovery either defaults to full recovery or is not available.

The following are conditions for delta node recovery failures:

  • If topology changes occur while a node is pending delta recovery, delta node recovery is impacted. For example, another node is added, a node is removed, or a node is swapped.

  • If a down node is hard failed over and is marked for removal.

  • If rebalance-in-out operations are performed where the number of in and out nodes do not match (swap rebalance works in this case).

  • If certain bucket operations are performed while a node is pending delta recovery, delta node recovery is impacted. For example, a new bucket is added, a bucket’s replica configuration is changed, or a bucket is flushed.

The following describes scenarios where delta node recovery either defaults to full recovery or is not available.

Node 1 is in delta recovery and Node 2, an active server node, crashes.
  1. Node 1 is failed over and delta recovery is specified. Now, Node 1 is pending delta recovery.

  2. Node 2, an active server, goes down.

    The rebalance operation is not available.
  3. Fail over Node 2.

  4. Cancel the pending delta recovery, specify full recovery, and rebalance.

  5. Repair Node 2, add the server to the cluster, and rebalance.

Node 1 is in delta recovery and Node 1 crashes during rebalance.
  1. Node 1 is failed over, delta recovery is specified, and the rebalance operation is started.

  2. Node 1 crashes and the rebalance operation fails.

  3. Repair Node 1, re-start the server node, and rebalance. Node1 is added back to the cluster using full recovery.

Node 1 is in delta recovery and a bucket operation is performed.

The bucket operations that cause rebalance to fail are adding bucket, changing replica configuration, or flushing bucket

  1. Node 1 is failed over, delta recovery is specified, and then a bucket operation is performed.

  2. Rebalance is performed and fails.

    delta node 01
  3. Cancel the pending delta recovery, specify full recovery, and rebalance.

    Bucket deletion does not lead to delta recovery failure.
Node 1 and Node 2 are in delta node recovery and Node 2 crashes.
  1. Both Node 1 and Node 2 are failed over, delta recovery is specified.

  2. Node 2 crashes.

  3. Rebalance is performed and fails.

    delta node 02