A newer version of this documentation is available.

View Latest

Rebalancing Nodes

Nodes are rebalanced with the POST /controller/rebalance HTTP method and URI.

Description

To start a rebalance process, provide the ejectedNodes and knownNodes parameters. These parameters contain the list of nodes that have been marked to be ejected and the list of nodes that are known within the cluster.

Retrieve information about ejected and known nodes by getting the current node configuration. This ensures that the client making the REST API request is aware of the current cluster configuration. Nodes should have been previously added or marked for removal as appropriate.

HTTP method and URI

The information must be supplied via the ejectedNodes and knownNodes parameters as a POST operation to the /controller/rebalance endpoint.

POST /controller/rebalance

Syntax

curl -v -X POST -u [admin]:[password]
  http://[localhost]:8091/controller/rebalance
  [-d ejectedNodes | -d knownNodes]

Examples

Curl request example for rebalancing new nodes in to a cluster:

curl -v -X POST -u Administrator:password \
'http://192.168.0.77:8091/controller/rebalance'\
-d 'knownNodes=ns_1@192.168.0.77,ns_1@192.168.0.56'

Curl request example for rebalancing nodes out of a cluster:

curl -v -X POST -u Administrator:password \
'http://192.168.0.77:8091/controller/rebalance'\
-d 'ejectedNodes=ns_1@192.168.0.56'

Raw HTTP request example:

POST /controller/rebalance HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*
Content-Length: 63
Content-Type: application/x-www-form-urlencoded

Response codes

The response is 200 (OK) if the operation was successfully submitted.

If the wrong node information has been submitted, JSON with the mismatch error is returned:

{"mismatch":1}

Getting rebalance progress

HTTP method and URI

There are two endpoints for rebalance progress. One is a general request which outputs high-level percentage completion at /pools/default/rebalanceProgress. The second possible endpoint is one corresponds to the detailed rebalance report available in the web console.

GET /pools/default/rebalanceProgress

Syntax

curl -u [admin]:[password]
'[localhost]:8091/pools/default/rebalanceProgress'

Example

Curl request example that returns a JSON structure containing the current progress information:

curl -u Administrator:password \
'192.168.0.77:8091/pools/default/rebalanceProgress'

Raw HTTP request example:

GET /pools/default/rebalanceProgress HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*

Response

The response data packet contains a JSON structure showing the rebalance progress for each node. The progress figure is provided as a percentage (shown as a floating point value between 0 and 1).

{
    "status":"running",
    "ns_1@192.168.0.56":{"progress":0.2734375},
    "ns_1@192.168.0.77":{"progress":0.09114583333333337}
}

Example: detailed

Curl request example with more details about the rebalance:

curl -u Administrator:password \
  'http://192.168.0.77:8091/pools/default/tasks'

Raw HTTP request example:

GET /pools/default/rebalanceProgress HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*

Response: detailed

The response data packet contains a JSON structure showing detailed progress:

{
  type: "rebalance",
  recommendedRefreshPeriod: 0.25,
  status: "running",
  progress: 9.049479166666668,
  perNode: {
    ns_1@10.3.3.61: {
      progress: 13.4765625
    },
    ns_1@10.3.2.55: {
      progress: 4.6223958333333375
    }
  },
  detailedProgress: {
    bucket: "default",
    bucketNumber: 1,
    bucketsCount: 1,
    perNode: {
      ns_1@10.3.3.61: {
        ingoing: {
          docsTotal: 0,
          docsTransferred: 0,
          activeVBucketsLeft: 0,
          replicaVBucketsLeft: 0
        },
        outgoing: {
          docsTotal: 512,
          docsTransferred: 69,
          activeVBucketsLeft: 443,
          replicaVBucketsLeft: 511
        }
      },
      ns_1@10.3.2.55: {
        ingoing: {
          docsTotal: 512,
          docsTransferred: 69,
          activeVBucketsLeft: 443,
          replicaVBucketsLeft: 0
        },
        outgoing: {
          docsTotal: 0,
          docsTransferred: 0,
          activeVBucketsLeft: 0,
          replicaVBucketsLeft: 443
        }
      }
    }
  }
}

This response shows percentage complete for each individual node undergoing rebalance. For each specific node, it provides the current number of docs transferred and other items. For details and definitions of these items.

If rebalance fails, the following response error occurs:

[
  {
    "type": "rebalance",
    "status": "notRunning",
    "errorMessage": "Rebalance failed. See logs for detailed reason. You can try rebalance again."
  }
]

Adjusting rebalance during compaction

Description

If a rebalance is performed while a node is undergoing index compaction, rebalance delays may be experienced. The parameter, rebalanceMovesBeforeCompaction, is used to improve rebalance performance. If this selection is made, index compaction performance is reduced which can result in larger index file size.

This needs to be made as POST request to the /internalSettings endpoint. By default, this setting is 64 which specifies the number of vBuckets which are moved per node until all vBucket movements pauses. After this pause, the system triggers index compaction. Index compaction is not performed while vBuckets are being moved, so if a larger value is specified, it means that the server spends less time compacting the index, which results in larger index files that take up more disk space.

HTTP method and URI

POST /internalSettings rebalanceMovesBeforeCompaction

Syntax

Curl request syntax:

curl -X POST -u admin:password 'http://[localhost]:8091/internalSettings'
    -d rebalanceMovesBeforeCompaction=[value]

Example

Curl request example:

curl -X POST -u Administrator:password 'http://10.5.2.54:8091/internalSettings' \
    -d 'rebalanceMovesBeforeCompaction=256'