cbbackupwrapper and cbrestorewrapper

Two tools are included with Couchbase Server Enterprise Edition that improve throughput for the database backup process: cbbackupwrapper and cbrestorewrapper.

cbbackupwrapper and cbrestorewrapper were specifically created to address the single worker thread limitation for one backup/restore target (Bucket, Node). They should be used instead of the standard backup tools, cbbackup and cbrestore.

cbbackupwrapper and cbrestorewrapper are designed to provide a parallel processing approach to backing up a Couchbase Server cluster. At a technical level, they divide each vBucket space into a subset of vBuckets for each backup process and thus improve the speed and efficiency.

In a future release of Couchbase Server, these two tools will be folded into cbbackup and cbrestore respectively and then deprecated.

Syntax

The basic syntax is:

cbbackupwrapper CLUSTER BACKUPDIR OPTIONS
cbrestorewrapper BACKUPDIR CLUSTER OPTIONS

Options

The following are the command options:

Table 1. Basic command options for cbbackupwrapper
Option Description

-h, --help

Show this help message and exit.

-b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE

Specify the bucket to backup. Defaults to all buckets.

--single-node

Use a single server node from the source only.

-u USERNAME, --username=USERNAME

REST username for source cluster or server node. Default is Administrator.

-p PASSWORD, --password=PASSWORD

REST password for source cluster or server node.

-s, --ssl

Transfer data with SSL enabled.

-v, --verbose

Enable verbose messaging.

--path=PATH

Specify the path to cbbackup. Defaults to current directory.

--port=PORT

Specify the bucket port. Defaults to 11210.

-n NUMBER, --number=NUMBER

Specify the number of vbuckets per process. Defaults to 100.

-P PARALLELISM, --parallelism=PARALLELISM

Number of vbucket backup jobs to run at a time. Defaults to 1.

-x EXTRA, --extra=EXTRA

Provide extra, uncommon config parameters; comma-separated key=val(,key=val)* pairs.

-m MODE, --mode=MODE

Backup mode: full, diff or accu [default:diff].

Table 2. Additional configuration options for cbbackupwrapper
Option Description

backoff_cap=10

Max backoff time during rebalance period.

batch_max_bytes=400000

Transfer this # of bytes per batch.

batch_max_size=1000

Transfer this # of documents per batch.

cbb_max_mb=100000

Split backup file on destination cluster if it exceeds MB.

data_only=0

For value 1, only transfer data from a backup file or cluster.

dcp_consumer_queue_length=1000

A DCP client needs a queue for incoming documents/messages. A large length is more efficient, but memory proportional to length*avg. doc size. Below length 150, performance degrades significantly.

design_doc_only=0

For value 1, transfer design documents only from a backup file or cluster.

flow_control=1

For value 0, disable flow control to improve throughput.

max_retry=10

Max number of sequential retries if transfer fails.

recv_min_bytes=4096

Amount of bytes for every TCP/IP call transferred.

rehash=0

For value 1, rehash the partition IDs of each item. Rehashing is needed when transferring data between clusters with a different number of partitions, such as when transferring data from an OS X server to a non-OS X cluster.

report=5

Number batches transferred before updating progress bar in console.

report_full=2000

Number batches transferred before emitting progress information in console.

seqno=0

By default, start seqno from beginning.

uncompress=0

For value 1, restore data in uncompressed mode.

This option is unsupported. To create backups with compression, you should use cbbackupmgr, which is available for Couchbase Server Enterprise Edition only.

Table 3. Basic command options for cbrestorewrapper
Option Description

-h, --help

Show this help message and exit.

-B BUCKET_DESTINATION

Allows you to specify the destination bucket for a restore (assuming that the bucket in the backup is named differently).

-b BUCKET_SOURCE, --bucket-source=BUCKET_SOURCE

Specify the bucket to restore. Defaults to all buckets.

-u USERNAME, --username=USERNAME

REST username for source cluster or server node. Default is Administrator.

-p PASSWORD, --password=PASSWORD

REST password for source cluster or server node.

-s, --ssl

Transfer data with SSL enabled.

-v, --verbose

Enable verbose messaging.

--path=PATH

Specify the path to cbrestore. Defaults to current directory.

--port=PORT

Specify the bucket port. Defaults to 11210.

--from-date=FROM_DATE

Restore data from the date specified as yyyy-mm-dd. By default, all data from the very beginning will be restored.

--to-date=TO_DATE

Restore data till the date specified as yyyy-mm-dd. By default, all data collected will be restored.

-x EXTRA, --extra=EXTRA

Provide extra, uncommon config parameters; comma-separated key=val(,key=val)* pairs.

Table 4. Additional configuration options for cbrestorewrapper
Option Description

backoff_cap=10

Max backoff time during rebalance period.

batch_max_bytes=400000

Transfer this # of bytes per batch.

batch_max_size=1000

Transfer this # of documents per batch.

cbb_max_mb=100000

Split backup file on destination cluster if it exceeds MB.

conflict_resolve=1

By defaault, enable conflict resolution.

data_only=0

For value 1, only transfer data from a backup file or cluster.

dcp_consumer_queue_length=1000

A DCP client needs a queue for incoming documents/messages. A large length is more efficient, but memory proportional to length*avg. doc size. Below length 150, performance degrades significantly.

design_doc_only=0

For value 1, transfer design documents only from a backup file or cluster.

flow_control=1

For value 0, disable flow control to improve throughput.

max_retry=10

Max number of sequential retries if transfer fails.

recv_min_bytes=4096

Amount of bytes for every TCP/IP call transferred.

rehash=0

For value 1, rehash the partition IDs of each item; this is needed when transferring data between clusters with different number of partitions, such as when transferring data from an OS X server to a non-OS X cluster.

report=5

Number batches transferred before updating progress bar in console.

report_full=2000

Number batches transferred before emitting progress information in console.

seqno=0

By default, start seqno from beginning.

uncompress=0

For value 1, restore data in uncompressed mode.

This option is unsupported. To restore from compressed backups, you should use cbbackupmgr, which is available for Couchbase Server Enterprise Edition only.