Managing Advanced XDCR Settings

    +
    XDCR advanced settings affect replications' behavior, performance, and timing.

    HTTP methods and URI

    POST /settings/replications/
    
    POST /settings/replications/<settings_URI>
    
    GET /settings/replications/
    
    GET /settings/replications/<settings_URI>

    Description

    Used with the POST method, the URIs respectively change global settings for all replications; and for a specific replication, which is referenced by its settings_URI. The settings_URI comprises the id for the replication.

    The global settings are the default values used if settings are not specified during the creation of a particular replication. If settings are specified for a particular replication, the specified settings overwrite the global settings.

    If the global settings are themselves changed, existing replications are not affected: only replications created after the change made to the global settings receive the updated global settings.

    Used with the GET method, the URIs respectively retrieve global settings for all replications; and for a specific replication, which is referenced by its settings_URI.

    Each command requires the Full Admin, Cluster Admin, ReadOnly Admin, External User Security Admin, Local User Security Admin, Backup Full Admin, or XDCR Admin role.

    Curl Syntax

    curl -u <username>:<password> -X POST \
      http://<ip-address-or-domain-name>:8091/settings/replications
    
    curl -u <username>:<password> -X POST \
      http://<ip-address-or-domain-name>:8091/settings/replications/<settings_URI>
    
    curl -u <username>:<password> -X GET \
      http://<ip-address-or-domain-name>:8091/settings/replications
    
    curl -u <username>:<password> -X GET \
      http://<ip-address-or-domain-name>:8091/settings/replications/<settings_URI>

    Responses

    If successful, GET and POST commands return 200 OK. Commands for establishing or retrieving either global settings or the settings for a specific replication or return an object containing the settings.

    Failure to authenticate returns 401 Unauthorized. A URI featuring an incorrect hostname or port returns a failure to connect message. An incorrect settings_URI returns 400 Bad Request, and an object such as {"errors":{"_":"requested resource not found"}}.

    An attempt to set a global-only setting for a specific replication returns 400 Bad Request, and an object indicating which submitted parameters were incorrect. For example, the attempted use of POST /settings/replications/<settings_URI> to establish values for goGC and goMaxProcs returns the following:

    {
      "gogc": "Invalid settings key",
      "gomaxprocs": "Invalid settings key"
    }

    A replication-specific parameter (such as pauseRequested) is ignored, if incorrectly specified in an attempt to modify global settings.

    Examples

    The following examples show how settings can be established and retrieved.

    Get Settings for All Replications

    The following example retrieves settings for all replications. Note that the output is piped to the jq command, to facilitate readability:

    curl -u Administrator:password -X GET http://10.144.210.101:8091/settings/replications | jq '.'

    If successful, the call returns an object similar to the following:

    {
      "checkpointInterval": 600,
      "compressionType": "Auto",
      "desiredLatency": 50,
      "docBatchSizeKb": 2048,
      "failureRestartInterval": 10,
      "filterBypassExpiry": false,
      "filterDeletion": false,
      "filterExpiration": false,
      "goGC": 100,
      "goMaxProcs": 4,
      "hlvPruningWindowSec": 259200,
      "jsFunctionTimeoutMs": 20000,
      "logLevel": "Info",
      "mergeFunctionMapping": {},
      "networkUsageLimit": 0,
      "optimisticReplicationThreshold": 256,
      "priority": "High",
      "retryOnRemoteAuthErr": true,
      "retryOnRemoteAuthErrMaxWaitSec": 3600,
      "sourceNozzlePerNode": 2,
      "statsInterval": 1000,
      "targetNozzlePerNode": 2,
      "workerBatchSize": 500
    }

    Get Settings for a Specific Replication

    To reference a specific replication, the replication’s settings_URI must first be retrieved. Use the following call, which is explained in detail in Getting Cluster Tasks. Note that the call pipes output to the jq program, to facilitate the readability of output.

    curl -u Administrator:password -v -X GET \
    http://10.143.192.101:8091/pools/default/tasks | jq '.'

    If successful, the call returns an array of ongoing cluster tasks, similar to the following:

    [
      {
        "statusId": "2bbbeac0668c700f43ad694361e1a948",
        "type": "rebalance",
        "status": "notRunning",
        "statusIsStale": false,
        "masterRequestTimedOut": false,
        "lastReportURI": "/logs/rebalanceReport?reportID=8fccd4f0764db237f64487afe5f06bb7"
      },
      {
        "cancelURI": "/controller/cancelXDCR/c371211ec399a7b20acc1ea264f0841e%2Ftravel-sample%2Fts",
        "settingsURI": "/settings/replications/c371211ec399a7b20acc1ea264f0841e%2Ftravel-sample%2Fts",
        "status": "running",
        "replicationType": "xmem",
        "continuous": true,
        "filterBypassExpiry": false,
        "filterDeletion": false,
        "filterExpiration": false,
        "filterExpression": "",
        "id": "c371211ec399a7b20acc1ea264f0841e/travel-sample/ts",
        "pauseRequested": false,
        "source": "travel-sample",
        "target": "/remoteClusters/c371211ec399a7b20acc1ea264f0841e/buckets/ts",
        "type": "xdcr",
        "recommendedRefreshPeriod": 10,
        "changesLeft": 0,
        "docsChecked": 84792,
        "docsWritten": 0,
        "maxVBReps": null,
        "errors": []
      }
    ]

    A settings_URI if provided for the single ongoing replication, within the second member of the array. The settings_URI can be used to retrieve information on the corresponding replication.

    For example, enter the following:

    curl -u Administrator:password -X GET http://10.144.210.101:8091/settings/replications/c371211ec399a7b20acc1ea264f0841e%2Ftravel-sample%2Fts | jq '.'

    If the call is successful, an object containing the settings for the specified replication is returned:

    {
      "checkpointInterval": 600,
      "colMappingRules": {},
      "collectionsExplicitMapping": false,
      "collectionsMigrationMode": false,
      "collectionsMirroringMode": false,
      "collectionsOSOMode": true,
      "compressionType": "Auto",
      "desiredLatency": 50,
      "docBatchSizeKb": 2048,
      "failureRestartInterval": 10,
      "filterBypassExpiry": false,
      "filterDeletion": false,
      "filterExpiration": false,
      "filterExpression": "",
      "hlvPruningWindowSec": 259200,
      "jsFunctionTimeoutMs": 20000,
      "logLevel": "Info",
      "mergeFunctionMapping": {},
      "networkUsageLimit": 0,
      "optimisticReplicationThreshold": 256,
      "pauseRequested": false,
      "priority": "High",
      "retryOnRemoteAuthErr": true,
      "retryOnRemoteAuthErrMaxWaitSec": 3600,
      "sourceNozzlePerNode": 2,
      "statsInterval": 1000,
      "targetNozzlePerNode": 2,
      "type": "xmem",
      "workerBatchSize": 500
    }

    Change a Setting for a Specific Replication

    The following changes the checkpointInterval for a specific replication:

    curl -X POST -u Administrator:password http://10.144.210.101:8091/settings/replications/c371211ec399a7b20acc1ea264f0841e%2Ftravel-sample%2Fts -d checkpointInterval=700 | jq '.'

    If successful, the call returns an object containing all current settings for the replication, including what has been changed:

    {
      "checkpointInterval": 700,
      "colMappingRules": {},
      "collectionsExplicitMapping": false,
      "collectionsMigrationMode": false,
      "collectionsMirroringMode": false,
      "collectionsOSOMode": true,
      "compressionType": "Auto",
      "desiredLatency": 50,
      "docBatchSizeKb": 2048,
      "failureRestartInterval": 10,
      "filterBypassExpiry": false,
      "filterDeletion": false,
      "filterExpiration": false,
      "filterExpression": "",
      "hlvPruningWindowSec": 259200,
      "jsFunctionTimeoutMs": 20000,
      "logLevel": "Info",
      "mergeFunctionMapping": {},
      "networkUsageLimit": 0,
      "optimisticReplicationThreshold": 256,
      "pauseRequested": false,
      "priority": "High",
      "retryOnRemoteAuthErr": true,
      "retryOnRemoteAuthErrMaxWaitSec": 3600,
      "sourceNozzlePerNode": 2,
      "statsInterval": 1000,
      "targetNozzlePerNode": 2,
      "type": "xmem",
      "workerBatchSize": 500
    }

    XDCR Advanced Settings

    The advanced settings for XDCR are as follows. The subset of advanced settings that can be configured by means of Couchbase Web Console is provided at XDCR Advanced Settings.

    Table 1. XDCR Advanced Settings
    Parameter Value Description

    checkpointInterval

    Integer (60 to 14400)

    Default: 600 The interval for checkpointing in seconds.

    This setting can be established and retrieved either for an individual replication or globally.

    colMappingRules

    JSON Object

    Collection-related rules according to which explicit mapping or migration is defined for the replication. See Rules for Explicit Mappings and Rules for Migration.

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    collectionsExplicitMapping

    Boolean (true or false)

    Whether the replication uses explicit mapping: see Rules for Explicit Mappings.

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    collectionsMigrationMode

    Boolean (true or false)

    Whether the replication uses migration mode: see Migration.

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    collectionsOSOMode

    Boolean (true or false)

    Whether the replication optimizes performance; by streaming, from a source bucket, mutations that could be out of order, in terms of sequence-number. Default is true.

    compressionType

    String

    Default: Auto. Specifies whether documents are to be compressed for XDCR, and if so, what compression type is to be used. For information, see XDCR Advanced Settings.

    This setting can be established and retrieved either for an individual replication or globally.

    desiredLatency

    Integer

    Specifies the amount of time in milliseconds within which a high-priority replication’s currently remaining changes should be reduced in number to zero, by ongoing transmission of the changes from source to target bucket. If Couchbase Server estimates that the time required is greater than that specified by desiredLatency, the replication is considered to be backlogged: XDCR therefore attempts to communicate with the Data Service, to establish a higher DCP priority for the replication, and thereby ensure swifter transmission of data.

    This setting applies only to high-priority replications, or to medium-priority replications that achieve high-priority status (see XDCR Priority). The default value is 50. The lower the value, the higher (potentially) the transmission rate, and the greater the load on the target cluster.

    This setting can be established and retrieved either for an individual replication or globally.

    docBatchSizeKb

    Integer (10 to 10000)

    Default: 2048. The size of a batch in kilobytes.

    This setting can be established and retrieved either for an individual replication or globally.

    failureRestartInterval

    Integer (1 to 300)

    Default: 10. The number of seconds to wait after a failure before restarting replication. This setting can be established and retrieved either for an individual replication or globally.

    filterBypassExpiry

    Boolean (true or false)

    Default: false. Whether a document’s TTL should be replicated with the document or not. A value of true means that the TTL is removed from the document.

    This setting can be established and retrieved either for an individual replication or globally.

    filterDeletion

    Boolean (true or false)

    Default: false. Whether mutations corresponding to the deletion of documents on the source cluster should be either filtered out of the replication to the target cluster, or allowed to remain in. A value of true means that the mutation is filtered out, ensuring that it is not replicated to the target cluster; while a value of false means that the mutation is not filtered out, ensuring that it is replicated to the target cluster.

    This setting can be established and retrieved either for an individual replication or globally.

    filterExpiration

    Boolean (true or false)

    Default: false. Whether mutations corresponding to the expiration of documents on the source cluster should be either filtered out of the replication to the target cluster, or allowed to remain in. A value of true means that the mutation is filtered out, ensuring that it is not replicated to the target cluster; while a value of false (the default) means that the mutation is not filtered out, ensuring that it is replicated to the target cluster.

    This setting can be established and retrieved either for an individual replication or globally.

    filterExpression

    String (a filter expression)

    Default: null. A filter expression to be matched against the ids, field-names, values, and extended attributes of documents in the source bucket. Each document that produces a successful match is replicated; other documents are not replicated. For supported expressions, see XDCR Advanced Filtering Reference.

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    goGC

    Integer (0 to 100) or String ("off")

    Default: 100. The initial garbage collection target percentage for the replication. A garbage collection is triggered when the ratio of freshly allocated data to live data remaining after the previous collection reaches this percentage. A value of "off" disables the garbage collector entirely.

    This setting is only returned when global settings are retrieved; and can only be set when global settings are set. It cannot be set when an individual replication is created, or when its settings are modified.

    goMaxProcs

    Integer

    Default: 4. The maximum number of threads used per node, to support XDCR. A greater number of threads increases parallelism, and may thereby produce enhanced XDCR performance.

    This setting is only returned when global settings are retrieved; and can only be set when global settings are set. It cannot be set when an individual replication is created, or when its settings are modified.

    hlvPruningWindowSec

    Integer

    Used in custom conflict-resolution, and available in Developer Preview Mode only. The integer specifies the duration, in seconds, of the time-window that starts when a remote cluster has modified the document that is currently being replicated. The remote cluster’s update is recorded in a Hybrid Logical Vector (HLV). If the specified time-window expires without further modification being made by the remote cluster, information about the remote cluster’s update is removed from the HLV.

    This setting can be established and retrieved either for an individual replication or globally.

    jsFunctionTimeoutMs

    Integer

    Used in custom conflict-resolution, and available in Developer Preview Mode only. Specifies the number of milliseconds that must elapse before the timeout of a JavaScript conflict-resolution function.

    This setting can be established and retrieved either for an individual replication or globally.

    logLevel

    String

    Default: Info. The level of logging, such as Error/Info/Debug/Trace.

    This setting can be established and retrieved either for an individual replication or globally.

    mergeFunctionMapping

    JSON Object

    This is available in Developer Preview Mode only. Each of the key-value pairs in the JSON object should map a collection-specifier (in the form scope.collection) to the name of a defined merge-function.

    networkUsageLimit

    Integer

    Default: 0. Specifies the upper limit for network usage during replication, for the entire cluster, in MB per second. The default is 0, meaning no limit is applied. For information, see XDCR Advanced Settings.

    This setting can be established and retrieved either for an individual replication or globally.

    optimisticReplicationThreshold

    Integer (0 to (20*1024*1024))

    Default: 256. Documents with sizes less than this threshold (in bytes) will be replicated optimistically.

    This setting can be established and retrieved either for an individual replication or globally.

    pauseRequested

    Boolean (true or false)

    Default: false. Indicates whether the replication has been issued a pause request.

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    priority

    High, Medium, or Low

    Default: High. Resource-allocation for the replication. For information, see XDCR Priority.

    This setting can be established and retrieved either for an individual replication or globally.

    retryOnRemoteAuthErr

    Boolean (true or false)

    Whether to retry connection for a replication that has failed due to a problem in authenticating with the target cluster. The default is true.

    This setting can be established and retrieved either for an individual replication or globally.

    retryOnRemoteAuthErrMaxWaitSec

    Integer

    The maximum number of seconds to wait before retrying a connection that failed due to a problem in authenticating with the target cluster. This only takes effect when retryOnRemoteAuthErr is true.

    This setting can be established and retrieved either for an individual replication or globally.

    sourceNozzlePerNode

    Integer (1 to 100)

    The number of nozzles that can be used for this replication per source cluster node. This together with target_nozzle_per_node controls the parallelism of the replication.

    This setting can be established and retrieved either for an individual replication or globally.

    statsInterval

    Integer (200 to 600000)

    Default: 1000. The interval (in milliseconds) for statistics updates.

    This setting can be established and retrieved either for an individual replication or globally.

    targetNozzlePerNode

    Integer (1 to 100)

    The number of outgoing nozzles per target node. This together with source_nozzle_per_node controls the parallelism of the replication.

    This setting can be established and retrieved either for an individual replication or globally.

    type

    String "xmem"

    The replication type, which must be "xmem".

    This setting can only be established for and retrieved from an individual replication: it cannot be established or retrieved as part of global settings.

    workerBatchSize

    Integer (500 to 10000)

    Default: 500. The number of mutations in a batch.

    This setting can be established and retrieved either for an individual replication or globally.

    See Also

    The REST call for creating a replication and specifying parameters is described in Creating XDCR Replications. An overview of XDCR is provided in Cross Data Center Replication (XDCR). The subset of advanced settings that can be configured by means of Couchbase Web Console is provided at XDCR Advanced Settings.