Filter a Replication

An XDCR replication can be filtered, by means of expressions; so that only selected documents are replicated from the source to the target cluster.

Understanding Filtering

XDCR Advanced Filtering allows a limited subset of documents to be replicated from the source bucket. An expression is created, and used to identify documents that provide a match. The expression can be applied to:

The document’s key
Fields and values in the document-body
The document’s extended attributes

This page explains the practical steps whereby filtering can be performed. Note that when entered by means of the UI, the expression can be a maximum of 250 bytes (characters) in length: this restriction does not apply to expressions entered by means of the CLI or the REST API.

For a full conceptual description and links to reference information on regular and filtering expressions, see XDCR Advanced Filtering.

Examples on This Page

The examples in the subsections below show how to filter the same replication; using the UI, the CLI, and the REST API respectively. As their starting-point, the examples assume the following:

Two clusters already exist; each containing a single node. These are named after their IP addresses: 10.143.192.101 and 10.143.192.102.
Cluster 10.143.192.101 contains a single bucket, which is the travel-sample bucket.
Cluster 10.143.192.102 contains three buckets, named France_airport, US_airline, and US_airport.
On cluster 10.143.192.101, a reference has been defined to 10.143.192.102, as a remote cluster.
Each cluster has the Full Administrator username of Administrator, and password of password.

Filter an XDCR Replication with the UI

Proceed as follows:

On 10.143.192.101, access Couchbase Web Console. Left-click on the XDCR tab, in the right-hand navigation menu.

This displays the XDCR Replications screen, whose Remote Clusters panel currently shows that a reference to 10.143.192.102 has been defined; and whose Outgoing Replications panel shows that no replications have yet been defined.
To define and filter a replication, left-click on the Add Replication tab, towards the right:

This brings up the Add Replication dialog:
Specify travel-sample as the source bucket, 10.143.192.10 as the target cluster, and France_airport as the destination bucket. Then, left-click on Replication Filters. This expands the lower part of the dialog, as follows:

To replicate only those documents whose key features the string airport, and whose body contains France as the value of country, enter the expression REGEXP_CONTAINS(META().id, "^airport") AND country = "France", in the Filter Expression field:
Test the expression against a specified document.

Note that an expression must be tested successfully, before it can be included as part of the replication: if an expression is specified and attemptedly saved without having been tested, the expression is ignored when saving occurs; and the replication is thus started in unfiltered form.

Enter the document’s id in the interactive field adjacent to the Test Filter button:

Left-click on the Test Filter button. If the specified document provides a successful match, this is indicated to the right of the Test Filter button:

If the test fails a no match notification is provided, in the same location.
When testing has proved successful, left-click on the Save button. The dialog disappears. The Outgoing Replications panel now appears as follows:

This indicates that a replication is now in process from the travel-sample bucket to the remote bucket France_airport, on cluster 10.143.192.102. To check the filter that has been applied, left-click on the the filter tab:

Note that if a filter has been specified, but has not been successfully tested, and therefore has not been included in the replication, the filter tab does not appear on the row for the replication.
To examine the content of bucket France_airport, on cluster 10.143.192.102, access the cluster by means of Couchbase Web Console, and left-click on the Buckets tab, in the left-hand navigation bar. The display shows the three buckets previously defined for the cluster:
Open the row for France_airport, by left-clicking on the row. Then, left-click on the Documents tab, at the upper-right:

The display now shows documents currently contained by the bucket.

Each has been replicated from 10.143.192.102, in accordance with the filtering expression specified.
Add two more filtered replications.

To replicate documents concerning US airlines to US_airline, use the expression REGEXP_CONTAINS(META().id, "^airline") AND country = "United States".

To replicate documents concerning US airports to US_airport, use the expression REGEXP_CONTAINS(META().id, "^airport") AND country = "United States".

Once these additional, two replications have been added, the Outgoing Replications panel appears as follows:

Subsequent examination of the buckets on 10.143.192.102 indicates that each bucket is receiving only the documents specified by its corresponding filter.

For lists of available regular and filtering expressions, see the XDCR Reference.

Deletion Filters

The Add Replication panel features optional deletion filters:

These filters control whether the deletion of a document at source causes deletion of a replica that has been created. Each filter covers a specific deletion-context:

Do not replicate document expirations. If checked, this means that if, having been replicated, the document at source expires and is deleted, the replicated copy of the document will not be deleted. Conversely, if this option is not checked, expirations at source are replicated; meaning that the replicated copy of the document will be deleted.
Do not replicate DELETE operations. If checked, this means that if, having been replicated, the document at source is expressly deleted, the replicated copy of the document will not be deleted. Conversely, if this option is not checked, deletions at source are replicated; meanining that the replicated copy of the document will be deleted.
Remove TTL from replicated items. If checked, this means that the TTL that a document bears at source is not made part of the replicated copy of the document. Conversely, if this option is not checked, the TTL is made part of the replicated copy of the document, and thereby determines when the replicated copy of the document expires.

For information on TTL and expiration, see Expiration.

Editing

Once established, an XDCR filter — along with Replication Priority and Advanced Replication Settings — can be edited.

In the Outgoing Replications panel, left-click on the row for the replication. When the Edit button is displayed, left-click on it. This brings up the Edit Replication dialog:

Make changes to the filter as required. Then, select one of the radio-button options, which are Save filter & restart replication, and Save & continue replicating. For a complete description of these options, see Filter-Expression Editing. Finally, left-click on the Save button.

Filter an XDCR Replication with the CLI

Staring from the scenario defined above, in Examples on This Page, use the CLI xdcr-replicate command to create a filtered XDCR replication, as follows:

/opt/couchbase/bin/couchbase-cli xdcr-replicate -c localhost:8091 \
-u Administrator -p password --create --xdcr-cluster-name 10.143.192.102 \
--xdcr-from-bucket travel-sample --xdcr-to-bucket US_airport \
--filter-expression 'REGEXP_CONTAINS(META().id, "^airport") AND country = "United States"'

This expression specifies that all documents filtered by means of the specified filter-expression will be replicated from travel-sample, on the localhost 10.143.192.101, to US_airport on the remote cluster 10.143.192.102.

If successful, the command returns the following output:

SUCCESS: XDCR replication created

For more information, see the complete reference for the xdcr-replicate command.

Filter an XDCR Replication with the REST API

Starting from the scenario defined above, in Examples on This Page, using the REST API’s POST /controller/createReplication HTTP method and URI, create a filtered XDCR reference as follows:

curl -v -X POST -u Administrator:password \
http://localhost:8091/controller/createReplication \
-d replicationType=continuous \
-d toBucket=US_airport \
-d toCluster=10.143.192.102 \
-d fromBucket=travel-sample \
-d filterExpression=REGEXP_CONTAINS%28META%28%29.id%2C+%22%5Eairport%22%29+AND+country+%3D+%22United+States%22

For more information, see Creating XDCR Replications.

Next Steps

Data, lost from a local cluster due to catastrophic outage, can be recovered from a remote cluster to which an XDCR replication was occurring. See Recover Data with XDCR.