A newer version of this documentation is available.

View Latest

Bulk operations

You can perform many operations at once by using the _multi family of methods.
See Couchbase Developer documentation: Batching operations

Most API functions have both single and multi-key (batched) variants. The batched variant has the same name as the single-key variant, but its method name has multi appended to it. The _twisted module also accepts the camelCased variants of getMulti, upsertMulti and the like to conform with standard Twisted naming conventions.

The batched operations are significantly quicker and more efficient, especially when dealing with many small values, because they allow pipelining of requests and responses, saving on network latency.

Batched operations tend to accept an iterable of keys (or a dict of keys, depending on the method) and return a dictionary of the following format:

c.get_multi(["key1", "key2", "key3"])

{
    "key1" : ValueResult(...),
    "key2" : ValueResult(...),
    "key3" : ValueResult(...)
}

Storing and retriving multiple items

To store multiple items, use the upsert_multi method. This accepts a dictionary of document IDs and their values. To retrieve multiple IDs, pass a Python list, tuple, or set containing the document IDs to retrieve.

from couchbase.bucket import Bucket
from pprint import pprint

bkt = Bucket('couchbase://192.168.33.101/default')
rvs = bkt.upsert_multi({
    'foo': 'foovalue',
    'bar': 'barvalue',
    'baz': 'bazvalue'
})
pprint(rvs)

print ""
print "Getting items"
rvs = bkt.get_multi(('foo', 'bar', 'baz'))
pprint(rvs)

Outputs:

{u'bar': OperationResult<RC=0x0, Key=u'bar', CAS=0xd8b0c8ba8960aa13>,
 u'baz': OperationResult<RC=0x0, Key=u'baz', CAS=0xb02b02bb8960aa13>,
 u'foo': OperationResult<RC=0x0, Key=u'foo', CAS=0x183e87bb8960aa13>}

Getting items
{u'bar': ValueResult<RC=0x0, Key=u'bar', Value=u'barvalue', CAS=0xd8b0c8ba8960aa13, Flags=0x2000000>,
 u'baz': ValueResult<RC=0x0, Key=u'baz', Value=u'bazvalue', CAS=0xb02b02bb8960aa13, Flags=0x2000000>,
 u'foo': ValueResult<RC=0x0, Key=u'foo', Value=u'foovalue', CAS=0x183e87bb8960aa13, Flags=0x2000000>}

Handling errors in bulk operations

Like their single item counterparts, the client will raise exceptions if they are encountered during a bulk operation. You can simply catch and handle the exception normally, just as in the single-item methods. Sometimes however, you may wish to determine which specific item failed and take action upon it. A CouchbaseError instance contains an all_results property which contains the dictionary which would have been returned, had the operation not failed. In the case of an exception, you can iterate over the items in the dictionary and handle only those whose success method is False

c.insert("foo", "foo value")

try:
    c.insert_multi({
        "foo" : "foo value",
        "bar" : "bar value",
        "baz" : "baz value"
    })
except CouchbaseError as exc:
    for k, res in exc.all_results.items():
        if res.success:
            # Handle successful operation
        else:
            print "Key {0} failed with error code {1}".format(k, res.rc)
            print "Exception {0} would have been thrown".format(
                CouchbaseError.rc_to_exctype(res.rc))

In the example above, we insert the foo ID; then we attempt to insert the items of foo, bar, baz. In this case, the first item will fail but the others may succeed.

For retrieval operations you may use the quiet=True keyword argument to suppress NotFoundError exceptions. In this case you will need to iterate over the returned dictionary object to determine which items failed or not. As an added convenience, the dictionary object is actually a MultiResult object which inherits from dict. You may query the all_ok property of the returned object to determine if all items succeeded or not, which may be more efficient than iterating over all the items

rvs = bkt.get_multi(('foo', 'bar', 'baz'), quiet=True)
if not rvs.all_ok:
    # Some items failed:
    for k, v in rvs.iteritems():
        if not v.success:
            # Handle failure

Bulk operations in the Twisted API

The Twisted API contains the getMulti and upsertMulti family of methods, however the need for these calls is typically not as great as it is within the synchronous API. Items are efficiently pipelined in an internal queue and are only sent to the cluster (in bulk) once control is returned back to the underlying reactor. Using bulk operations may still help decrease CPU usage and may be more intuitive for your application to handle if the items are treated as a collection.

As in their synchronous counterparts, the callback for a Deferred returned from a multi method contains the MultiResult dictionary subclass, and may be used like so:

from twisted.internet import reactor
from txcouchbase.bucket import Bucket
from pprint import pprint

def on_store_ok(results):
    print "Stored results"
    pprint(results)

def on_get_ok(results):
    print "Retrieve results:"
    pprint(results)

bkt = Bucket('couchbase://192.168.33.101/default')
bkt.upsertMulti({
    'foo': 'foovalue',
    'bar': 'barvalue',
    'baz': 'bazvalue'
}).addCallback(on_store_ok)

bkt.getMulti(('foo', 'bar', 'baz')).addCallback(on_get_ok)
reactor.run()
Stored results
{u'bar': OperationResult<RC=0x0, Key=u'bar', CAS=0x28f307217e61aa13>,
 u'baz': OperationResult<RC=0x0, Key=u'baz', CAS=0xe8f9d7207e61aa13>,
 u'foo': OperationResult<RC=0x0, Key=u'foo', CAS=0x505a50217e61aa13>}
Retrieve results:
{u'bar': ValueResult<RC=0x0, Key=u'bar', Value=u'barvalue', CAS=0x28f307217e61aa13, Flags=0x2000000>,
 u'baz': ValueResult<RC=0x0, Key=u'baz', Value=u'bazvalue', CAS=0xe8f9d7207e61aa13, Flags=0x2000000>,
 u'foo': ValueResult<RC=0x0, Key=u'foo', Value=u'foovalue', CAS=0x505a50217e61aa13, Flags=0x2000000>}