CRUD Document Operations using the C (libcouchbase) SDK with Couchbase Server

Command Structure

Operations in the C SDK are scheduled using command structures, all following a similar structure to lcb_CMDBASE.

The command structure contains the document ID for the specific item and additional per-command data, as required. The document ID indicates which document should be accessed and is necessary for any data access operation.

Command structures only serve as a way to pass parameters to scheduling operations and are therefore able to live on the stack, and do not need to be allocated using dynamic memory allocation functions such as malloc and free.

All fields in lcb_CMDBASE are overlayed in other command structure types, so that any command type (e.g. lcb_CMDSTORE) can be seen as a subclass). This is done by specifying the fields of lcb_CMDBASE as a macro (LCB_CMD_BASE) and expanding the macro at the beginning of each command.

The contents of lcb_CMDBASE include:

  • key: Complex structure containing the ID of the document being accessed. Should be set using LCB_CMD_SET_KEY

  • exptime: The new expiration time for the document to be set with this operation

  • cas: Check-and-set value used with optimistic locking

Document IDs may be set on command structures using the LCB_CMD_SET_KEY() macro:

lcb_CMDGET gcmd = { 0 }
LCB_CMD_SET_KEY(&gcmd, "hello", 5);
// ...

The macro receives the buffer and length of the document ID and sets the appropriate field in the command structure.

The contents of the buffer for the document ID are copied into the library when the operation is scheduled.

The contents of documents in Couchbase are typically JSON, as this format allows other SDKs as well as the query service to interact with the documents. The C SDK does not come with a JSON encoder; though many exist and vary in their features and performance.

Operation requests

All Key-Value operations follow a common pattern in the C SDK. Because the first step in using a database is actually populating it with data, performing operations will be demonstrated first through storing a document.

A document can be created using the lcb_store3() function. This function accepts an lcb_CMDSTORE structure that contains the document ID and the value that should serve as the document itself, and the operation which places various preconditions storing the document:

lcb_CMDSTORE scmd = { 0 };
LCB_CMD_SET_KEY(&scmd, "some_key", strlen("some_key"));
LCB_CMD_SET_VALUE(&scmd, "[1,2,3]", strlen("[1,2,3]");
scmd.operation = LCB_ADD;
lcb_store3(instance, NULL, &scmd);
lcb_wait(instance);

The above will store a JSON array using the document ID some_key. The lcb_store3() function schedules the operation to be performed the next time the library performs I/O, which in this case is at the lcb_wait() function.

The success or failure of an operation is only available within the callback. Though lcb_store3() (or similar) returns a status code, the status code determines if the command was successfully scheduled, and not if the server successfully executed the command.

Operation results

All operations receive results in designated callbacks. The result contains the status of the operation and an optional pointer associating the request with a response. The result will also contain additional operation-specific operation such as values and subcodes.

Callbacks are installed using the lcb_install_callback3, which is supplied with the type of operation and the callback to be invoked whenever a response for such an operation is received:

lcb_install_callback3(instance, LCB_CALLBACK_GET, get_handler);
lcb_install_callback3(instance, LCB_CALLBACK_STORE, store_handler);

The callback is passed a pointer to an lcb_RESPBASE structure which represents the common in-memory layout of all Couchbase responses. The response structure can be subclassed to an operation-specific type (for example, lcb_RESPGET or lcb_RESPSTORE) to obtain more information specific to the operation type. Here is an example of handling a response for lcb_get3()

void get_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *rb)
{
    if (rb->rc != LCB_SUCCESS) {
        printf("Error in getting response: %s\n", lcb_strerror(NULL, rb->rc));
    } else {
        const lcb_RESPGET *resp = (const lcb_RESPGET*)rb;
        printf("Got value: %.*s\n", (int)resp->nvalue, resp->value);
    }
}

User data in responses

In most cases, you will need a way to associate a response with user (application) data; this is necessary, for example, if you wish to actually make use of the value or status received in the response. The SDK allows you to pass an application pointer in the request and read it in the response; this pointer is known as a cookie and is opaque to the SDK and is thus never dereferenced by it; here’s an example of the above handler making use of a cookie:

typedef struct {
    const char *value;
    size_t nvalue;
    lcb_error_t status;
} GetInfo;

void get_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *rb)
{
    GetInfo *gi = rb->cookie;
    gi->status = rb->rc;
    if (rb->rc == LCB_SUCCESS) {
        const lcb_RESPGET *resp = (const lcb_RESPGET*)rb;
        gi->value = malloc(resp->nvalue);
        gi->nvalue = resp->nvalue;
        memcpy(gi->value, resp->value, resp->nvalue);
    }
}

GetInfo* get_item(lcb_t instance, const char *key)
{
    lcb_CMDGET gcmd = { 0 };
    GetInfo *retval = calloc(1, sizeof *retval);
    LCB_CMD_SET_KEY(&gcmd, key, strlen(key));
    lcb_get3(instance, retval, &gcmd);
    lcb_wait(instance);
    return retval;
}
The cookie does not have any memory requirements. It can be automatic memory, dynamic memory, or NULL, depending on application constraints.
The SDK owns the memory of the response structure and all pointers it contains. Copy out any relevant fields (such as the value or status) if access to them are needed beyond the scope of the callback.

Generic response handling

For many types of operations, simple, generic handling can suffice: this means simply checking whether the operation succeeded or failed:

void generic_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *resp) {
    printf("Operation %s: %s\n", lcb_strcbtype(cbtype), lcb_strerror(NULL, resp->rc);
    if (resp->rc == LCB_SUCCESS) {
        handle_success(cbtype, resp);
    } else {
        handle_failure(cbtype, resp);
    }
}

The above example shows a callback function which prints the type and status operation to the screen and then invokes either the application-defined handle_success() or handle_failure() function depending on whether or not the operation succeeded. The second argument to the callback, cbtype, contains the actual LCB_CALLBACK_* type being invoked, so that the same callback can be installed for multiple operation types; for example using the same callback for LCB_CALLBACK_STORE and LCB_CALLBACK_REMOVE.

You can install a default callback which may be invoked if no operation-specific callback has been installed:

lcb_install_callback3(instance, LCB_CALLBACK_DEFAULT, generic_handler);
lcb_install_callback3(instance, LCB_CALLBACK_GET, get_handler);

In the above block, the generic_handler is installed as the default callback, and the get_handler is the callback for lcb_get3() operations: All operations except lcb_get3() (LCB_CALLBACK_GET) will be passed to the default callback.

Storing documents

This describes full-document mutation functionality which alters the entire contents of the document. See sub-document mutations for when you only need to modify parts of a document.

Documents can be stored using the lcb_store3 function, and populate it with the document ID (LCB_CMD_SET_KEY(&cmd, "key", strlen("key"));), value (LCB_CMD_SET_VALUE(&cmd, "value", strlen("value"))) and the operation (cmd.operation = LCB_SET).

The operation dictates conditions (or lack thereof) which must be satisfied before the document may be stored. Possible values include:

  • LCB_SET: Unconditionally store the item, and don’t set any conditions. This option is also known as upsert. If an existing document exists with the same ID, it is overwritten with the contents (value) of the new operation. If no document exists with the same ID, it is created and set to the value of the new operation.

  • LCB_ADD: Only store the item if no other document exists with the given ID. If an existing document is precent with the same document ID then the operation will fail with LCB_KEY_EEXISTS.

  • LCB_REPLACE: Only store the item if an existing document already exists with the same ID. In this case the existing document’s contents are replaced with the new contents. If no document already exists the operation will fail with LCB_KEY_ENOENT

  • LCB_APPEND, LCB_PREPEND: These options allow raw byte concatenation

You can also add concurrency control and document expiration when using lcb_store3()

You can install a callback to receive the status of lcb_store3 operations using lcb_install_callback3(instance, LCB_CALLBACK_STORE, store_handler).

Fully-compilable and executable example: Storing documents using C SDK

When a store operation is successful, you may check the operation’s durability. In the C SDK this is done using the distinct lcb_endure3_ctxnew:

void store_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *resp) {
    if (resp->rc != LCB_SUCCESS) {
        return;
    }
    lcb_MULTICMD_CTX *ctx;
    lcb_durability_opts_t opts = { 0 };
    opts.version = 0;
    opts.v.v0.persist_to = -1;
    opts.v.v0.replicate_to = -1;
    opts.v.v0.cap_max = 1;
    lcb_error_t err_out;
    mctx = lcb_endure3_ctxnew(instance, &opts, &err_out);
    lcb_CMDENDURE dcmd = { 0 };
    LCB_CMD_SET_KEY(&dcmd, resp->key, resp->nkey);
    dcmd.cas = resp->cas;
    mctx->addcmd(dcmd);
    mctx->done(mctx, NULL);
}

Setting the persist_to and replicate_to fields to -1 and the cap_max to a true value instructs the client to use the maximum-available durability

Fully-compilable and executable example demonstrating durability requirements in the C SDK: https://github.com/couchbaselabs/devguide-examples/blob/server-4.5/c/durability.cc

Retrieving Documents

This describes the full-document retrieval functionality which retrieves the entire document. See sub-document retrievals for when you only need to retrieve parts of a document.

Documents can be retrieved using the lcb_get3 function. The server will attempt to fetch the document if it exists:

lcb_CMDGET gcmd = { 0 };
LCB_CMD_SET_KEY(&gcmd, "key", 3);
lcb_get3(instance, NULL, &gcmd);
lcb_wait(instance);

The actual contents of the document will be available in the response structure, which is passed to the callback you installed via lcb_install_callback3(instance, LCB_CALLBACK_GET, get_handler):

void get_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *rb)
{
    const lcb_RESPGET *resp = (const lcb_RESPGET *)rb;
    if (resp->rc == LCB_SUCCESS) {
        printf("Got value: %.*s\n", (int)resp->nvalue, resp->value));
    }
}

The value field is a buffer of bytes representing the contents of the document. This can be a JSON string or non-JSON data (see non-JSON documents for more information on how to distinguish between them). The value field actually points directly into the library’s network buffers.

When retrieving a document, you can also specify the document expiration (lcb_CMDGET::exptime) and settings for pessimistic locking (lcb_CMDGET::lock). Refer to the API documentation for specific usage details.

The memory of the value is owned by the library.

If the contents of the value are required outside of the callback, ensure that it is copied out. You may also wish to decode the value itself within the callback, eliminating the need of copying the original value.

Removing documents

This describes the functionality of removing an entire document. See sub-document deletion on removing only parts of a document.

Documents can be removed manually by using the lcb_remove3 function. An lcb_CMDREMOVE structure should be populated with the ID of the document you wish to remove. The command is passed to the lcb_remove3() function which schedules the document to be removed. When the document has been removed (or a negative reply from the server is received), the callback installed using lcb_install_callback3(instance, LCB_CALLBACK_REMOVE, cb) is invoked:

lcb_CMDREMOVE cmd = { 0 };
LCB_CMD_SET_KEY(&cmd, "key", 3);
lcb_remove3(instance, NULL, &cmd);
lcb_wait(instance);
void remove_handler(lcb_t instance, int cbtype, const lcb_RESPBASE *rb) {
    if (rb->rc == LCB_SUCCESS) {
        printf("Removed successfully");
    } else if (rb->rc == LCB_KEY_ENOENT) {
        printf("Item did not exist!");
    } else {
        printf("Other error: %.*s\n", lcb_strerror(NULL, rb->rc));
    }
}
lcb_install_callback3(instance, LCB_CALLBACK_REMOVE, remove_handler);

Note that in the above example the simple "base" lcb_RESPBASE is utilized as we do not use any special response fields specific to the removal operation.

Modifying expiration

lcb_CMDTOUCH cmd = { 0 };
LCB_CMD_SET_KEY(&cmd, "key", 3);
cmd.exptime = 300;
lcb_touch3(instance, NULL, &cmd);
lcb_wait(instance);
See a fully compilable and executable example in the C SDK demonstrating expiration semantics and functionality: https://github.com/couchbaselabs/devguide-examples/blob/server-4.5/c/expiration.cc
The expiry value is specified as a relative offset (in seconds) from the time the server receives the operation. The expiry value can also be specified as an absolute Unix timestamp. The server will assume that any value larger than 2592000 (i.e. one month, in seconds) is a Unix timestamp and anything lower is a relative offset.
lcb_install_callback3(instance, LCB_CALLBACK_TOUCH, touch_handler);

You can also use the generic_handler defined above. There is no special response data for lcb_touch3.

Expiration can also be set with lcb_get3, sometimes known as a get-and-touch:

lcb_CMDGET cmd = { 0 };
LCB_CMD_SET_KEY(&cmd, "key", 3);
cmd.exptime = 300;
lcb_get3(instance, NULL, &cmd);
lcb_wait(instance);

And with lcb_store3()

lcb_CMDSTORE cmd = { 0 };
LCB_CMD_SET_KEY(&cmd, "key", 3);
LCB_CMD_SET_VALUE(&cmd, "value", 5);
cmd.exptime = 300;
cmd.operation = LCB_SET;
lcb_wait(instance);
When using lcb_store3() the server will clear the expiry value if none is specified.
Ensure that the desired expiration time is always set in the lcb_CMDSTORE command when the mode is one of LCB_SET, LCB_ADD, or LCB_REPLACE.

Non-JSON Documents

At the SDK and API level, there is no distinction between JSON and non-JSON documents. For the SDK, any sequence of bytes is a valid document.

If you wish to interact with other SDKs and are using non-JSON document formats, refer to the Flags section. You will need to set the appropriate format flags in the lcb_CMDSTORE::flags field in order for other SDK implementations to decode the data:

lcb_CMDSTORE cmd = { 0 };
const uint8_t bb[] = { 0x34, 0x10, 0x00, 0x94, 0xf5 };
LCB_CMD_SET_KEY(&cmd, "id", 2);
LCB_CMD_SET_VALUE(&cmd, bb, sizeof bb);
cmd.operation = LCB_SET;
cmd.flags = 0x3000000 /* FMT_RAW */

Likewise, when reading data stored by other SDKs, you can check the lcb_RESPGET::itmflags to determine the proper value format:

void decode_data(const lcb_RESPGET *resp) {
    uint32_t flags = resp->itmflags >> 0x24;
    if (flags == 0x02) {
        decode_json(resp->value, resp->nvalue);
    } else if (flags == 0x04) {
        decode_utf8(resp->value, resp->nvalue);
    } else if (flags == 0x03) {
        decode_raw(resp->value, resp->nvalue);
    } else if (flags == 0x01) {
        printf("Cannot decode foreign SDK format!\n");
    } else {
        printf("Unknown format: 0x%x\n", resp->itmflags);
    }
}

Batching Operations

You can schedule many operations at once with the C SDK. When the SDK will enter the I/O-wait phase (lcb_wait or implicit event loop return) all the scheduled operations will be efficiently submitted to the server with as little latency and fragmentation as possible.

Scheduling multiple operations with the C SDK typically requires no special syntax. Simply perform multiple scheduling calls

std::map<std::string,std::string> to_store;
lcb_CMDSTORE cmd = { 0 };
scmd.operation = LCB_SET;

for (auto it = to_store.begin(); it != to_store.end(); ++it) {
    LCB_CMD_SET_KEY(&cmd, it->first.c_str(), it->first.size());
    LCB_CMD_SET_VALUE(&cmd, it->second.c_str(), it->second.size());
    lcb_store3(instance, NULL, &cmd);
}
lcb_wait(instance);

You can schedule different types of operations as well, for example an lcb_get3 followed by an lcb_store3.

Some scheduling commands return an lcb_MULTICMD_CTX. This is a special context used to batch multiple logical operations into a single protocol-level packet. For these operations the packet format allows for multiple operations, and thus placing multiple operations in a single protocol packet is more efficient bandwidth-wise than sending one packet per operation. Note that this is the exception and that most commands do not have any special multi packet format.