Importing Existing Data into Couchbase Server Community Edition

    +

    This tutorial will guide you through the insertion of existing data into Couchbase Community Edition. To achieve this, you will use some tools for importing data from different sources into a Couchbase cluster, all of them provided by Couchbase CE.

    Import JSON Data into Couchbase Server CE

    To import data from a JSON file, you will use the cbdocloader tool, which is used to load one or a group of JSON document in a given directory or a group of JSON documents in a single .zip file into a Couchbase bucket.

    The following line shows the basic syntax of cbdocloader:

    cbdocloader -c [host]:8091 -u [Administrator] -p [password] -b [bucket-name] -m [bucket-quota-MB] -d [source-file]

    For more information about cbdocloader’s command options, refer to the Couchbase documentation for cbdocloader tool.

    Description

    To achieve correct data upload all JSON files must contain well-formatted JSON and the file names should exclude spaces.

    Types of Errors:

    • Files are not well-formatted

    • Credentials are incorrect

    • RAM quota for a new bucket to contain the JSON is too large given the current quota for Couchbase Server

    The cbdocloader can be found in the following locations, depending on the platform:

    • Linux: /opt/couchbase/bin/cbdocloader

    • Windows: C:\Program Files\Couchbase\Server\bin\cbdocloader

    • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbdocloader

    Example

    The following code is an example of uploading a JSON file, as you can see, it is only necessary to specify the directory in which the JSON file is located, so if there is more than one JSON file in this directory cbdocloader will also upload it.

    cbdocloader -c localhost:8091 -u Administrator -p password -b colors-bucket -m 100 -d /home/couchbase/import-data/resources/json

    Other way to upload a group of JSON files at the same time, is create a .zip with them, and upload the .zip with the same previous code, just changing the source-file, as shown below:

    cbdocloader -c localhost:8091 -u Administrator -p password -b colors-bucket -m 100 -d /home/couchbase/import-data/resources/json/colors-zip.zip

    Once these commands have been executed you can see a successful data upload confirmation:

    cbdocloader -c localhost:8091 -u Administrator -p password -b colors-bucket -m 100 -d /home/couchbase/import-data/resources/json/colors-zip.zip
    Data loaded successfully

    Import CSV Data into Couchbase Server CE

    To import data from a CSV file, you can use cbtransfer. This tool is a lightweight extract-transform-load (ETL) tool that transfers data between clusters and to and from files. The source and destination parameters are similar to URLs or file paths. The syntax to load data from a CSV file using this tool is shown below:

    cbtransfer [source-file] [host]:8091 -B [bucket-name] -u [Administrator] -p [password]

    For more information about cbtransfer’s command options, refer to the Couchbase documentation for cbtransfer-tool.

    Description

    To accomplish a correct data upload every CSV file must contain an id field in the first line of CSV and this field needs to be unique, because in the case that it is not, its value will be overwritten.

    cbtransfer can only restore files that where generated by the same version of cbtransfer, if a file that was not generated by the tool is given it will fail.

    The cbtransfer can be found in the following locations, depending on the platform:

    • Linux: /opt/couchbase/bin/cbtransfer

    • Windows: C:\Program Files\Couchbase\Server\bin\cbtransfer

    • Mac OS X: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbtransfer

    Example

    Data is imported into Couchbase Server as documents, in the case of CSV file, each line is imported as an independent document.

    The following code is an example of uploading a CSV file:

    cbtransfer /home/couchbase/import-data/resources/csv/people_name-age_127.0.1.1%3A8091.csv http://localhost:8091 -B name-age -u Administrator -p password

    After executing this command line you will see a successful data upload confirmation:

    cbtransfer /home/couchbase/import-data/resources/csv/people_name-age_127.0.1.1%3A8091.csv http://localhost:8091 -B name-age -u Administrator -p password
    .
    bucket: people_name-age_127.0.1.1%3A8091.csv, msgs transferred...
           :                total |       last |    per sec
     byte  :                   48 |         48 |      796.2
    done

    Next Steps

    We recommend you to follow our next tutorials, go to the Getting Started with Couchbase Community Edition page to find the full list.

    Also, you could review Couchbase Documentation to learn more about all sorts of topics.