.NET Sample App Backend Tutorial

The .NET SDK tutorial bridges the gap between simple and advanced concepts by walking through a complete web application.

The full source code for the tutorial is available on GitHub couchbaselabs/try-cb-dotnet. The primary focus of the tutorial is to explain the function and theory behind the Couchbase .NET client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code, but is not discussed in this tutorial.

Specific .NET prerequisites and set up

In addition to the prerequisites mentioned in Sample Application, you’ll need:

  • Visual Studio 2015 (Community is fine)

To get set up for the tutorial proper, follow these steps:

  • git clone https://github.com/couchbaselabs/try-cb-dotnet.git or download the source

  • if you don’t want to connect to localhost, change the couchbaseServer entry in src/try-cb-dotnet/Web.config

  • open the project in Visual Studio and build the project, nuget references will be restored automatically

If you want to code it yourself, the real work is done in the following classes:

There’s currently no "fill-in-the-blanks" branch so you’ll have to delete method bodies and try to take it from there.

This tutorial focuses on querying through N1QL and FTS rather than views. If you want information about using views, see Views.

Walking Through the API

The following sections lead you through the primary functions of the sample application. All of the REST API application code is in several files according to function. The following shows you how to program with the various features and services of Couchbase including: connecting to a cluster and bucket, key/value iteraction, document query through N1QL and full text searches.

Configure and Bootstrap the SDK

Goals: Connecting to the Cluster and getting familiar with ClusterHelper.

Relevant Documentation Topics: Managing Connections

The first step in the Couchbase SDK is to configure your connection to the cluster. This can be done either programattically or through configuration files. For simplicity, in this sample app we’re going to do it programmatically.

In the sample app, the CouchbaseConfig.Register() method is responsible for reading the hostnames and/or IP addresses for the cluster and initializing the connection. First it reads the CouchbaseServer setting from the web.config and then creates a ClientConfiguration, setting the Servers with the hostname from the app setting. The ClientConfiguration is then passed into the ClusterHelper utility which uses the configuration to setup everything required.

var couchbaseServer = ConfigurationManager.AppSettings.Get("CouchbaseServer");
ClusterHelper.Initialize(new ClientConfiguration
    Servers = new List { new Uri(couchbaseServer) }

The ClusterHelper is a utility built into the .NET SDK to help with managing the cluster connection and can be used to get references to a bucket (which is your entry point for the whole storage API). The cluster connection should be reused as much as possible, typically called a singleton, because it is expensive to create. The ClusterHelper stores the connection to the cluster for easy reuse within your applications.

To get a references to a bucket, with which you can then execute operations such as Get, Insert and Remove, you can use the GetBucket method on ClusterHelper. This returns an IBucket which exposes the majority of the operations available in Couchbase.

An example of getting a reference to a bucket and retrieving a document is below:

var bucket = ClusterHelper.GetBucket("default");
var user = bucket.Get<User>("user::" + 123); // retrieves the document 'user::123' and deserializes it into the application User class

Key/Value: Users

Goals: Use Bucket's Key/Value operations and discover the Document API.

Couchbase Server is a document-oriented database which provides access to your data both through its document ID (for high performance access), as well as through map/reduce views and N1QL (SQL like query language).

This is noticeable in the API, where the methods reflect Key/Value operations (get, create, etc...) and work with a Document<T> interface that has an Id and a Content.

Typically, in this type of scenario you would implement the Repository pattern to separte functional logic from business logic. However, to make this sample as easy to get up to speed with controllers are used instead.

Creating New Users

For the purposes of this sample application users are required to register before they can book flights. All the user related operations are going to exist in the UserController.

The SignUp method is where new users are registered. The method takes a LoginModel (src/Models/LoginModel.cs) as the only method parameter. ASP.NET has an automatic model binder where it can map a POST request’s properties to a custom class, like the LoginModel. The property types and names have to match for it to assign the value.

public async Task<IHttpActionResult> SignUp(LoginModel model)

The model is validated to ensure that the username and password fields are non-empty and then checked to see if the username already exists. For the purposes of this sample app, the user document keys are in the format user::<username>. The ExistsAsync method is used to do a non-blocking check to see if the document already exists in the bucket.

var userKey = CreateUserKey(model.Username);
if (await _bucket.ExistsAsync(userKey))
    return Content(HttpStatusCode.Conflict, new Error($"Username '{model.Username}' already exists"));

Next a document with the user details is created and stored in Couchbase Server. The userKey created earlier is used and alsoa document expiry time is included if one was set on the model. As you do not want to store plain text passwords, the passwords are MD5-hashed before storing it in the user document.

var userDoc = new Document<User>
    Id = userKey,
    Content = new User
        Username = model.Username,
        Password = CalcuateMd5Hash(model.Password)
    Expiry = model.Expiry

var result = await _bucket.InsertAsync(userDoc);

The non-blocking InsertAsync method is used to ensure that a thread waiting for the response from Couchbase Server is not blocked. Once the code receives the response, it continues where it left off.

The last thing to do is to create a security token for the browser, so that the front end knows that additional requests are for a valid user. For this a JWT (JSON Web Token) is used, which includes the username in a list of claims and is then encrypted with a secret key. The secret is stored in the Web.Config.

var data = new
    token = BuildToken(model.Username)
var context = $"Created user with ID '{userKey}' in bucket '{_bucket.Name}' that expires in {userDoc.Expiry}ms";
return Content(HttpStatusCode.Accepted, new Result(data, context));

The response content has two parts, the first is the JWT and the second part is a narration string which is something the frontend app understands and will display in a console. The narration enables the users of the application to get an idea of what is going on on the server side while browsing the app. It is similar to a log, but sent to the frontend.

Loging in Signed up Users

The Login method enables users who have already signed up to sign in and use the application. The Login method signature looks like this:

public async Task<IHttpActionResult> Login(LoginModel model)

The LoginModel includes Username and Password properties that can be used to find the user document and verify the passwords.

First the user document must be retrieved and the password checked to ensure that it matches with the model. The user document key needs to be built using the model’s username property, the document is then retrieved from Couchbase Server.

var userKey = CreateUserKey(model.Username);
var userDocument = await _bucket.GetDocumentAsync<User>(userKey);

Now the application has a user document, it can check the passwords match. Remember the password was hashed in the document for added security so the model’s password will also have to be hashed before they are compared.

var user = userDocument.Content;
if (user.Password != CalcuateMd5Hash(model.Password))
    return Content(HttpStatusCode.Unauthorized, new Error("Invalid username and/or password"));

All that is left to do is create the security token like was done for the SignUp method and return it. A narration to go in the response content is also created for the frontend app to report on.

var data = new
    token = BuildToken(user.Username)
var context = $"User {model.Username} logged in successfully";
return Content(HttpStatusCode.OK, new Result(data, context));

Getting a User’s Stored Flights

A way to retrieve the flights that a user has booked is required for the application. The GetFlightsForUser method does this.

public async Task<IHttpActionResult> GetFlightsForUser(string username)

This is going to be the first time that the application checks for a valid security token; it has only generated these so far. The security token is fairly simple and only includes the username of the user, but that is enough for this sample application. To verify the token the authentication header needs to be fetched and then decrypted. If this fails for any reason, the application returns either a 401 (Unauthorized) or a 403 (Forbidden) response.

var authHeaderValue = GetAuthHeaderValue(Request.Headers);
if (string.IsNullOrEmpty(authHeaderValue))
    return Content(HttpStatusCode.Unauthorized, string.Empty);
if (!VerifyToken(authHeaderValue, username))
    return Content(HttpStatusCode.Forbidden, string.Empty);

The username is passed in as one of the method parameters, so it can be used to create the user document key and get the document from Couchbase Server.

var userKey = CreateUserKey(username);
var userDocument = await _bucket.GetDocumentAsync<User>(userKey);

The final thing to do is return the list of flights for the user with some narration for the frontend application to record.

var data = userDocument.Content.Flights;
var context = $"Retrieved flights for user {username}.";
return Content(HttpStatusCode.OK, new Result(data, context));

N1QL: Flight Paths

Goals: Use N1QL to perform SELECT on Couchbase.

In the SDK, there is a query method that accepts all variants of querying with Couchbase (views, spatial/geo views, N1QL and FTS). For N1QL, the IQueryRequest is expected. This allows to wrap a N1QL Statement, use positional parameters and provide query tuning (eg Timeout).

N1QL is a super-set of SQL, so if you’re familiar with SQL you’ll feel at ease.

This controller has one method, GetFlights, which provides flight routes between two airports. It uses a N1QL query to get them. The method has three parameters; from, to and leave (string for departure date). The first thing the application does is validate the parameters, returning a 500 (InternalServerError) if it’s not.

if (string.IsNullOrEmpty(from) || string.IsNullOrEmpty(to))
    return Content(HttpStatusCode.InternalServerError, new Error("Missing or invalid from and/or to airports"));

DateTime leaveDate;
if (!DateTime.TryParse(leave, out leaveDate))
    return Content(HttpStatusCode.InternalServerError, new Error("Missing or invalid leave date"));

var dayOfWeek = (int) leaveDate.DayOfWeek + 1; // Get weekday number; Sun (0) to Sat (7)

Next the from and to parameters are used to get the airport FAA code along with its geo-location latitude and longitude.

var airportQuery = new QueryRequest()
    .Statement("SELECT faa AS fromAirport, geo.lat, geo.lon " +
        "FROM `travel-sample` " +
        "WHERE airportname = $1 " +
        "UNION " +
        "SELECT faa AS toAirport, geo.lat, geo.lon " +
        "FROM `travel-sample` " +
        "WHERE airportname = $2;")
    .AddPositionalParameter(from, to);
var airportQueryResult = await _bucket.QueryAsync<dynamic>(airportQuery);

After doing some checks to ensure there are results from both airport codes, the application then does some Geo-location calculations to find the distance between the two airports and the estimated travel time. The distance and travel time are then used when calculating ticket prices.

var fromCoordinate = new GeoCoordinate((double) fromAirport.lat, (double) fromAirport.lon);
var toCoordinate = new GeoCoordinate((double) toAirport.lat, (double) toAirport.lon);
var distance = fromCoordinate.GetDistanceTo(toCoordinate);
var flightTime = Math.Round(distance/AverageFlightSpeed, 2);

Next, all flights between the the two airports have to be retrieved.

var flightQuery = new QueryRequest()
    .Statement("SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment " +
        "FROM `travel-sample` AS r " +
        "UNNEST r.schedule AS s " +
        "JOIN `travel-sample` AS a ON KEYS r.airlineid " +
        "WHERE r.sourceairport = $1 " +
        "AND r.destinationairport = $2 " +
        "AND s.day = $3 " +
        "ORDER BY a.name ASC;")
    .AddPositionalParameter((string) fromAirport.fromAirport, (string) toAirport.toAirport, dayOfWeek);
Yes, you read that right, N1QL can do joins (on a single bucket or on several). It works as long as the "foreign key" described by ON KEYS clause can be mapped to a document’s key in the joined bucket.

A specificity of N1QL that is seen in the second statement is UNNEST. It extracts a sub-JSON and puts it at the same root level as the bucket (so its possible to do joins on each element in this sub-JSON as if they were entries in a left-hand side bucket).

The application now has all flights between the from and to airports but there are not any prices any prices. These are then calculated.

var flights = flightQueryResult.Rows;
foreach (var flight in flights)
    flight.FlightTime = flightTime;
    flight.Price = _random.Next(2000);

LINQ: Airports

Goals: Use the LINQ provider to build N1QL queries to retrieve Airport details.

LINQ is a standardised way of constructing queries over a data storage engine, such as in-memory collections, SQL and even NoSQL like Couchbase. It’s a very simple yet powerful tool that enables developers to write complicated queries programatically.

In this Controller the application is trying to find the aiport name, given some additional information about the airport. It uses the LINQ provider to build the queries.

The first query looks for an airport using its FAA code.

airports = _context.Query<Airport>()
    .Where(x => x.Faa == search.ToUpper())
    .Select(x => x.Airportname);
"SELECT airportname FROM `travel-sample` WHERE type = 'airport' AND faa = '{search.ToUpper()}'"

The second query looks for an airport using its ICAO code.

airports = _context.Query<Airport>()
    .Where(x => x.Icao == search.ToUpper())
    .Select(x => x.Airportname);
"SELECT airportname FROM `travel-sample` WHERE type = 'airport' AND icao = '{search.ToUpper()}'"

The third query looks for an airport using its name.

airports = _context.Query<Airport>()
    .Where(x => x.Airportname.Contains(search))
    .Select(x => x.Airportname);
"SELECT airportname FROM `travel-sample` WHERE type = 'airport' AND airportname LIKE '%{search}%'"

Once one of the above queries has been executed, the result then needs to be returned to the frontend application along with a narration of the query that was executed.

var data = airports.Select(airportname => new {airportname});
return Content(HttpStatusCode.OK, new Result(data, query));

Indexing the Data: N1QL & GSI

Index management is a bit more advanced (and is already done when loading the sample), so now that you’ve learned about N1QL, you can have a look. For N1QL to work, you must first ensure that at least a Primary Index has been created. For that you can use the DSL from the BucketManager class:

Goals: Use the Index DSL to make sure data is indexed ready for N1QL to query it.

bucketManager.CreateN1qlPrimaryIndex(false); // create primary index, and don't defer building it

The fluent API will guide you with the available options, you just have to declare that you want to CreateN1qlPrimaryIndex().

You can also create secondary indexes on specific fields of the JSON, for better performance:

bucketManager.CreateN1qlIndex("index_name", false, "name", "address", etc);

In this case, give a name to your index, specify if the index is to be deferred for building then an array of property names to index.

Full Text Search: Finding Hotels

Goals: Use FTS to search for matching Hotels. Use subdoc API to fetch the relevant data for each hit.

In this service, hotels are searched for using more fuzzy criterias, like the content of the address or the description of an hotel. This is done using Full Text Search (FTS). When some results match the specified criteria, only the relevant data for each result to be displayed in the UI is fetched using the subdocument API.

Let’s have a look at the FindHotel method. It accepts two parameters, location and description, which are the two possible refining criterias for an hotel search.

public HttpResponseMessage FindHotel(string description = null, string location = null)

A ConjunctionQuery allows you to combine multiple FTS queries into one, in a logical AND fashion. This query includes an exact match criteria that restricts it to the hotel data type (as reflected in the type field of the JSON document).

var query = new ConjunctionQuery(
    new TermQuery("hotel").Field("type")

If the user provided a location keyword, a second component is added to the FTS query that will look for that keyword in several address-related fields of the document. That is done in an OR fashion, using a Disjunction query:

if (!string.IsNullOrEmpty(location) && location != "*")
    query.And(new DisjunctionQuery(
        new PhraseQuery(location).Field("address"),
        new PhraseQuery(location).Field("city"),
        new PhraseQuery(location).Field("state"),
        new PhraseQuery(location).Field("country")

Similarly, if a description keyword was provided by the user, the freeform text is inspected of the description field and name field of the document:

if (!string.IsNullOrEmpty(description) && description != "*")
    query.And(new DisjunctionQuery(
        new PhraseQuery(description).Field("name"),
        new PhraseQuery(description).Field("description")

The matchPhrase FTS query can contain several words and will search for variations of the words (eg. including plural forms or words with the same root).

The compound FTS query is now ready to be executed. A SearchQuery object is built out of it, which also determines which FTS index to use ("hotel") and allows you to set various parameters (like a limit of maximum 100 hits to return). The query is logged (and kept for narration) then executed, returning an ISearchQueryResult object:

var search = new SearchQuery();
search.Index = "hotel";
search.Query = query;

The FTS results are then iterated over, and the document corresponding to each result is fetched. In actuality, only the parts of the document that will be displayed in the UI are required. This is where the sub-document API comes in.

The sub-document API allows you to fetch or mutate only a set of paths inside a JSON document, without having to send the whole document back and forth. This can save network bandwidth if the document is large and the parts that we’re interested in are small. So here the results of the FTS search are iterated over and appropriate subdoc calls are triggered:

var result = _bucket.Query(search);
foreach (var row in result)
    var fragment = _bucket.LookupIn<Hotel>(row.Id)

Each FTS result is represented as an ISearchQueryRow which exposes the document’s Id. The sub-document API can then be used to fetch data (bucket.LookupIn<T>(documentId)) and specify what parts are wanted: name, description, address, city, state and country. The application then Execute() the sub-document query. In the rest of the code, the address-related fields are aggregated together and the data obtained is returned.

Now the results are obtained, the application can build up the Hotel objects and return them along with the FTS query narration.

        name = fragment.Value.Name,
        description = fragment.Value.Description,
        address = fragment.Value.GetFullAddress()

return Request.CreateResponse(new Result(hotels, queryJson));