Search Response

These pagination settings can be used to obtain a subset of results and works deterministically when combined with a certain sort order.

Using size/limit and offset/from would fetch at least size + from ordered results from a partition and then return the size number of results starting at offset from.

Deep pagination can therefore get pretty expensive when using size + from on a sharded index due to each shard having to possibly return large result sets (at least size + from) over the network for merging at the coordinating node before returning the size number of results starting at offset from.

The default sort order is based on score (relevance) where the results are ordered from the highest to the lowest score.

For an efficient pagination, you can use the search_after/search_before settings.

search_after is designed to fetch the size number of results after the key specified and search_before is designed to fetch the size number of results before the key specified.

These settings allow for the client to maintain state while paginating - the sort key of the last result (for search_after) or the first result (for search_before) in the current page.

Both the attributes accept an array of strings (sort keys) - the length of this array will need to be the same length of the "sort" array within the search request.

The pagination of search results can be done using the 'from' and 'size' parameters in the search request. But as the search gets into deeper pages, it starts consuming more resources.

To safeguard against any arbitrary higher memory requirements, FTS provides a configurable limit bleveMaxResultWindow (10000 default) on the maximum allowable page offsets. However, bumping this limit to higher levels is not a scalable solution.

To circumvent this problem, the concept of key set pagination in FTS, is introduced.

Instead of providing from as a number of search results to skip, the user will provide the sort value of a previously seen search result (usually, the last result shown on the current page). The idea is that to show the next page of the results, we just want the top N results of that sort after the last result from the previous page.

This solution requires a few preconditions be met:

A common solution to this is to always include the document ID as the final sort criteria.

For example, if you want to sort by [“name”, “-age”], instead of sort by [“name”, “-age”, "_id”].

With search_after/search_before paginations, the heap memory requirement of deeper page searches is made proportional to the requested page size alone. So it reduces the heap memory requirement of deeper page searches significantly down from the offset+from values.

FTS sorting is sorted by descending order of relevance. It can, however, be customized to sort by different fields, depending on the application.

On query-completion, sorting allows specified members of the result-set to be displayed prior to others: this facilitates a review of the most significant data.

Within a JSON query object, the required sort-type is specified by using the sort field.

This takes an array of either strings, objects, or numeric as its value.

You can specify the value of the sort field as an array of strings. These can be of three types:

Fine-grained control over sort-procedure can be achieved by specifying objects as array-values in the sort field.

Each object can have the following fields:

To fetch more accurate sort results, we strongly recommend specifying the type of the sort fields in the sort section of the search request.

You can specify the value of the sort field as a numeric type. You can use the type field in the object that you specify with the sort.

With type field, you can specify the type of the search order to numeric, string, or DateTime.

When you sort results on a field that is not indexed, or when a particular document is missing a value for that field, you will see the following series of Unicode non-printable characters appear in the sort field:

\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd

The same characters may render differently when using a graphic tool or command line tools like jq.

Check your index definition to confirm that you are indexing all the fields you intend to sort by. You can control the sort behavior for missing attributes using the missing field.

The following sample query response shows the score field for each document retrieved for the query request:

tf-idf, a short form of term frequency-inverse document frequency, is a numerical statistical value that is used to reflect how important a word is to a document in collection or scope.

tf-idf is used as a weighting factor in a search for information retrieval and text mining. The tf–idf value increases proportionally to the number of times a word appears in the document, and it is offset by the number of documents in the collection or scope that contains the word.

Search engines often use the variations of tf-idf weighting scheme as a tool in scoring and ranking a document’s relevance for a given query. The tf-idf scoring for a document relevancy is done on the basis of per-partition index, which means that documents across different partitions may have different scores.

When bleve scores a document, it sums a set of sub scores to reach the final score. The scores across different searches are not directly comparable as the scores are directly dependent on the search criteria. So, changing the search criteria, like terms, boost factor etc. can vary the score.

The more conjuncts/disjuncts/sub clauses in a query can influence the scoring. Also, the score of a particular search result is not absolute, which means you can only use the score as a comparison to the highest score from the same search result.

FTS does not provide any predefined range for valid scores.

In Couchbase application, you get an option to explore the score computations during any search in FTS.

On the Search page, you can search for a term in any index. The search result displays the search records along with the option Explain Scoring to view the score deriving details for search hits and which are determined by using the tf-idf algorithm.

You can disable the scoring by setting score to none in the search request. This is recommended in a situation where scoring (document relevancy) is not needed by the application.

For a select term, FTS calculates the relevancy score. So, the documents having a higher relevancy score automatically appear at the top in the result.

It is often observed that users are using Full-Text Search for the exact match queries with a bit of fuzziness or other search-specific capabilities like geo.

Text relevancy score does not matter when the user is looking for exact or more targeted searches with many predicates or when the dataset size is small.

In such a case, FTS unnecessarily uses more resources in calculating the relevancy score. Users can, however, optimize the query performance by skipping the scoring. Users may skip the scoring by passing a “score”: “none” option in the search request.

In this workflow, it is possible that the document written to vBucket is not yet indexed. So, when FTS starts searching that document, the most up-to-date document versions are not retrieved, and only the indexed versions are queried.

Therefore, the Couchbase server provides a consistency mechanism to overcome this issue and ensures that the FTS index can search the most up-to-date document written to vBucket.

The consistency level is a parameter that either takes an empty string indicating the unbounded (not_bounded) consistency or at_plus indicating the bounded consistency.

The consistency vectors supporting the consistency mechanism in Couchbase contain the mapping of the vBucket and sequence number of the data stored in the vBucket.

For more information about consistency mechanism, see Consistency

It is the amount of time (in milliseconds) the search service will allow for a query to execute at an index partition level.

If the query execution surpasses this timeout value, the query is canceled. However, at this point if some of the index partitions have responded, you might see partial results, otherwise no results at all.

Consistency result is the attribute that you can use to set the query result option, such as complete.

The complete option allows you to set the query result as "complete" which indicates that if any of the index partitions are unavailable due to the node not being reachable, the query will display an error in response instead of partial results.

Consistency vectors provide "read your own writes" functionality where the read operation waits for a specific time until the write operation is finished.

When users know that their queries are complex which require more time in completing the write operations, they can set the timeout value higher than the default timeout of 10 seconds so that consistency can be obtained in the search operations.

However, if this consistency is not required, the users can optimize their search operations by using the default timeout of 10 seconds.

For each facet that you build, a FacetResult is returned containing the following:

All of the facet examples given in this topic are for the query "water" on the beer-sample dataset.

FTS supports the following types of facets:

As per the following example, when you search the content in the index, the matched content in the address field is highlighted in the search response.

As per the following example, when you search the content in the index, the matched content in the description field is highlighted in the search response.

Total hits represent the total number of matches for this result. It can be any integer starting from 0.

It is possible that when some of the index partitions have reported failures, the "status" in the response does not show fail. This is when the user will see a partial result set for their query.

Below is the sample response

If one or more of the index partitions failed to cater to the request, the user can see partial results. In such a case, the response status is shown as 200 , and the errors object in the response will be a non-zero length value.

The status object includes the number of successful and failed partition indexes queries.

We recommend that the status object be checked for failures as a preferred alternative to relying on HTTP response codes alone. For example, FTS returns an HTTP 200 response in case of partition index failures or timeouts (not consistency timeouts). This is done so that you can choose to accept partial results in an application. However, this also means FTS returns an HTTP 200 response even when ALL partition indexes fail.

Refer to the following table for more information about the possible status messages.

Scoring of results, according to relevancy, allows users to obtain result-sets that only contain documents awarded the highest scores. This keeps result-sets manageably small, even when the total number of documents returned is extremely large. The response returns the score of the search query.

Scores are directly dependent on the search criteria. Changing the search criteria can vary the score, and therefore, scores are not comparable across searches.

For more details on scores, follow the link Scoring