Manage Full Text Indexes

    +
    Full text searches are supported by specially purposed indexes, which can be created and managed from the Couchbase Cloud UI.

    Every Full Text Search is performed on a user-created Full Text Index, which contains the targets on which searches are to be performed: these targets are values derived from the textual and other contents of documents within a specified bucket.

    Index-creation is configurable, and can be highly selective: documents can be user-grouped into different types (for example, airline documents versus hotel documents), based on the document-IDs or the values of a designated document-field; and each document-type can be assigned its own index-mapping. Each index-mapping in turn can be assigned its own analyzers, can be applied to a specific subset of document-fields, and can be explicitly included in or excluded from the index.

    Accessing Full Text Indexes in the Couchbase Cloud UI

    If a cluster is running the Search Service, Full Text Indexes can be accessed under the cluster’s Tools > Full Text Search tab. (Full Text Indexes are different from the Global indexes that are accessed under the Tools > Indexes tab.)

    The cluster’s 'Tools > Full Text Search' tab.
    Permissions Required

    In order to access Full Text Indexes in the Couchbase Cloud UI, the following permissions are required:

    • You must have Project View privileges for the project that contains the cluster.

    • You must have a database user associated with your organization user account.

      Without a database user, the Tools > Full Text Search tab will display a message saying that you don’t have the proper permissions to access Full Text Indexes.

      In additions, only Full Text Indexes on buckets for which the database user has at least Read permissions will appear in the Tools > Full Text Search tab.

    Index Summary

    A cluster’s Tools > Full Text Search tab shows a summary of all the Full Text Indexes on the cluster. The summary is displayed in table format, with sortable columns and a row for each index.

    The Full Text Indexes list displays the following information about each index:

    Index Name

    The name of the Full Text Index.

    Bucket

    The bucket for which the Full Text Index was created.

    Doc Count

    Number of documents examined across active index partitions.

    Indexing Progress

    The build-progress of the index. When 100% is reached, the index build is considered complete. An index can be searched as soon as it’s created, but only partial results can be expected until the index build is complete.

    Indexing progress is determined by index_doc_count / source_doc_count, with index_doc_count being retrieved from the Search Service endpoint and source_doc_count being retrieved from a Data Service endpoint. In the event that one or more Data Service nodes in the cluster become unavailable and/or become failed over, the indexing progress may show a value greater than 100% since source_doc_count for the bucket would be missing some active partitions.

    Create a Full Text Index

    To create a Full Text Index, the database user associated with your organization user account must have Read/Write permissions for the bucket on which you are creating the index.

    You can also create a Full Text Index by importing a JSON-formatted index definition. This can be helpful if you want to create a new index that is very similar to an existing index, or if you want to re-create an index from another cluster.
    1. Go to the cluster’s Tools > Full Text Search tab.

      1. Go to the Clusters tab in the main navigation.

      2. Find and click on the cluster that contains the bucket on which you plan to create a Full Text Index.

        This opens the cluster with its Overview tab selected.

      3. Click the Tools > Full Text Search tab.

    2. Click Add Index.

      This opens the Add Index fly-out menu.

      The 'Add Index' fly-out menu.
    3. Specify a name for the index.

      In the Name field, enter a unique name for the index. Only alphanumeric characters, hyphens, and underscores are allowed for Full Text Index names. Note also that the first character of the name must be an alphabetic character.

    4. Use the Bucket drop-down menu to select the bucket on which the index is to be created.

    5. (Optional) Configure advanced options.

      Advanced settings can be specified by expanding the Advanced Options panel.

      The expanded 'Advanced Options' section of the 'Add Index' fly-out menu.

      The following interactive fields are displayed:

      • Default Type: The default type for documents in this bucket. The default value for this field is _default.

      • Default Analyzer: The default analyzer to be used for this bucket. The default value is standard. Clicking the drop-down menu displays a list of available options.

      • Default Date/Time Parser: The default date/time parser to be used for this bucket. The default value is dateTimeOptional. Clicking the drop-down menu displays a list of available options.

      • Default Field: The default field for this bucket. the default value is _all.

      • Store Dynamic Fields: When checked, ensures inclusion of field-content in returned results. When unchecked, no such inclusion occurs.

      • Index Dynamic Fields: When checked, ensures dynamic fields are indexed. When unchecked, they are not indexed.

    6. To begin creating the Full Text Index, click Create Index.

      The fly-out menu closes and the new index appears in the Full Text Indexes section. Note the percentage figure in the Indexing Progress column. This figure is incremented in correspondence with the build-progress of the index. When 100% is reached, the index build is considered complete. An index can be searched as soon as it’s created, but only partial results can be expected until the index build is complete.

      The index is created without a default type mapping. This means that no document or field is indexed (and thus no documents will be returned if you try to perform a search) until you specify a type mapping.

    Import a Full Text Index

    You can create a Full Text Index by importing a JSON-formatted index definition. This can be helpful if you want to do things like:

    • Clone an index from the same cluster

    • Re-create an index from another cluster, such as another cluster in Couchbase Cloud or a self-managed Couchbase Server cluster

    To import a Full Text Index, the database user associated with your organization user account must have Read/Write permissions for the bucket on which you are creating the index.

    1. Go to the cluster’s Tools > Full Text Search tab.

      1. Go to the Clusters tab in the main navigation.

      2. Find and click on the cluster that contains the bucket on which you plan to import the Full Text Index.

        This opens the cluster with its Overview tab selected.

      3. Click the Tools > Full Text Search tab.

    2. (Optional) Copy the index definition of an existing index.

      To copy the definition of an existing index, find and click on the index that you wish to copy. On the index’s configuration page, click Copy Index to Clipboard.

      You can also copy an index definition from Couchbase Server. In the Couchbase Web Console, start by going to the Search tab in the left-hand navigation. Find and click on the index that you wish to copy, and in the expanded configuration menu, click Show index definition JSON. Copy the index definition by clicking Copy to Clipboard.

      The 'Search' tab of the Couchbase Web Console, showing an index with the 'Show index definition JSON' panel expanded.
    3. From the Tools > Full Text Search tab, click Import Index.

      This opens the Import FTS Index fly-out menu.

      The 'Import FTS Index' fly-out menu.
    4. Specify a name for the index.

      In the Index Name field, enter a unique name for the index. Only alphanumeric characters, hyphens, and underscores are allowed for Full Text Index names. Note also that the first character of the name must be an alphabetic character.

    5. Use the Bucket drop-down menu to select the bucket on which the index is to be created.

    6. In the code editor field, enter or paste the index definition.

      Note that the "name" and "sourceName" values in any imported index definition will be overridden by what was specified for Index Name and Bucket in the previous steps.

    7. Once you’re satisfied with the configuration, click Create.

      The fly-out menu closes and the new index appears in the Full Text Indexes section. Note the percentage figure in the Indexing Progress column. This figure is incremented in correspondence with the build-progress of the index. When 100% is reached, the index build is considered complete. An index can be searched as soon as it’s created, but only partial results can be expected until the index build is complete.

    After importing an index, you can view and modify its configuration from the index’s configuration page.

    View and Modify a Full Text Index

    To modify a Full Text Index, the database user associated with your organization user account must have Read/Write permissions for the bucket on which the index was created.

    1. Go to the cluster’s Tools > Full Text Search tab.

      1. Go to the Clusters tab in the main navigation.

      2. Find and click on the cluster that contains the Full Text Index that you wish to modify.

        This opens the cluster with its Overview tab selected.

      3. Click the Tools > Full Text Search tab.

    2. Find and click on the index that you wish to modify.

      This opens the index’s configuration page.

      The configuration page of a Full Text Index.

    You can take several actions from the index’s configuration page.

    Two buttons are displayed in the upper-right:

    • Copy Index to Clipboard: Copies the JSON-formatted index definition of the current index. The copied index definition can be used to import a duplicate Full Text Index on the same or different cluster.

    • Search Index: Opens the Search fly-out menu, which allows you to search the current index.

    The Index Settings section displays controls for the following:

    Search a Full Text Index

    To search a Full Text Index, the database user associated with your organization user account must have Read permissions for the bucket on which the index was created.

    Full Text Indexes that are created through the Couchbase Cloud UI do not have a default type mapping (unless you imported an index definition that already had type mappings configured). Searching a Full Text Index will not return any results until you modify the index configuration to add a type mapping.
    1. Go to the cluster’s Tools > Full Text Search tab.

      1. Go to the Clusters tab in the main navigation.

      2. Find and click on the cluster that contains the Full Text Index that you wish to search.

        This opens the cluster with its Overview tab selected.

      3. Click the Tools > Full Text Search tab.

    2. Find and click on the index that you wish to search.

      This opens the index’s configuration page.

      The configuration page of a Full Text Index.
    3. Click Search Index.

      This opens the Search fly-out menu.

      The 'Search' fly-out menu.
    4. Enter a search query into the Search field.

      As you type, documents matching the current query will show up as search results, listed by their document ID.

      The 'Search' fly-out menu showing a query for 'restaurants'.

      The Search field supports Query String Queries. A term without any other syntax is interpreted as a match query for the term in the default field. The default field is _all. For example, pool performs a match query for the term pool.

    Delete a Full Text Index

    To delete a Full Text Index, the database user associated with your organization user account must have Read/Write permissions for the bucket on which the index was created.

    1. Go to the cluster’s Tools > Full Text Search tab.

      1. Go to the Clusters tab in the main navigation.

      2. Find and click on the cluster that contains the Full Text Index that you wish to delete.

        This opens the cluster with its Overview tab selected.

      3. Click the Tools > Full Text Search tab.

    2. Find the index that you wish to delete, and click the Trash icon at the end of the row on the right side.

    3. When prompted to confirm the deletion, click Confirm.

      The index is deleted and removed from the list of indexes on the Tools > Full Text Search tab.

    Deleting a bucket also deletes any Full Text Indexes configured on that bucket.

    Specifying Type Mappings

    Whereas a type identifer tells the index how to determine the position in each document of the characters that specify the document’s type, a type mapping specifies the characters themselves.

    Type mappings can be configured after an index has been created from the index’s configuration page. Type mappings are located in the Index Settings section of the page. Unless the index was imported with existing type mappings, the type mappings section will be empty.

    The 'Type Mappings' section showing no existing type mappings, and a button labeled

    Click Add Mapping. The Add Mapping fly-out menu appears.

    The 'Add Mapping' fly-out menu.

    When the Enabled checkbox is selected, it means that the type mapping is considered when building the index. Unselecting this checkbox will allow the type mapping to be saved, but it will not be considered when the index is built.

    To specify a type mapping, enter an appropriate string (for example, hotel) into the Name field. Note the Only index specified fields checkbox: if this is checked, only user-specified fields from the document are included in the index. (For an example, refer to the Specifying Fields section below.)

    Each type mapping can be either dynamic, meaning that all fields are considered available for indexing; or only index specified fields, meaning that only fields specified by the user are indexed. Leaving Only index specified fields unselected means that the type mapping will be dynamic. For information on how values are data-typed when dynamic mapping is specified, refer to the Document-Fields and Data-Types section below.

    Optionally, the Default Analyzer dropdown menu can be used to specify an analyzer for the type mapping. For all queries that do indeed support use of an analyzer, the specified analyzer will be applied, rather than the default analyzer (which is itself specified in the Advanced Options section during index creation). The default value, inherit, means that the type mapping inherits the default analyzer.

    The 'Add Mapping' fly-out menu showing the expanded 'Default Analyzer' dropdown menu.

    Once you’re satisfied with the configuration, click Create Mapping to save the type mapping. The new type mapping is displayed in the Type Mappings section as follows:

    The 'Type Mappings' section showing the newly created type mapping.

    To edit or delete a type mapping, start by clicking the vertical ellipses next to the desired type mapping. From the menu that appears, click either Edit Mapping or Delete Type Mapping to correspondingly edit or delete the type mapping.

    The 'Type Mappings' section showing the expanded menu after clicking the vertical ellipses next to a particular type mapping.

    Specifying Fields

    A Full Text Index can be defined not only to include (or exclude) documents of a certain type, but also to include (or exclude) specified fields within each of the typed documents.

    To specify a field, start by clicking the vertical ellipses next to the type mapping.

    The 'Type Mappings' section showing the expanded menu after clicking the vertical ellipses next to a particular type mapping.

    In the menu that appears, select one of the following options for specifying fields:

    Inserting a Child Field

    The option Insert Child Field allows a field to be individually included for (or excluded from) indexing, provided that it contains a single value or an array, rather than a JSON object. Selecting this option opens the Add Child Field fly-out menu:

    The 'Add Child Field' fly-out menu.

    The interactive fields and checkboxes are:

    • Field: The name of any field within the document that contains a single value or an array, rather than a JSON object.

    • Type: The data-type of the value of the field. This can be text, number, datetime, boolean, disabled, or geopoint; and can be selected from the field’s dropdown menu.

    • Searchable As: Typically identical to the Field (and dynamically supplied during text-input of the Field-value). This can be modified, to indicate an alternative field-name, whose associated value thereby becomes included in the indexed content, rather than that associated with the field-name specified in Field.

    • Analyzer: An analyzer optionally to be used for the field. The list of available analyzers can be displayed by means of the field’s dropdown menu, and so selected from.

    • Index: When checked, the field is indexed; when unchecked, the field is not indexed. This may be used, therefore, to explicitly remove an already-defined field from the index.

    • Store: When checked, the field-content is included in the set of values returned from a search; when unchecked, the field-content is not so included. Note that inclusion of field-content specifically permits highlighting of results, so that matched expressions can be easily seen; and generally assists in debugging procedures. However, it also results in larger indexes and longer processing-times.

    • Include in _all field: When checked, the field is included in the definition of _all, which is the field specified by default in the Advanced Options panel. When unchecked, the field is not so included. Inclusion means that when query strings are used to specify searches, the text in the current field is searchable without the field-name requiring a prefix (thus, a search on description:modern can be accomplished simply by specifying modern).

    • Include term vectors: When checked, term vectors are included. When unchecked, term vectors are not included. Term vectors are the locations of terms in a particular field. Certain kinds of functionality (such as highlighting, and phrase search) require term vectors. Inclusion of term vectors results in larger indexes and correspondingly slower index build-times.

    • Docvalues: When checked, the value for each instance of the field is itself included in the index. This provides essential support for Search Facets, and for the sorting of search results based on field values: refer to Sorting Query Results. Note that whenever this checkbox is checked, the resulting index will be correspondingly larger. If it is unchecked, the values are not added to the index; and in consequence, neither Search Facets nor value-based result-sorting is supported.

    Note that when the value of the specified field is an array, the array-values are all indexed and searched individually: no special configuration is required.

    Once you’re satisfied with the configuration, click Create Child Field to save the child field. The new child field is displayed under the type mapping as follows:

    The 'Type Mappings' section showing a type mapping with a child field.

    To edit or delete a child field definition, start by clicking the vertical ellipses next to the desired child field. From the menu that appears, click either Edit Mapping or Delete Type Mapping to correspondingly edit or delete the child field.

    The 'Type Mappings' section showing the expanded menu after clicking the vertical ellipses next to a particular child field.

    Inserting a Child Mapping

    The option Insert Child Mapping specifies a document-field whose value is a JSON object. Selecting this option opens the Add Child Mapping fly-out menu:

    The 'Add Child Mapping' fly-out menu.

    The interactive fields and checkboxes are:

    • Enabled: When this checkbox is selected, it means that the child mapping is considered when building the index. Unselecting this checkbox will allow the child mapping to be saved, but it will not be considered when the index is built.

    • Name: The name of a field whose value is a JSON object.

    • Default Analyzer: Optionally specify an analyzer for the type mapping For all queries that do indeed support use of an analyzer, the specified analyzer will be applied, rather than the default analyzer (which is itself specified in the Advanced Options section during index creation).

    • Only index specified fields: When checked, only fields explicitly specified are added to the index. Note that the JSON object specified as the value for Name has multiple fields of its own. Checking this box ensures that all or a subset of these can be selected for indexing.

    Once you’re satisfied with the configuration, click Create Child Mapping to save the child mapping. The new child mapping is displayed under the type mapping as follows:

    The 'Type Mappings' section showing a type mapping with a child mapping.

    Note that reviews is a field within the hotel-type documents of the travel-sample bucket whose value is a JSON object.

    If you click the vertical ellipses next to the child mapping, a menu appears containing the following options:

    The 'Type Mappings' section showing the expanded menu after clicking the vertical ellipses next to a child mapping.

    The Edit Mapping and Delete Type Mapping options can be used to correspondingly edit or delete the child field. The Insert Child Mapping and Insert Child Field options are present because reviews is an object that contains child-fields; which can now themselves be individually indexed. Adding a child-field, such as content, can be specified:

    The 'Type Mappings' section showing a type mapping with a child mapping that itself has a child field defined.

    Document-Fields and Data-Types

    During index creation, for each document-field for which the data-type has not been explicitly specified (which is to say, text, number, datetime, boolean, disabled, or geopoint), the field-value is examined, and the best-possible determination made, as follows:

    Type of JSON value Indexed as...

    Boolean

    Boolean

    Number

    Number

    String containing a date

    Date

    String (not containing a date)

    String

    Note that the indexer attempts to parse String date-values as dates, and indexes them as such if the operation succeeds. Note, however, that on query-execution, Full Text Search expects dates to be in the format specified by RFC-3339, which is a specific profile of ISO-8601.

    Note also that String values such as 7 or true are not respectively indexed as numbers or Booleans: they remain as Strings.

    The number-type is modeled as a 64-bit floating-point value internally.