A newer version of this documentation is available.

View Latest

Overview

  • reference
    +
    With the SELECT statement, you can query and manipulate JSON data. You can select, join, project, nest, unnest, group, and sort in a single SELECT statement.

    The SELECT statement takes a set of JSON documents from keyspaces as its input, manipulates it and returns a set of JSON documents in the result array. Since the schema for JSON documents is flexible, JSON documents in the result set have flexible schema as well.

    A simple query in N1QL consists of three parts:

    • SELECT: specifies the projection, which is the part of the document that is to be returned.

    • FROM: specifies the keyspaces to work with.

    • WHERE: specifies the query criteria (filters or predicates) that the results must satisfy.

    To query on a keyspace, you must either specify the document keys or use an index on the keyspace.

    Prerequisites

    The user executing the SELECT statement must have the Query Select privileges granted on all keyspaces referred in the query. For more details about user roles, see Authorization.

    For example,

    To execute the following statement, the user does not need any special privileges.

    SELECT 1

    To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.airline.

    SELECT * FROM `travel-sample`.inventory.airline;

    To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.route and `travel-sample`.inventory.airline.

    SELECT * FROM `travel-sample`.inventory.route
    JOIN `travel-sample`.inventory.airline
    ON KEYS route.airlineid
    WHERE route.airlineid IN ["airline_330", "airline_225"]

    To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.airport and `travel-sample`.inventory.landmark.

    SELECT * FROM `travel-sample`.inventory.airport
    WHERE city IN (SELECT RAW city FROM `travel-sample`.inventory.landmark);

    To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.hotel and `travel-sample`.inventory.landmark.

    SELECT * FROM `travel-sample`.inventory.hotel WHERE city = "Gillingham"
    UNION
    SELECT * FROM `travel-sample`.inventory.landmark WHERE city = "Gillingham";

    The following example uses an index to query the keyspace for airports that are in the America/Anchorage timezone and at an altitude of 2100ft or higher, and returns an array with the airport name and city name for each airport that satisfies the conditions.

    Query
    SELECT t.airportname, t.city
    FROM   `travel-sample`.inventory.airport t
    WHERE  tz = "America/Anchorage"
           AND geo.alt >= 2100;
    Results
    [
      {
            "airportname": "Anaktuvuk Pass Airport",
            "city": "Anaktuvuk Pass",
      }
    ]

    The next example queries the keyspace using the document key "airport_3469".

    Query
    SELECT * FROM `travel-sample`.inventory.airport USE KEYS "airport_3469";
    Results
    [
      {
        "airport": {
          "airportname": "San Francisco Intl",
          "city": "San Francisco",
          "country": "United States",
          "faa": "SFO",
          "geo": {
            "alt": 13,
            "lat": 37.618972,
            "lon": -122.374889
          },
          "icao": "KSFO",
          "id": 3469,
          "type": "airport",
          "tz": "America/Los_Angeles"
        }
      }
    ]

    With projections, you retrieve just the fields that you need and not the entire document. This is especially useful when querying for a large dataset as it results in shorter processing times and better performance.

    The SELECT statement provides a variety of data processing capabilities such as filtering, querying across relationships using JOINs or subqueries, deep traversal of nested documents, aggregation, combining result sets using operators, grouping, sorting, and more. Follow the links for examples that demonstrate each capability.

    SELECT Statement Processing

    The SELECT statement queries a keyspace and returns a JSON array that contains zero or more objects.

    The following diagram shows the query execution workflow at a high level and illustrates the interaction with the query, index, and data services.

    1. Client submits the query over the REST API; 2. Query Service parses, analyzes, and creates plan; 3. Query Service sends request and index filters to Index Service; 4. Index Service returns qualified doc keys to Query Service; 5. Query Service sends fetch request with doc keys to Data Service; 6. Data Service returns documents to the Query Service; 7. Query Service evaluates the documents; 8. Query Service returns result to client
    Figure 1. Query Execution Workflow

    The SELECT statement is executed as a sequence of steps. Each step in the process produces result objects that are then used as inputs in the next step until all steps in the process are complete. While the workflow diagram shows all the possible phases a query goes through before returning a result, the clauses and predicates in a query decide the phases and the number of times that the query goes through. For example, sort phase can be skipped when there is no ORDER BY clause in the query; scan-fetch-join phase will execute multiple times for correlated subqueries.

    The following diagram shows the possible elements and operations during query execution.

    Parse, Plan, Scan, Fetch, Join, Filter, Pre-Aggregate, Aggregate, Sort, Offset, Limit, Project
    Figure 2. Query Execution Phases

    Some phases are done serially while others are done in parallel, as specified by their parent operator.

    The below table summarizes all the Query Phases that might be used in an Execution Plan:

    Query Phase Description

    Parse

    Analyzes the query and available access path options for each keyspace in the query to create a query plan and execution infrastructure.

    Plan

    Selects the access path, determines the Join order, determines the type of Joins, and then creates the infrastructure needed to execute the plan.

    Scan

    Scans the data from the Index Service.

    Fetch

    Fetches the data from the Data Service.

    Join

    Joins the data from the Data Service.

    Filter

    Filters the result objects by specifying conditions in the WHERE clause.

    Pre-Aggregate

    Internal set of tools to prepare the Aggregate phase.

    Aggregate

    Performs aggregating functions and window functions.

    Sort

    Orders and sorts items in the resultset in the order specified by the ORDER BY clause

    Offset

    Skips the first n items in the result object as specified by the OFFSET clause.

    Limit

    Limits the number of results returned using the LIMIT clause.

    Project

    Receives only the fields needed for final displaying to the user.

    The possible elements and operations in a query include:

    • Specifying the keyspace that is queried.

    • Specifying the document keys or using indexes to access the documents.

    • Fetching the data from the data service.

    • Filtering the result objects by specifying conditions in the WHERE clause.

    • Removing duplicate result objects from the resultset by using the DISTINCT clause.

    • Grouping and aggregating the result objects.

    • Ordering (sorting) items in the resultset in the order specified by the ORDER BY expression list.

    • Skipping the first n items in the result object as specified by the OFFSET clause.

    • Limiting the number of results returned using the LIMIT clause.

    Data Processing Capabilities

    Filtering

    You can filter the query results using the WHERE clause. Consider the following example which queries for all airports in the America/Anchorage timezone that are at an altitude of 2000ft or more. The WHERE clause specifies the conditions that must be satisfied by the documents to be included in the resultset, and the resultset is returned as an array of airports that satisfy the condition.

    The keys in the result object are ordered alphabetically at each level.
    Query
    SELECT *
    FROM   `travel-sample`.inventory.airport
    WHERE  tz = "America/Anchorage"
           AND geo.alt >= 2000;
    Result
    [
      {
        "airport": {
          "airportname": "Arctic Village Airport",
          "city": "Arctic Village",
          "country": "United States",
          "faa": "ARC",
          "geo": {
            "alt": 2092,
            "lat": 68.1147,
            "lon": -145.579
          },
          "icao": "PARC",
          "id": 6729,
          "type": "airport",
          "tz": "America/Anchorage"
        }
      },
      {
        "airport": {
          "airportname": "Anaktuvuk Pass Airport",
          "city": "Anaktuvuk Pass",
          "country": "United States",
          "faa": "AKP",
          "geo": {
            "alt": 2103,
            "lat": 68.1336,
            "lon": -151.743
          },
          "icao": "PAKP",
          "id": 6712,
          "type": "airport",
          "tz": "America/Anchorage"
        }
      }
    ]

    Querying Across Relationships

    You can use the SELECT statement to query across relationships using the JOIN clause or subqueries.

    JOIN Clause

    Before we delve into examples, let’s take a look at the data model of the inventory scope in the travel-sample bucket, which is used in the following examples. For more details about the data model, see Travel App Data Model.

    travel app data model
    Figure 3. Data model of inventory scope

    The first example uses a JOIN clause to find the distinct airline details which have routes that start from SFO. This example JOINS the document from the route keyspace with documents from the airline keyspace using the KEY "airlineid".

    • Documents from the route keyspace are on the left side of the JOIN, and documents from the airline keyspace are on the right side of the JOIN.

    • The documents from the route keyspace (on the left) contain the foreign key "airlineid" of documents from the airline keyspace (on the right).

    Query
    SELECT DISTINCT airline.name, airline.callsign,
      route.destinationairport, route.stops, route.airline
    FROM `travel-sample`.inventory.route
      JOIN `travel-sample`.inventory.airline
      ON KEYS route.airlineid
    WHERE route.sourceairport = "SFO"
    LIMIT 2;
    Results
    [
      {
        "airline": "B6",
        "callsign": "JETBLUE",
        "destinationairport": "AUS",
        "name": "JetBlue Airways",
        "stops": 0
      },
      {
        "airline": "B6",
        "callsign": "JETBLUE",
        "destinationairport": "BOS",
        "name": "JetBlue Airways",
        "stops": 0
      }
    ]

    Let’s consider another example which finds the number of distinct airports where AA has routes. In this example:

    • Documents from the airline keyspace are on the left side of the JOIN, and documents from the route keyspace are on the right side.

    • The WHERE clause predicate airline.iata = "AA" is on the left side keyspace airline.

    This example illustrates a special kind of JOIN where the documents on the right side of JOIN contain the foreign key reference to the documents on the left side. Such JOINs are referred to as index JOIN. For details, see Index JOIN Clause.

    Index JOIN requires a special inverse index route_airlineid on the JOIN key route.airlineid. Create this index using the following command:

    CREATE INDEX route_airlineid ON `travel-sample`.inventory.route(airlineid);

    Now we can execute the following query.

    Query
    SELECT Count(DISTINCT route.sourceairport) AS distinctairports1
    FROM `travel-sample`.inventory.airline
      JOIN `travel-sample`.inventory.route
      ON KEY route.airlineid FOR airline
    WHERE airline.iata = "AA";
    Results
    [
       {
          "distinctairports1": 429
       }
    ]

    Subqueries

    A subquery is an expression that is evaluated by executing an inner SELECT query. Subqueries can be used in most places where you can use an expression such as projections, FROM clauses, and WHERE clauses.

    A subquery is executed once for every input document to the outer statement and it returns an array every time it is evaluated. See Subqueries for more details.

    Query
    SELECT *
    FROM   (SELECT t.airportname
            FROM   (SELECT *
                    FROM   `travel-sample`.inventory.airport t
                    WHERE  country = "United States"
                    LIMIT  1) AS s1) AS s2;
    Results
    [
      {
        "s2": {
          "airportname": "Barter Island Lrrs"
        }
      }
    ]

    Deep Traversal for Nested Documents

    When querying a keyspace with nested documents, SELECT provides an easy way to traverse deep nested documents using the dot notation and NEST and UNNEST clauses.

    Dot Notation

    The following query looks for the schedule, and accesses the flight id for the destination airport "ALG". Since a given flight has multiple schedules, the attribute schedule is an array containing all schedules for the specified flight. You can access the individual array elements using the array indexes. For brevity, we’re limiting the number of results in the query to 1.

    Query
    SELECT t.schedule[0].flight AS flightid
    FROM `travel-sample`.inventory.route t
    WHERE destinationairport="ALG"
    LIMIT 1;
    Results
    [
      {
        "flightid": "AH631"
      }
    ]

    NEST and UNNEST

    Note that an array is created with the matching nested documents. In this example:

    • The airline field in the result is an array of the airline documents that are matched with the key route.airlineid.

    • Hence, the projection is accessed as airline[0] to pick the first element of the array.

    Query
    SELECT DISTINCT route.sourceairport,
                    route.airlineid,
                    airline[0].callsign
    FROM `travel-sample`.inventory.route
    NEST `travel-sample`.inventory.airline
      ON KEYS route.airlineid
    WHERE route.airline = "AA"
    LIMIT 4;
    Results
    [
      {
        "airlineid": "airline_24",
        "callsign": "AMERICAN",
        "sourceairport": "ABE"
      },
      {
        "airlineid": "airline_24",
        "callsign": "AMERICAN",
        "sourceairport": "ABI"
      },
      {
        "airlineid": "airline_24",
        "callsign": "AMERICAN",
        "sourceairport": "ABQ"
      },
      {
        "airlineid": "airline_24",
        "callsign": "AMERICAN",
        "sourceairport": "ABZ"
      }
    ]

    The following example uses the UNNEST clause to retrieve the author names from the reviews object.

    Query
    SELECT r.author
    FROM `travel-sample`.inventory.hotel t
    UNNEST t.reviews r
    LIMIT 4;
    Results
    [
      {
        "author": "Ozella Sipes"
      },
      {
        "author": "Barton Marks"
      },
      {
        "author": "Blaise O'Connell IV"
      },
      {
        "author": "Nedra Cronin"
      }
    ]

    Aggregation

    As part of a single SELECT statement, you can also perform aggregation using the SUM, COUNT, AVG, MIN, MAX, or ARRAY_AVG functions.

    The following example counts the total number of flights to SFO:

    Query
    SELECT count(schedule[*]) AS totalflights
    FROM `travel-sample`.inventory.route t
    WHERE destinationairport="SFO";
    Results
    [
      {
        "totalflights": 250
      }
    ]

    Combining Resultsets Using Operators

    You can combine the result sets using the UNION or INTERSECT operators.

    Consider the following example which looks for the first schedule for flights to "SFO" and "BOS":

    Query
    (SELECT t.schedule[0]
     FROM `travel-sample`.inventory.route t
     WHERE destinationairport = "SFO"
     LIMIT 1)
    UNION ALL
    (SELECT t.schedule[0]
     FROM `travel-sample`.inventory.route t
     WHERE destinationairport = "BOS"
     LIMIT 1);
    Results
    [
      {
        "$1": {
          "day": 0,
          "flight": "AI339",
          "utc": "23:05:00"
        }
      },
      {
        "$1": {
          "day": 0,
          "flight": "AM982",
          "utc": "09:11:00"
        }
      }
    ]

    Grouping, Sorting, and Limiting Results

    You can perform further processing on the data in your result set before the final projection is generated. You can group data using the GROUP BY clause, sort data using the ORDER BY clause, and you can limit the number of results included in the result set using the LIMIT clause.

    The following example looks for the number of airports at an altitude of 5000ft or higher and groups the results by country and timezone. It then sorts the results by country names and timezones (ascending order by default).

    Query
    SELECT COUNT(*) AS count,
           t.country AS country,
           t.tz AS timezone
    FROM `travel-sample`.inventory.airport t
    WHERE geo.alt >= 5000
    GROUP BY t.country, t.tz
    ORDER BY t.country, t.tz;
    Results
    [
      {
        "count": 2,
        "country": "France",
        "timezone": "Europe/Paris"
      },
      {
        "count": 57,
        "country": "United States",
        "timezone": "America/Denver"
      },
      {
        "count": 7,
        "country": "United States",
        "timezone": "America/Los_Angeles"
      },
      {
        "count": 4,
        "country": "United States",
        "timezone": "America/Phoenix"
      },
      {
        "count": 1,
        "country": "United States",
        "timezone": "Pacific/Honolulu"
      }
    ]

    Index Selection

    The optimizer attempts to select an appropriate secondary index for a query based on the filters in the WHERE clause. If it cannot select a secondary query, the query service falls back on the primary index for the keyspace.

    Secondary indexes do not index a document if the leading index key is MISSING in the document. This means that when a query selects a field which is MISSING in some documents, the optimizer will not be able to choose a secondary index which uses that field as a leading key.

    To enable the optimizer to choose the required index, you must use a WHERE clause of some kind to filter out any documents in which the required field is MISSING. The minimum filter you can use to do this is IS NOT MISSING. This is usually only necessary in queries which do not otherwise have a WHERE clause; for example, some GROUP BY and aggregate queries.

    A query is said to be sargable if the optimizer is able to select an index to speed up the execution of the query.

    Example 1. Field with MISSING values — cannot choose the secondary index

    This example uses an index idx_image_direct_url that is defined by following statement.

    Index
    CREATE INDEX `idx_image_direct_url`
    ON `travel-sample`.inventory.landmark(`image_direct_url`);
    Query
    EXPLAIN SELECT image_direct_url FROM `travel-sample`.inventory.landmark; (1)
    Result
    [
      {
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "PrimaryScan3",
              "bucket": "travel-sample",
              "index": "def_inventory_landmark_primary", (2)
              "index_projection": {
                "primary_key": true
              },
              "keyspace": "landmark",
              "namespace": "default",
              "scope": "inventory",
              "using": "gsi"
            },
    // ...
          ]
        },
        "text": "SELECT image_direct_url FROM `travel-sample`.inventory.landmark;"
      }
    ]
    1 The query selects the image_direct_url field, which is MISSING for many documents in the landmark keyspace.
    2 Therefore the optimizer falls back on the def_inventory_landmark_primary index.
    Example 2. Filter out MISSING values — correct secondary index is chosen
    EXPLAIN SELECT image_direct_url FROM `travel-sample`.inventory.landmark
    WHERE image_direct_url IS NOT MISSING; (1)
    [
      {
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan3",
              "bucket": "travel-sample",
              "covers": [
                "cover ((`landmark`.`image_direct_url`))",
                "cover ((meta(`landmark`).`id`))"
              ],
              "filter": "(cover ((`landmark`.`image_direct_url`)) is not missing)",
              "index": "idx_image_direct_url", (2)
              "index_id": "ed5be54231ea184",
              "index_projection": {
                "entry_keys": [
                  0
                ]
              },
    /// ...
            }
          ]
        },
        "text": "SELECT image_direct_url FROM `travel-sample`.inventory.landmark\nWHERE image_direct_url IS NOT MISSING;"
      }
    ]
    1 The WHERE clause filters out documents where image_direct_url is not MISSING.
    2 The optimizer correctly chooses the idx_image_direct_url index.

    For further examples, refer to Group By and Aggregate Performance.