You are viewing the documentation for a prerelease version.

Updating Statistics for Index Expressions

  • Couchbase Server 7.0
    +
    You can use the UPDATE STATISTICS statement to gather statistics for an index key expression.

    Purpose

    The UPDATE STATISTICS statement provides a syntax which enables you to gather statistics for an index key expression.

    When you use an index with a query, you typically create the index on the fields which the query uses to filter. To use the cost-based optimizer with that query, you must collect statistics on the same fields that you used to create the index.

    A query may have predicates on non-indexed fields, and you can collect statistics on those fields also to help the optimizer.

    For a query which filters on an array or array of objects, you must collect the statistics using exactly the same expression that you used to create the index.

    Syntax

    update-statistics-expressions ::=
      ( UPDATE STATISTICS [ FOR ] | ANALYZE [ KEYSPACE | COLLECTION ] )
      keyspace-ref '(' index-expr [ ',' index-expr ]* ')' [ index-with ]
    ( 'UPDATE' 'STATISTICS' 'FOR'? | 'ANALYZE' ( 'KEYSPACE' | 'COLLECTION')? ) keyspace-ref '(' index-expr ( ',' index-expr )* ')' index-with?

    For this syntax, UPDATE STATISTICS and ANALYZE are synonyms. The statement must begin with one of these alternatives.

    When using the UPDATE STATISTICS keywords, the FOR keyword is optional. Including this keyword makes no difference to the operation of the statement.

    When using the ANALYZE keyword, the COLLECTION or KEYSPACE keywords are optional. Including either of these keywords makes no difference to the operation of the statement.

    Keyspace Reference

    keyspace-ref ::= keyspace-path | keyspace-partial
    keyspace-path | keyspace-partial
    keyspace-path ::= [ namespace ':' ] bucket [ '.' scope '.' collection ]
    ( namespace ':' )? bucket ( '.' scope '.' collection )?
    keyspace-partial ::= collection
    collection

    The simple name or fully-qualified name of the keyspace for which you want to gather statistics. Refer to the CREATE INDEX statement for details of the syntax.

    Index Expression

    The expression for which you want to gather statistics. This may be any expression that is supported as an index key, including, but not limited to:

    WITH Clause

    index-with ::= WITH expr
    'WITH' expr

    Use the WITH clause to specify additional options.

    expr

    An object with the following properties:

    sample_size

    [Optional] An integer specifying the sample size to use for distribution statistics. A minimum sample size is also calculated. If the specified sample size is smaller than the minimum sample size, the minimum sample size is used instead.

    resolution

    [Optional] A float representing the percentage of documents to store in each distribution bin. If omitted, the default value is 1.0, meaning each distribution bin contains 1% of the documents, and therefore 100 bins are required. The minimum resolution is 0.02 (5000 distribution bins) and the maximum is 5.0 (20 distribution bins).

    update_statistics_timeout

    [Optional] A number representing a duration in seconds. The command times out when this timeout period is reached. If omitted, a default timeout value is calculated based on the number of samples used.

    Refer to Distribution Statistics for more information on sample size and resolution.

    Result

    The statement returns an empty array.

    Examples

    Example 1. Single predicate
    Create indexes
    CREATE INDEX idx_country_city ON `travel-sample`.inventory.hotel(country, city);
    CREATE INDEX idx_city_country ON `travel-sample`.inventory.hotel(city, country);
    Update statistics
    UPDATE STATISTICS FOR `travel-sample`.inventory.hotel(city, country);
    Query
    EXPLAIN SELECT COUNT(*) FROM `travel-sample` WHERE country = 'France';
    Result
    [
      {
        "cardinality": 1, (1)
        "cost": 29.362844982943358, (2)
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan3",
              "bucket": "travel-sample",
              "cardinality": 1, (3)
              "cost": 29.29658036191524,
              "covers": [
                "cover ((`hotel`.`country`))",
                "cover ((`hotel`.`city`))",
                "cover ((meta(`hotel`).`id`))",
                "cover (count(*))"
              ],
              "index": "idx_country_city",
              "index_group_aggs": {
                "aggregates": [
                  {
                    "aggregate": "COUNT",
                    "expr": "1",
                    "id": 3,
                    "keypos": -1
                  }
                ]
              },
              "index_id": "7cba9563eb478906",
              "index_projection": {
                "entry_keys": [
                  3
                ]
              },
              "keyspace": "hotel",
              "namespace": "default",
              "scope": "inventory",
              "spans": [
                {
                  "exact": true,
                  "range": [
                    {
                      "high": "\"France\"",
                      "inclusion": 3,
                      "low": "\"France\""
                    }
                  ]
                }
              ],
              "using": "gsi"
            },
            {
              "#operator": "Parallel",
              "~child": {
                "#operator": "Sequence",
                "~children": [
                  {
                    "#operator": "InitialProject",
                    "cardinality": 1, (4)
                    "cost": 29.362844982943358,
                    "result_terms": [
                      {
                        "expr": "cover (count(*))"
                      }
                    ]
                  }
                ]
              }
            }
          ]
        },
        "text": "SELECT COUNT(*) FROM `travel-sample`.inventory.hotel WHERE country = 'France';"
      }
    ]
    1 The overall cardinality estimate for the query.
    2 The overall cost estimate for the query.
    3 Cardinality and cost estimates for the index scan operator.
    4 Cardinality and cost estimates for the initial project operator.
    Example 2. Multiple predicates

    This example uses the same indexes as Example 1.

    Update statistics
    UPDATE STATISTICS FOR `travel-sample`.inventory.hotel(city, country, free_breakfast);

    There is no index on the free_breakfast field. However, the query below refers to this field as a predicate, so we collect statistics on this field also.

    Query
    EXPLAIN SELECT COUNT(*) FROM `travel-sample`.inventory.hotel
    WHERE country = 'United States' AND free_breakfast = true;
    Result
    [
      {
        "cardinality": 1, (1)
        "cost": 1294.4644872408794,
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan3",
              "bucket": "travel-sample",
              "cardinality": 361.00000000000006, (2)
              "cost": 65.70723143606952,
              "index": "idx_country_city",
              "index_id": "7cba9563eb478906",
              "index_projection": {
                "primary_key": true
              },
              "keyspace": "hotel",
              "namespace": "default",
              "scope": "inventory",
              "spans": [
                {
                  "exact": true,
                  "range": [
                    {
                      "high": "\"United States\"",
                      "inclusion": 3,
                      "low": "\"United States\""
                    }
                  ]
                }
              ],
              "using": "gsi"
            },
            {
              "#operator": "Fetch",
              "bucket": "travel-sample",
              "cardinality": 361.00000000000006, (3)
              "cost": 1267.7836409935865,
              "keyspace": "hotel",
              "namespace": "default",
              "scope": "inventory"
            },
            {
              "#operator": "Parallel",
              "~child": {
                "#operator": "Sequence",
                "~children": [
                  {
                    "#operator": "Filter",
                    "cardinality": 267.3053435114504, (4)
                    "condition": "(((`hotel`.`country`) = \"United States\") and ((`hotel`.`free_breakfast`) = true))",
                    "cost": 1291.7051691847369
                  },
                  {
                    "#operator": "InitialGroup",
                    "aggregates": [
                      "count(*)"
                    ],
                    "cardinality": 1, (5)
                    "cost": 1294.3782226198514,
                    "group_keys": []
                  }
                ]
              }
            },
            {
              "#operator": "IntermediateGroup",
              "aggregates": [
                "count(*)"
              ],
              "cardinality": 1, (6)
              "cost": 1294.3882226198514,
              "group_keys": []
            },
            {
              "#operator": "FinalGroup",
              "aggregates": [
                "count(*)"
              ],
              "cardinality": 1, (7)
              "cost": 1294.3982226198514,
              "group_keys": []
            },
            {
              "#operator": "Parallel",
              "~child": {
                "#operator": "Sequence",
                "~children": [
                  {
                    "#operator": "InitialProject",
                    "cardinality": 1, (8)
                    "cost": 1294.4644872408794,
                    "result_terms": [
                      {
                        "expr": "count(*)"
                      }
                    ]
                  }
                ]
              }
            }
          ]
        },
        "text": "SELECT COUNT(*) FROM `travel-sample`.inventory.hotel WHERE country = 'United States' AND free_breakfast = true;"
      }
    ]
    1 Overall cardinality and cost estimates for the query.
    2 Cardinality and cost estimates for the index scan operator.
    3 Cardinality and cost estimates for the fetch operator.
    4 Cardinality and cost estimates for the filter operator.
    5 Cardinality and cost estimates for the initial group operator.
    6 Cardinality and cost estimates for the intermediate group operator.
    7 Cardinality and cost estimates for the final group operator.
    8 Cardinality and cost estimates for the initial project operator.
    Example 3. Filter on an array
    Create index
    CREATE INDEX idx_public_likes
    ON `travel-sample`.inventory.hotel((DISTINCT (`public_likes`)));
    Update statistics
    UPDATE STATISTICS FOR `travel-sample`.inventory.hotel((DISTINCT (`public_likes`)));
    Query
    EXPLAIN SELECT COUNT(1)
    FROM `travel-sample`.inventory.hotel
    WHERE ANY p IN public_likes SATISFIES p LIKE 'A%' END;
    Results
    [
      {
        "cardinality": 1,
        "cost": 109.7611716607931,
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "DistinctScan",
              "cardinality": 277.5315701314658,
              "cost": 106.89959133845032,
              "scan": {
                "#operator": "IndexScan3",
                "bucket": "travel-sample",
                "cardinality": 339.4999999999998,
                "cost": 84.40275249940453,
                "covers": [
                  "cover ((distinct ((`hotel`.`public_likes`))))",
                  "cover ((meta(`hotel`).`id`))"
                ],
                "filter": "cover (any `p` in (`hotel`.`public_likes`) satisfies ((\"A\" <= `p`) and (`p` < \"B\")) end)",
                "filter_covers": {
                  "cover (any `p` in (`hotel`.`public_likes`) satisfies ((\"A\" <= `p`) and (`p` < \"B\")) end)": true,
                  "cover (any `p` in (`hotel`.`public_likes`) satisfies (`p` like \"A%\") end)": true
                },
                "index": "idx_public_likes",
                ...
              }
            },
            {
              "#operator": "Parallel",
              "~child": {
                "#operator": "Sequence",
                "~children": [
                  {
                    "#operator": "InitialGroup",
                    "aggregates": [
                      "count(1)"
                    ],
                    "cardinality": 1,
                    "cost": 109.67490703976497,
                    "group_keys": []
                  }
                ]
              }
            },
            ...
          ]
        },
        "text": "SELECT COUNT(1)\nFROM `travel-sample`.inventory.hotel\nWHERE ANY p IN public_likes SATISFIES p LIKE 'A%' END;"
      }
    ]
    Example 4. Filter on an array of objects
    Create index
    CREATE INDEX idx_reviews_ratings_overall
    ON `travel-sample`.inventory.hotel(DISTINCT ARRAY r.ratings.Overall
    FOR r IN reviews END);
    Update statistics
    UPDATE STATISTICS
    FOR `travel-sample`.inventory.hotel(DISTINCT ARRAY r.ratings.Overall
    FOR r IN reviews END);
    Query
    EXPLAIN SELECT COUNT(1)
    FROM `travel-sample`.inventory.hotel
    WHERE ANY r IN reviews SATISFIES r.ratings.Overall = 4 END;
    Results
    [
      {
        "cardinality": 1,
        "cost": 107.30834694637392,
        "plan": {
          "#operator": "Sequence",
          "~children": [
            {
              "#operator": "IndexScan3",
              "bucket": "travel-sample",
              "cardinality": 1,
              "cost": 107.2420823253458,
              "covers": [
                "cover ((distinct (array ((`r`.`ratings`).`Overall`) for `r` in (`hotel`.`reviews`) end)))",
                "cover ((meta(`hotel`).`id`))",
                "cover (count(1))"
              ],
              "filter_covers": {
                "cover (any `r` in (`hotel`.`reviews`) satisfies (((`r`.`ratings`).`Overall`) = 4) end)": true
              },
              "index": "idx_reviews_ratings_overall",
              ...
            },
            ...
          ]
        },
        "text": "SELECT COUNT(1)\nFROM `travel-sample`.inventory.hotel\nWHERE ANY r IN reviews SATISFIES r.ratings.Overall = 4 END;"
      }
    ]