CREATE a Vector Index
- Operational Insights
- reference
Use a CREATE statement to create a vector index for approximate nearest neighbor (ANN) search on a remote or standalone collection.
A vector index, of type VTREE, stores the embeddings from a single vector field and organizes them into a routing tree of leaf clusters.
The query optimizer uses a VTREE index to accelerate approximate nearest neighbor (ANN) queries, pruning the search to a subset of clusters instead of scanning every record.
For more information about the query-side functions that use this index, see Vector Functions.
A vector index covers only 1 vector field.
Before you create a vector index on a collection, you must run ANALYZE COLLECTION on that collection to gather the samples the index build requires.
Creating a vector index on a collection with no samples fails.
For more information, see Collecting Samples.
|
Syntax
CreateVectorIndex EBNF
CreateVectorIndex ::= "CREATE" "INDEX" Identifier ("IF" "NOT" "EXISTS")?
"ON" QualifiedName
"(" VectorIndexField ")"
("INCLUDE" "(" NestedField ( "," NestedField )* ")")?
("TYPE" "VTREE")?
"WITH" ObjectConstructor
CreateVectorIndex Diagram
Show VectorIndexField Diagram
Show NestedField Diagram
A vector index does not accept the CAST(…) modifier or an INCLUDE/EXCLUDE UNKNOWN KEY modifier, both of which are available on a standard secondary index.
For more information, see CREATE INDEX Statements.
|
Examples
The following example creates a vector index on the review_vector_emb field, specifying only the mandatory dimension and similarity parameters and omitting the optional TYPE clause.
CREATE INDEX review_vector_idx
ON products.ptype.product (review_vector_emb VECTOR)
WITH {"dimension": 384, "similarity": "cosine"};
Show additional examples
The following example creates a vector index and uses the INCLUDE clause to replicate the rating and genre fields in the index.
A query that filters on rating or genre, orders by vector distance, and selects only the primary key or INCLUDE fields can then be served entirely from the index, without a lookup against the base collection.
If the query selects other fields, the filter is still evaluated inside the index search, but each result is then looked up in the collection and re-ranked.
CREATE INDEX review_vector_idx
ON products.ptype.product (review_vector_emb VECTOR)
INCLUDE (rating, genre)
TYPE VTREE
WITH {"dimension": 384, "similarity": "cosine"};
The following example fine-tunes the index build and storage behavior by overriding the default settings for train_list_fraction, quantization, and epsilon:
CREATE INDEX ix_vtree
ON Entertainment.Movie (embedding VECTOR)
INCLUDE (title, vote_average, revenue)
TYPE VTREE
WITH {
"dimension": 384,
"similarity": "cosine",
"train_list_fraction": 0.2,
"quantization": "SQ8",
"epsilon": 0.25
};
Drop a vector index the same way you drop any other index. For more information, see DROP Statements.
DROP INDEX Entertainment.Movie.ix_vtree IF EXISTS;
Arguments
- Identifier
-
For the
CREATE INDEXstatement, the firstIdentifieris the index name.
- QualifiedName
-
The
QualifiedNameidentifies the collection to build the index on.
- VectorIndexField
-
The
VectorIndexFieldconsists of aNestedFieldthat specifies the vector field to index, followed by theVECTORkeyword. The field must contain an array of floating point numbers with the dimensionality specified in theWITHclause. You can have only 1 vector field per vector index.
- INCLUDE
-
The optional
INCLUDEclause replicates the listed non-key fields directly in the vector index. When you omitINCLUDE, the index stores only the primary key and the vector field. Replicating fields in the index lets the query optimizer take ANN query paths that filter on those fields from within the closest leaf clusters, instead of looking up each candidate record in the base collection.Do not specify a type for an
INCLUDEfield of a vector index. Specifying one causes a compilation error.
- TYPE
-
The optional
TYPEclause specifies the index type.VTREEis the only supported vector index type, and it’s the default when you omitTYPEon an index over aVECTORfield.
- WITH
-
The
WITHclause is mandatory for a vector index. ItsObjectConstructorrepresents an object containing key-value pairs, 1 for each parameter. You can define the following parameters:Name Description Schema dimension
RequiredThe dimensionality of the vector field. This specifies the length of the vector arrays stored in the indexed field, such as 384 for a common sentence-transformer model.
integer
similarity
RequiredThe distance, or similarity, metric used to train and query the index. Accepts one of the following string values:
-
euclidean -
l2 -
euclidean_squared -
l2_squared -
cosine -
dot(supported only for normalized embeddings in this release)
enum: euclidean, l2, euclidean_squared, l2_squared, cosine, dot
train_list_fraction
OptionalThe decimal fraction of dataset records sampled to train the leaf centroid structure during index creation. For example,
0.1samples 10% of the dataset. Valid values range from greater than 0 to 1. Operational Insights rejects 0.
Default: derived fromnum_clustersand the number of records in each storage partition, or1.0on a small dataset.float
quantization
OptionalThe scalar quantization applied to stored vectors to reduce index size and speed up distance computation, at the cost of a small reduction in recall. Accepts
SQ4(4-bit) orSQ8(8-bit).
Default:SQ8enum: SQ4, SQ8
epsilon
OptionalA fractional tolerance, greater than 0, that extends the search radius beyond the closest cluster boundary during index insertion and ANN query execution. A larger epsilon increases recall at the cost of visiting more clusters.
Default:0.75float
num_clusters
OptionalThe number of leaf clusters created in each storage partition.
Default: the square root of the partition’s record count. If that square root is less than 100, Operational Insights uses the ceiling of the record count divided by 100 instead. Thecompiler.index.vector.max.num.clusterssetting caps this value, and defaults to65536.integer
storage-block-compression
OptionalThe block-level compression applied to the index’s storage. Takes a nested object with 1 field,
scheme, whose value issnappyornone.
Default: the cluster’s default storage compression scheme,snappyunless changed.object
-
A num_clusters value greater than the compiler.index.vector.max.num.clusters cap, or greater than the partition’s record count, results in an error.
A value more than 10% away from the recommended default draws a warning.
|
Setting both num_clusters and train_list_fraction is an error.
Setting num_clusters derives train_list_fraction from it, but setting train_list_fraction does not derive num_clusters — it keeps its own record-count-based default instead.
|
| Name | Description | Schema |
|---|---|---|
seed |
A seed value for the index build’s random number generator.
Set a specific value to reproduce the same exact index structure each time you create the index. |
integer |
| The VTREE index in Operational Insights does not index records whose vector embedding field is any of the following, without raising a warning: |
-
missing -
null -
The wrong type
-
Empty
-
The wrong length