Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Data Model & Hierarchy

  • concept
This topic introduces the database objects that make up the data model hierarchy in Operational Insights.
Diagram

Operational Insights organizes your data in a hierarchy: each database contains one or more scopes, and each scope contains collections and other database objects. To support your analytical queries and manage access to data sources, you add the following objects:

Database

A database is the top-level container for organizing related information in an Operational Insights cluster. When you create a cluster, Operational Insights automatically creates a database named Default. You can add more databases using the UI or a SQL++ CREATE DATABASE statement.

Scopes

Scopes are intermediary containers that exist within a database to group related objects like collections, indexes, and functions.

When you create a cluster, Operational Insights automatically creates a scope named Default in the database named Default. You can add more scopes as needed using the UI or a SQL++ CREATE SCOPE statement.

You must make scope names unique within a database, but you can use the same scope name across different databases.

Collections

Collections are containers that can contain metadata and data that you can query and manipulate. You can add a standalone collection using UI or a CREATE COLLECTION SQL++ statement.

You must make collection names unique within a scope but you can use the same collection name across different scopes, either in the same database or across different databases.

Operational Insights has 3 types of collections:

  • Remote collections contain a shadow or mirror copy of data streamed from a remote data source. The remote data source can be a Kafka pipeline or a Couchbase database. A remote collection is associated with a link that provides authentication and connection information for the remote data source. When the link is connected to the remote source, Operational Insights streams data from the remote source into the collection. This streaming means that the remote collection has a local replica of the data in the data source. When the link is disconnected, the collection retains the data as it was when the link disconnected. Queries on remote collections are efficient because of the local shadow copy of the streamed data.

    The remote collection also contains metadata about the data format of the remote source as well as optional data filters.

    You can use the Operational Insights UI or the SQL++ CREATE COLLECTION statement to add a remote collection.

  • External collections let you query data stored in an S3 bucket. Like remote collections, they’re associated with a link. Unlike remote collections, Operational Insights does not copy data from the external data source into the external collection. Instead, every query reads data from the external storage location. The external collection contains just the metadata necessary to read data from the S3 bucket. As a result, Operational Insights cannot index external collections.

    You can use the Operational Insights UI or a CREATE EXTERNAL COLLECTION SQL++ statement to add an external collection.

  • Standalone collections allow you to assemble and manipulate groups of documents on an as-needed basis. These are stored, manipulated, and managed locally. Standalone collections do not use links.

    You populate these collections with data by importing data files or by using SQL++ statements to INSERT, COPY INTO, and otherwise add and update documents in a purpose-built collection.

    You can use the Operational Insights UI or a CREATE COLLECTION SQL++ statement to add a standalone collection.

A link is a metadata store for the authorization and authentication credentials that Operational Insights uses when connecting to a remote or external data source. You can associate multiple collections in different scopes across different databases, with a single link.

Links are categorized into 2 types:

  • Remote links have connected and disconnected states. When connected, the link provides continuous, real-time updates to the data shadowed in its associated Operational Insights remote collections.

    You incur charges when you connect a remote link.

  • External links contain the credentials Operational Insights needs to view an external storage location. These links do not have connected or disconnected states. Instead, each time you query an associated external collection, Operational Insights connects to the external data storage to read its data.

You use the Operational Insights UI to add links. See Stream Data from Remote Sources or Set Up an External Data Source.

Synonyms

A synonym is an alternative name that you can use to refer to a database object, without changing the underlying object itself. You use SQL++ statements to create synonyms.

User-Defined Functions

User-defined functions have the same syntax as built-in functions, with brackets () to contain any arguments. When you have created a user-defined function, you can call it in any expression, just like a built-in function.

The name of the function is usually an unqualified identifier, such as func1 or func-1. In this case, the path to the function is determined by the current query context.

The name of a user-defined function is case-sensitive, unlike that of a built-in function. You must call the user-defined function using the same case that you used when you created it.

You use SQL++ statements to create user-defined functions.

See Also