Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Set Up an External Data Source

  • how-to
To provide query access to data that resides in external object storage, you create an external link and associate it with an external collection.

Supported External Sources

Operational Insights supports external links to the following object storage providers:

  • Amazon S3

  • S3-compatible object stores

  • Azure Blob Storage

  • Google Cloud Storage (GCS)

This data remains in the source location — Operational Insights does not copy it into a collection. You can query Delta tables residing in S3 buckets, S3 objects, or Azure Blob Storage containers, in one of the following formats:

  • JSON

  • CSV

  • TSV

  • Parquet

  • Avro

Apache Iceberg

Operational Insights also supports read-only access to Apache Iceberg tables through external catalogs. Unlike S3 external collections, Iceberg tables are registered via a catalog entity rather than a direct link to object storage.

To query Iceberg data:

  1. Register a catalog using CREATE CATALOG.

  2. Register an Iceberg table on that catalog using CREATE EXTERNAL COLLECTION.

See Iceberg Support for the full list of supported catalog types and prerequisites.

You can also register catalogs and Iceberg collections with the workbench, as described in Set Up Iceberg Data with the Workbench.

Set Up Iceberg Data with the Workbench

Before you begin, create the link or links that authenticate access to the catalog and to the object storage that holds its data files. See Query Iceberg Tables.

To register a catalog:

  1. In the Web Console, click Workbench in the navigation bar.

  2. In the explorer, next to Catalogs, click + new catalog. The Create Catalog dialog opens.

  3. In Catalog Name, enter a name for the catalog. The Catalog Type is always Iceberg.

  4. In Catalog Source, select the kind of catalog: REST, AWS Glue, AWS Glue REST, S3 Tables, BigLake Metastore, Nessie, or Nessie REST.

  5. In Link, select the link that authenticates access to the catalog.

  6. Complete the fields for the selected source, such as URI, Warehouse, SigV4 Signing Name, SigV4 Signing Region, or Quota Project ID. These correspond to the WITH properties for that source, described in CREATE CATALOG.

  7. Optionally, under Additional Parameters (Optional), click + Add Parameter to add any other property as a name and value pair.

  8. Click Save.

The catalog appears under Catalogs in the explorer. Expand it to see its type, source, and link, or to drop it.

To add an Iceberg collection on a catalog:

  1. In the explorer, under Catalogs, click + collection next to the catalog. The Add Iceberg Collection dialog opens.

  2. Enter a Collection Name, and select the Database and Scope for the collection.

  3. Select the Iceberg Namespace and Table Name. The lists are read from the catalog.

  4. Optionally, select the type conversions to apply: Decimal to Double, Timestamp to Long, Time to Int, or Date to Int, and enter a Timezone. See CREATE EXTERNAL COLLECTION.

  5. In Link, select the link that authenticates access to the table’s data files.

  6. Optionally, under Time Travel, pin the collection to a Snapshot ID or a Snapshot Timestamp.

  7. Click Save.

Operational Insights uses different types of links to store credentials for accessing different types of data sources. You can use the UI to create links.

To create a link with an SDK, note the following:

  • External: in an SDK, you use the S3ExternalAnalyticsLink class and then the AnalyticsIndexManager class to create the link.

For more information about data sources, see Data Model & Hierarchy.

See Also