Configure Search to Include Table and Column Metadata from Atlan

With the introduction of Analytics Marketplace in v7.1.0, discovery and search capabilities across multiple data catalog platforms has been significantly expanded.

You can search for the associated database table assets without leaving the Metric Insights UI and without recreating them as objects in Metric Insights. Global Search will return the assets with the list of the available metadata and links to proceed to the asset in the data catalog tool.

For now, this functionality is available for Atlan. Support for other data catalog tools is planned in future releases.

PREREQUISITES:

  • Established connectivity with Atlan;
  • Dataset created from the Atlan All Table Assets object. RECOMMENDATION: Set the "Columns(string)" Length to "2000". At 2001 characters, the system automatically changes this from a VARCHAR datatype to TEXT (65535 characters). You may need this if you have a table with many columns and/or very verbose descriptions.  
  • Dataset created based on the above mentioned Dataset to surface additional metadata such as Glossary Terms in search, if needed. See instructions below.

Optionally, you can have Atlan metadata synced with Metric Insights before configuring search. This will help Glossary Terms appear tidier in the UI.

Create Dataset from All Table Assets Dataset
  1. Data Source: select the Data Source used as Data Storage when creating Dataset from the Atlan All Table Assets object;
  2. Query: the query is intended to:
    • Set Object Type Column to "Table";
    • Transform standard Atlan fields, such as Database name, Schema, Row Count, etc., into JSON objects. NOTE: Atlan Tags are handled via the Dataset setup and do not require to be transformed into JSON objects;
    • Transform user-added fields into JSON objects.
      • This step will ensure that metadata is searchable and appears in search properly formatted.
      • The query must include the ID of Dataset created from the Atlan All Table Assets object in "FROM dataset_datasetID".
  3. Set the Dataset Column Length for "columns__string_" to "2000". You can also change the name of the column for tidier display in global search.
Example Query

SELECT *

, "Table" as MI_object_type

, JSON_OBJECT('Database Name', database_name__string_) as "Database Name"

, JSON_OBJECT('Schema', schema__string_) as "Schema"

, JSON_OBJECT('Row Count', rows_count__string_) as "Row Count"

, JSON_OBJECT('Size', size__string_) as "Size"

, JSON_OBJECT('Data Type', data_type__data_type_) as "Data Type"

, JSON_OBJECT('Pii Data',pii___choose_) AS "Pii Data"

, JSON_OBJECT('Atlan Owners',JSON_ARRAY(owners__string_)

, 'Business Owner', JSON_ARRAY(business_owner__string_)

, 'Technical Owner', JSON_ARRAY(technical_owner__string_)

, 'Data Steward', JSON_ARRAY(data_steward__string_)

, 'Business Owner Group', JSON_ARRAY(business_owner_group__string_)

) AS "Owners"

FROM dataset_12345

For more details on how to create Datasets, refer to Understanding Datasets.

Access Admin > System > Search Setup

  1. Under Datasets to include in Search, [+Add Dataset].
  2. Dataset & View: choose the Dataset created at Step Create Dataset from All Reporting Assets Dataset.
  3. [Save]

2. Map Dataset Columns

Use the following mapping example with Allow click through to external tool enabled.

Field Dataset Column
Title Column table_name__string_
Description Column main_description__string_
Object View URL bi_tool_url__string_
External Tool Type Atlan
External Tool URL asset_url__string_
Content Type Column connection_type__string_
Object Type Column MI_object_type
Tags Column tag_s__string_
Certification Flag Column certification_status__string_
Certification Level Column certification_level__certification_level_
Certified By Column certificate_by__string_
Certification Date Column certificate_at__string_

For more details on configuring search across Datasets and Access, refer to Configure Search Across Specific Dataset (v7.1.0+).

3. Add Glossary and Custom Fields Mapping

  1. Under Dataset Columns for Glossary Terms, [+Add Dataset Columns].
  2. Dataset Column: choose the Glossary column from the Dataset.
  3. Glossary Section: choose the corresponding Glossary Section from Metric Insights.
    • NOTE: Fields will map better if Atlan Glossary has been synced with Metric Insights.
  4. [Save] and add as many as needed.
  5. Under Dataset Columns for Search, [+Add Dataset Columns].
  6. Dataset Column: choose a custom field column from the Dataset.
    • MANDATORY:
      • Add the Tags column without checking Column data is formatted in JSON. This will make objects searchable by Tags.
      • Add the Columns column and check Column data is formatted in JSON.
  7. Enable Column data is formatted in JSON.
  8. [Save] and add as many as needed.
  9. [Back to Search Setup] and [Update All Search Indexes] at the upper right.

4. Verify Result