Configuring Databricks External Resource (v7.2.0)

This article describes the second step of Connecting to Databricks MCP Server. Proceed to creating and configuring a Concierge External Resource and finish establishing connection between Concierge and Databricks Genie.

Prerequisites:

Metric Insights Admin account;
Data collected from Databricks account:
- Genie Space ID
- Server URL
- For "Defined in configuration Auth Type:
  - Token
- For "Service Account Token" Auth Type:
  - Application ID
  - Client Secret Value

1. Create an External Resource

Go to Metric Insights, access Admin > Concierge Setup and open the Content Sources tab

Click [+ Add External Resource Configuration].
Select Databricks Genie from the list.

After that, the User will be redirected to the External Resource Editor.

2. Authentication Tab

The Name is generated automatically, but it can be changed. This name will be used to identify this external resource when assigning domains and if using Service Account Token, when user are asked to Authenticate.

2.1. Auth Type: Defined in Configuration

Choosing "Defined in configuration" type of authentication means that the Token, copied in the Databricks account will be inserted into the configuration code as is.

This type of authentication is the most flexible and should be used when the long-lasting Token can be generated within the external tool. Be aware, that when the Token expires, it has to be re-entered into the code on Configuration tab manually.

2.2. Auth Type: Service Account Token

Choosing "Service Account Token" type of authentication means that the Token will be requested and then automatically inserted to the configuration code by Metric Insights. The User has to provide access credentials to the Databricks account.

This type of authentication is used when the external tool supports OAuth authorization. If the Token requires regular refreshing, it will be re-inserted into the configuration code automatically.

3. Configuration Tab

Proceed to Configuration tab

Icon: To add a custom icon, upload an SVG image on a transparent background.
A block of example JSON code is added to the field automatically. The parts in double curly braces ("{{}}") has to be filled with information specific to your environment.
- For more details about how to build the configuration, check the Connecting External Agents to Concierge via MCP Server article.

Check the JSON code usage example for Databricks External Resource configuration:

Code Usage Example

{
  "name": "databricks",
  "enabled": true,
  "title": "Databricks",
  "display": "raw_or_llm_for_charting",
  "data_transformer": "ai.scenarios.mcp_data_transformers.databricks.DatabricksTransformer.transform",
  "ignore_native_mcp_tool_details": true,
  "apply_data_transformer_before_llm_response_processing": false,
  "optional_tool_params_to_remove": [
    "conversation_id"
  ],
  "scenario_type": "mcp_single_tool",
  "scope": "Databricks talk to your data solution, questions about support tickets.",
  "mcp_server": {
    "url": "https://{{instance}}.cloud.databricks.com/api/2.0/mcp/genie/{{space_id}}",
    "name": "databricks",
    "auth": "{{databricks token}} or <auth_token>",
    "transport": "streamable_http"
  },
  "tool": "query_space_{{space_id}}",
  "timeout": 300
}

Databricks Code Specifics

Field		Description
`name`		The value of this variable should be unique and exactly match the Name field on the Authentication tab. It may contain A-Z/a-z letters, numeric digits, and underscore (_).
`enabled`		This field allows for the External Resource to be disable instead of having to delete it. If it is `true`, the External Resource is enabled. If it is `false`, the External Resource is disabled. NOTE: After disabling the External Resource make sure to disconnect or disable all Domains connected to it or the user will see an error message in Concierge.
`title`		The display name for this external resource. This name is shown when associating Domains and sometimes in error messages if there is a problem with the External Resource.
`display`		Defines how data from the external agent is processed and shown. The `raw_or_llm_for_charting` is the recommended option for this variable. It means that data is received from the MCP tool as is and is only sent to be processed by the LLM on the Concierge side if there is a chart to be created.
`data_transformer`		Custom response processor for this MCP server. Metric Insights provide the following transformer for processing raw responses, which will be quicker than use the LLM for processing: `ai.scenarios.mcp_data_transformers.databricks.DatabricksTransformer.transform`
`ignore_native_mcp_tool_details`		Determines whether to pass the description provided by the MCP server into the LLM request context. Can be `true` or `false`.
`apply_data_transformer_before_llm_response_processing`		Determines whether to use the custom data transformer defined in "data_transformer" before LLM processing of the MCP response. This should be set to `true`.
`optional_tool_params_to_remove`		This field defines optional parameters for the MCP tool to be removed from the request. For Databricks to work properly, the `conversation_id` has to be removed.
`scenario_type`		The `mcp_single_tool` option is mandatory for this variable.
`scope`		This optional field may be included to provide a general description for this external resource. This is useful for providing instructions that are shared across all Domains that use this external resource.
`mcp_server`	`url`	URL of the MCP Server is a required value. The URL to use is constructed like this: `https://{{instance}}.cloud.databricks.com/api/2.0/mcp/genie/{{space_id}}` where {{instance}} is the server URL and {{space_id}} is the Space ID value that were obtained in Databricks account. Space ID(s) can be included directly in the JSON block or via a variable (See the Add Variables section below).
	`name`	A unique name for this MCP server that identifies this specific connection. This can be the same as the "name" field above.
	`auth`	If the Auth Type on the Authentication tab is "Defined in configuration", this field should contain the `{{databricks token}}`, generated and copied in the Databricks account. It would be best to include this as an encrypted variable. (See the Add Variables section below). If the Auth Type on the Authentication tab is "Service Account Token", this field must be set to `<auth_token>`
	`transport`	This variable is required. Databricks MCP server supports only the `streamable_http` transport mode.
`tool`		The `scenario_type` variable is set to `mcp_single_tool`, so this field is required. The value to use is `query_space_`{{space_id}} where {{space_id}} is the Space ID that was obtained in Databricks account. This can also be included as a variable. (See the Add Variables section below).
`timeout`		How long Concierge will wait for a response from the external resource. The recommended value is `300` seconds.

4. Get Token

If the "Service Account Token" Auth Type was chosen, go back to the Info tab.

Click [Get Token], so Metric Insights will obtain the Token from Databricks. If Token was generated successfully, the confirmation message "Token generated for <...>" will appear.

5. Add Variables (Optional)

It is possible to use Variables to replace sensitive information in the JSON code block.

Click the arrow icon to open the Variables side menu.
Click [+ Create Variable] and add the Name and Value of the Databricks token into corresponding fields. No need to to save anything. After that you can use the Variable Name in the square brackets (for example, [DATABRICKS_TOKEN]) in the code instead of the actual Token itself. Use Variables to keep the sensitive information safe. It is optional to encrypt the variable by selecting the checkbox in Encrypted column.

Don't forget to [Save] the External Resource.

Databricks External Resource will not appear in the All Sources menu. Add a Domain to it and that Domain will be visible in All Sources menu.

Configuring Databricks External Resource (v7.2.0)

1. Create an External Resource