Configuring Databricks External Resource

This article describes a second step of Connecting to Databricks MCP Server. Proceed to setting Concierge External Resource and finish establishing connection Between Concierge and Databricks Genie.

Prerequisites:

Metric Insights Admin account;
Data collected from Databricks account:
- Genie Space ID;
- Server URL;
- For "Defined in configuration Auth Type:
  - Token;
- For "Service Account Token" Auth Type:
  - Application ID;
  - Client Secret Value.

1. Create an External Resource

Go to Metric Insights, access Admin > Concierge Setup and open the Content Sources tab

Click [+ Add External Resource Configuration].
Select Databricks Genie from the list.

After that, the User will be redirected to the External Resource Editor.

2. Authentication Tab

The Name is generated automatically, but it can be changed if necessary.

2.1. Auth Type: Defined in Configuration

Choosing "Defined in configuration" type of authentication means that the Token, copied in the Databricks account will be inserted into the configuration code as is.

This type of authentication is the most flexible and should be used when the long-lasting Token can be generated withing the external tool. Be aware, that if the Token expires, it has to be re-entered into the code on Configuration tab manually.

2.2. Auth Type: Service Account Token

Choosing "Service Account Token" type of authentication means that the Token will be requested and then automatically inserted to the configuration code by Metric Insights. The User has to provide access credentials to the Databricks account.

This type of authentication is used when the external tool supports OAuth authorization. If the Token requires regular refreshing, it will be re-inserted into the configuration code automatically.

3. Configuration Tab

Proceed to Configuration tab

Icon: To add a custom icon, upload an SVG image on a transparent background.
A block of example JSON code is added to the field automatically. The parts in double curvy brackets ("{{}}") has to be filled with information specific to your instance.
- For more details about how to build the configuration, check the Connecting External Agents to Concierge via MCP Server article.

Check the JSON code usage example for Databricks External Resource configuration:

Code Usage Example

NOTE: Pay attention to the info in the {blue curly braces}. Those pieces of data are obtained in Steps 3 and 6. For more details, check the table below.

{
  "name": "databricks",
  "enabled": true,
  "title": "Databricks",
  "display": "raw_or_llm_for_charting",
  "data_transformer": "ai.scenarios.mcp_data_transformers.databricks.DatabricksTransformer.transform",
  "ignore_native_mcp_tool_details": true,
  "apply_data_transformer_before_llm_response_processing": false,
  "optional_tool_params_to_remove": ["conversation_id"],
  "scenario_type": "mcp_single_tool",
  "scope": "Databricks talk to your data solution, questions about support tickets.",
  "mcp_server": {
    "url": "https://{DATABRICKS_HOST}.cloud.databricks.com/api/2.0/mcp/genie/{GENIE_SPACE_ID}",
    "name": "databricks",
    "auth": "{DATABRICKS_TOKEN} or <auth_token>",
    "transport": "streamable_http"
  },
  "tool": "query_space_{GENIE_SPACE_ID}",
  "timeout": 300
}

Databricks Code Specifics

Field		Description
`name`		The value of this variable should be unique and exactly match the Name field in the UI above the editor. It is allowed to use A-Z/a-z alphabetic symbols, digits, and underscore (_) when setting a name.
`enabled`		This field allows to disable External Resource instead of deleting it. If it is `true`, the External Resource is enabled. If it is `false`, the External Resource is disables. NOTE: After disabling the External Resource make sure to disconnect or disable all Domains connected to it.
`title`		The display name. This name is shown in the All Sources button and in Concierge responses.
`display`		Defines how data from the external agent is shown. The `raw_or_llm_for_charting` is the required option for this variable. It means that data is received from the MCP tool as is and is processed by the LLM on Concierge side if there is a need to create a chart.
`data_transformer`		Custom response processor for MCP server. Mainly used as an alternative to LLM processing but can be also used in combination with it.
`ignore_native_mcp_tool_details`		Determines whether to pass the description provided by the MCP server into the LLM request context. Can be `true` or `false`.
`apply_data_transformer_before_llm_response_processing`		Determines whether to use custom data transformer if it is set. Can be `true` or `false`.
`optional_tool_params_to_remove`		This field defines what optional parameters of MCP tool will be removed. For Databricks to work properly, the `conversation_id` has to be removed.
`scenario_type`		The `mcp_single_tool` option is mandatory for this variable.
`scope`		A prompt that instructs the LLM when to use this external resource.
`mcp_server`	`url`	URL of the MCP Server is a required value. The URL to use is `DATABRICKS_HOST/api/2.0/mcp/genie/GENIE_SPACE_ID` where `DATABRICKS_HOST` is the server URL and `GENIE_SPACE_ID` is the Space ID value that were copied in the Step 6. They can be copied directly or via Variables in the Add External Resource Configuration window.
	`name`	Name of MCP Server. It should be unique, but is an optional variable.
	`auth`	If the Auth Type on the Authentication tab is "Defined in configuration", this field contains the `DATABRICKS_TOKEN`, copied in the Step 3. If the Auth Type on the Authentication tab is "Service Account Token", this field has to contain <auth_token> value.
	`transport`	This variable is required. Databricks MCP server supports only the `streamable_http` transport mode.
`tool`		`scenario_type` variable is set to `mcp_single_tool`, so this field is required. The value to use is `query_space_GENIE_SPACE_ID` where `GENIE_SPACE_ID` is the Space ID value that were copied in the Step 6 or you can use the encrypted Variable.
`timeout`		Recommended value is `300` seconds.

4. Get Token

If the "Service Account Token" Auth Type was chosen, go back to the Info tab.

Click [Get Token], so Metric Insights will obtain the Token from Databricks. If Token was generated successfully, the confirmation message "Token generated for <...>" will appear.

5. Add Variables (Optionally)

It is possible to use Variables instead of sensitive information in the code. Add the data copied from the Databricks account as Variables.

Click the arrow icon to open the Variables side menu.
Click [+ Create Variable] and add Databricks token as Variable. After that you can use the Variable Name in the square brackets (for example, [DATABRICKS_TOKEN]) instead of the actual Token itself. Use Variables to keep the sensitive information safe. It is optional to encrypt the variable by selecting the checkbox in Encrypted column.

Don't forget to [Save] the External Resource.

Databricks External Resource will not appear in the All Sources menu. Add a Domain to it and that Domain will be visible in All Sources menu.