Databricks to JSM Assets
Databricks to JSM Assets Integration Guide
This guide explains how to use OnLink to import Databricks data into JSM Assets. It covers the Databricks API connection, sample data, field mapping, validation, and optional query parameters.
1. Overview
OnLink can import Databricks API responses into JSM Assets object types by using mapping statements such as key:, map:, and config:. The same approach can be used for Databricks data such as SQL statements, query results, warehouses, catalogs, schemas, tables, and users, provided the corresponding Databricks API endpoint and Assets object type are configured.
The high-level process is:
Create or confirm the target JSM Assets schema and object type.
Configure an OnLink Assets Data Source that uses the required authentication to connect to Databricks.
Set the API method and Databricks endpoint, then fetch sample data.
Map Databricks JSON fields to JSM Assets attributes.
Use Get Data to validate the result, then run the import.
2. Prerequisites
A JSM Assets schema and object type for the data you want to import.
OnLink installed and available from Manage your apps.
Databricks credentials or access token with permission to access the required API endpoints.
Permission in JSM Assets to configure data sources, mappings, and imports.
The Databricks API endpoint URL for the object type you plan to import.
3. Supported Databricks object types
The examples on this page use the Databricks Workspace API https://docs.databricks.com/api/workspace/. The same pattern can be adapted for other Databricks data, such as SQL statements, query results, warehouses, catalogs, schemas, tables, and users, by selecting the appropriate Databricks API endpoint and mapping the returned JSON fields to the matching Assets attributes.
4. Prepare Databricks credentials
Before configuring OnLink, confirm that you have Databricks OAuth client id and secret, with permission to access the required API endpoints. For Databricks API details, see https://docs.databricks.com/api/workspace/.
Keep the credentials available for the OnLink data source configuration.
5. Configure the OnLink data source
In OnLink, add an Assets Data Source.
Set Type to the authentication method required for your Databricks workspace OAuth client credentials.
Provide the workspace URL and required authentication details for the Databricks API.
6. Databricks API authentication
Use these steps to get the Client ID and secret from Databricks.
Create a service principal. In Databricks, go to Settings → Identity and access → Service principals, then select Add service principal.
Generate an OAuth secret. Open the service principal, select the Secrets tab, and click Generate secret.
Copy the credentials immediately. Save the Client ID and Secret in a secure location. The secret is shown only once.
Set the required permissions. Grant the service principal access to the Databricks workspace resources and APIs that OnLink will call, such as SQL warehouses, statements, catalogs, schemas, tables, or workspace objects.
Use the credentials in OnLink. Enter the Client ID and Secret in the OnLink connection configuration for the Databricks Assets Data Source.
7. Configure the API method
Use a GET request to retrieve statement details or results from the Databricks SQL Statement Execution API. Configure the request with the required Databricks workspace endpoint and response data path for the data you want to import.
Set Method to
GET.Set URL to your Databricks workspace URL plus the statement endpoint, for example
https://<workspace-host>/api/2.0/sql/statements.Set Response data path to the array or object path returned by your query, such as result.data_array for SQL statement results.
Leave other settings at their defaults unless the endpoint requires query parameters.
GET /api/2.0/sql/statementsUse query parameters supported by the Databricks endpoint when you need to retrieve a specific statement or result page. Example:
GET /api/2.0/sql/statements/{statement_id}?include_result=trueSample query
SELECT * FROM main.default.employees LIMIT 108. Fetch sample data
Save the connection, then run it to fetch sample values from Databricks. Use the returned sample JSON to confirm that the endpoint is correct and to identify the field names needed for mapping.
Sample response
{
"statement_id": "01ef7e2a-9c41-1b3d-8f5e-6a2b4c7d9e10",
"status": {
"state": "SUCCEEDED"
},
"manifest": {
"format": "JSON_ARRAY",
"schema": {
"column_count": 6,
"columns": [
{ "name": "employee_id", "type_text": "BIGINT", "type_name": "LONG", "position": 0 },
{ "name": "full_name", "type_text": "STRING", "type_name": "STRING", "position": 1 },
{ "name": "email", "type_text": "STRING", "type_name": "STRING", "position": 2 },
{ "name": "department", "type_text": "STRING", "type_name": "STRING", "position": 3 },
{ "name": "hire_date", "type_text": "DATE", "type_name": "DATE", "position": 4 },
{ "name": "is_active", "type_text": "BOOLEAN", "type_name": "BOOLEAN", "position": 5 }
]
},
"total_chunk_count": 1,
"chunks": [
{ "chunk_index": 0, "row_offset": 0, "row_count": 3 }
],
"total_row_count": 3,
"truncated": false
},
"result": {
"chunk_index": 0,
"row_offset": 0,
"row_count": 3,
"data_array": [
["1001", "Asha Menon", "asha.menon@example.com", "Engineering", "2021-03-15", "true"],
["1002", "Daniel Park", "daniel.park@example.com", "Engineering", "2022-07-01", "true"],
["1003", "Maria Lopez", null, "Engineering", "2019-11-20", "false"]
]
}
}9. Configure the Assets import and mapping
After the connection is established, create the Assets import by following the JSM Assets Data Synchronization guide.
Use the fetched API sample to identify the result from Databricks JSON field names and map them to the corresponding JSM Assets attributes. The key: mapping should identify the stable unique value used to match and update existing Assets objects.
Databricks column | Value from | OnLink mapping | Asset field |
|---|---|---|---|
|
|
| Employee ID |
|
|
| Full Name |
|
|
| |
|
|
| Department |
|
|
| Hire Date |
|
|
| Active |
9.1 Mapping guidance
Start with one unique
key:mapping so OnLink can identify whether to create or update an Assets object.Add
map:entries for each Databricks query result field you want to store in Assets.Compare the Databricks sample JSON with the Assets object type attributes before running an import.
If a value is nested in the JSON response, inspect the sample output carefully and use the field path supported by the OnLink mapping configuration.
10. Validate and run the import
Use Get Data to preview the data returned by Databricks.
Confirm that the sample records contain the expected fields and values.
Verify that the key field maps to a unique JSM Assets attribute.
Review object type mappings and attribute names for spelling and case consistency.
Run the import only after the previewed data and mapping are correct.