forked from databrickslabs/ucx
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added static code analysis results to assessment dashboard (databrick…
…slabs#2696) ## Changes Add job/query problem widgets to the dashboard Add directfs access widget to the dashboard ### Linked issues Resolves databrickslabs#2595 ### Functionality None ### Tests - [x] added integration tests using mock data - [x] manually tested widgets, see below: https://github.com/user-attachments/assets/3684c30f-761a-4de6-bc67-de650c5d5353 --------- Co-authored-by: Eric Vergnaud <eric.vergnaud@databricks.com> Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
- Loading branch information
1 parent
2a09a8f
commit d1dd0c5
Showing
15 changed files
with
344 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 changes: 8 additions & 0 deletions
8
...databricks/labs/ucx/queries/assessment/main/35_0_code_compatibility_problems.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
## Code compatibility problems | ||
|
||
The tables below assist with verifying if workflows and dashboards are Unity Catalog compatible. It can be filtered on the path, | ||
problem code and workflow name. | ||
Each row: | ||
- Points to a problem detected in the code using the code path, query or workflow & task reference and start/end line & column; | ||
- Explains the problem with a human-readable message and a code. | ||
|
35 changes: 35 additions & 0 deletions
35
...bricks/labs/ucx/queries/assessment/main/35_1_code_compatibility_problems_in_workflows.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
/* | ||
--title 'Workflow migration problems' | ||
--width 6 | ||
--overrides '{"spec":{ | ||
"encodings":{ | ||
"columns": [ | ||
{"fieldName": "path", "booleanValues": ["false", "true"], "linkUrlTemplate": "/#workspace/{{ link }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "path"}, | ||
{"fieldName": "code", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string", "title": "code"}, | ||
{"fieldName": "message", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string", "title": "message"}, | ||
{"fieldName": "workflow_name", "booleanValues": ["false", "true"], "linkUrlTemplate": "/jobs/{{ workflow_id }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "workflow_name"}, | ||
{"fieldName": "task_key", "booleanValues": ["false", "true"], "imageUrlTemplate": "{{ @ }}", "linkUrlTemplate": "/jobs/{{ workflow_id }}/tasks/{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "task_key"}, | ||
{"fieldName": "start_line", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "number", "title": "start_line"}, | ||
{"fieldName": "start_col", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "number", "title": "start_col"}, | ||
{"fieldName": "end_line", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "number", "title": "end_line"}, | ||
{"fieldName": "end_col", "booleanValues": ["false", "true"], "type": "integer", "displayAs": "number", "title": "end_col"} | ||
]}, | ||
"invisibleColumns": [ | ||
{"name": "link", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "link"}, | ||
{"name": "workflow_id", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "workflow_id"} | ||
] | ||
}}' | ||
*/ | ||
SELECT | ||
substring_index(path, '@databricks.com/', -1) as path, | ||
path as link, | ||
code, | ||
message, | ||
job_id AS workflow_id, | ||
job_name AS workflow_name, | ||
task_key, | ||
start_line, | ||
start_col, | ||
end_line, | ||
end_col | ||
FROM inventory.workflow_problems |
29 changes: 29 additions & 0 deletions
29
...ricks/labs/ucx/queries/assessment/main/35_2_code_compatibility_problems_in_dashboards.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
/* | ||
--title 'Dashboard compatibility problems' | ||
--width 6 | ||
--overrides '{"spec":{ | ||
"encodings":{ | ||
"columns": [ | ||
{"fieldName": "code", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string", "title": "code"}, | ||
{"fieldName": "message", "booleanValues": ["false", "true"], "type": "string", "displayAs": "string", "title": "message"}, | ||
{"fieldName": "dashboard_name", "booleanValues": ["false", "true"], "linkUrlTemplate": "/sql/dashboards/{{ dashboard_id }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "Dashboard"}, | ||
{"fieldName": "query_name", "booleanValues": ["false", "true"], "linkUrlTemplate": "/sql/editor/{{ query_id }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "Query"} | ||
]}, | ||
"invisibleColumns": [ | ||
{"name": "dashboard_parent", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "dashboard_parent"}, | ||
{"name": "dashboard_id", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "dashboard_id"}, | ||
{"name": "query_parent", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "query_parent"}, | ||
{"name": "query_id", "booleanValues": ["false", "true"], "linkUrlTemplate": "{{ @ }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "type": "string", "displayAs": "link", "title": "query_id"} | ||
] | ||
}}' | ||
*/ | ||
SELECT | ||
dashboard_id, | ||
dashboard_parent, | ||
dashboard_name, | ||
query_id, | ||
query_parent, | ||
query_name, | ||
code, | ||
message | ||
FROM inventory.query_problems |
14 changes: 14 additions & 0 deletions
14
...icks/labs/ucx/queries/assessment/main/36_0_direct_filesystem_access_problems.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- | ||
height: 4 | ||
--- | ||
|
||
# Direct filesystem access problems | ||
|
||
The table below assists with verifying if workflows and dashboards require direct filesystem access. | ||
As a reminder, `dbfs:/` is not supported in Unity Catalog, and more generally direct filesystem access is discouraged. | ||
Rather, data should be accessed via Unity tables. | ||
|
||
Each row: | ||
- Points to a direct filesystem access detected in the code using the code path, query or workflow & task reference and start/end line & column; | ||
- Provides the _lineage_ i.e. which `workflow -> task -> notebook...` execution sequence leads to that access. | ||
|
62 changes: 62 additions & 0 deletions
62
src/databricks/labs/ucx/queries/assessment/main/36_1_direct_filesystem_accesses.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
/* | ||
--title 'Direct filesystem access problems' | ||
--width 6 | ||
--overrides '{"spec":{ | ||
"encodings":{ | ||
"columns": [ | ||
{"fieldName": "location", "title": "location", "type": "string", "displayAs": "string", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "is_read", "title": "is_read", "type": "boolean", "displayAs": "boolean", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "is_write", "title": "is_write", "type": "boolean", "displayAs": "boolean", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "source", "title": "source", "type": "string", "displayAs": "link", "linkUrlTemplate": "{{ source_link }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "booleanValues": ["false", "true"]}, | ||
{"fieldName": "timestamp", "title": "last_modified", "type": "datetime", "displayAs": "datetime", "dateTimeFormat": "ll LTS (z)", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "lineage", "title": "lineage", "type": "string", "displayAs": "link", "linkUrlTemplate": "{{ lineage_link }}", "linkTextTemplate": "{{ @ }}", "linkTitleTemplate": "{{ @ }}", "linkOpenInNewTab": true, "booleanValues": ["false", "true"]}, | ||
{"fieldName": "lineage_data", "title": "lineage_data", "type": "complex", "displayAs": "json", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "assessment_start", "title": "assessment_start", "type": "datetime", "displayAs": "datetime", "dateTimeFormat": "ll LTS (z)", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "assessment_end", "title": "assessment_end", "type": "datetime", "displayAs": "datetime", "dateTimeFormat": "ll LTS (z)", "booleanValues": ["false", "true"]} | ||
]}, | ||
"invisibleColumns": [ | ||
{"fieldName": "source_link", "title": "source_link", "type": "string", "displayAs": "string", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "lineage_type", "title": "lineage_type", "type": "string", "displayAs": "string", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "lineage_id", "title": "lineage_id", "type": "string", "displayAs": "string", "booleanValues": ["false", "true"]}, | ||
{"fieldName": "lineage_link", "title": "lineage_link", "type": "string", "displayAs": "string", "booleanValues": ["false", "true"]} | ||
] | ||
}}' | ||
*/ | ||
SELECT | ||
path as location, | ||
is_read, | ||
is_write, | ||
if( startswith(source_id, '/'), substring_index(source_id, '@databricks.com/', -1), split_part(source_id, '/', 2)) as source, | ||
if( startswith(source_id, '/'), concat('/#workspace/', source_id), concat('/sql/editor/', split_part(source_id, '/', 2))) as source_link, | ||
source_timestamp as `timestamp`, | ||
case | ||
when lineage.object_type = 'WORKFLOW' then concat('Workflow: ', lineage.other.name) | ||
when lineage.object_type = 'TASK' then concat('Task: ', split_part(lineage.object_id, '/', 2)) | ||
when lineage.object_type = 'NOTEBOOK' then concat('Notebook: ', substring_index(lineage.object_id, '@databricks.com/', -1)) | ||
when lineage.object_type = 'FILE' then concat('File: ', substring_index(lineage.object_id, '@databricks.com/', -1)) | ||
when lineage.object_type = 'DASHBOARD' then concat('Dashboard: ', lineage.other.name) | ||
when lineage.object_type = 'QUERY' then concat('Query: ', lineage.other.name) | ||
end as lineage, | ||
lineage.object_type as lineage_type, | ||
lineage.object_id as lineage_id, | ||
case | ||
when lineage.object_type = 'WORKFLOW' then concat('/jobs/', lineage.object_id) | ||
when lineage.object_type = 'TASK' then concat('/jobs/', split_part(lineage.object_id, '/', 1), '/tasks/', split_part(lineage.object_id, '/', 2)) | ||
when lineage.object_type = 'NOTEBOOK' then concat('/#workspace/', lineage.object_id) | ||
when lineage.object_type = 'FILE' then concat('/#workspace/', lineage.object_id) | ||
when lineage.object_type = 'DASHBOARD' then concat('/sql/dashboards/', lineage.object_id) | ||
when lineage.object_type = 'QUERY' then concat('/sql/editor/', split_part(lineage.object_id, '/', 2)) | ||
end as lineage_link, | ||
lineage.other as lineage_data, | ||
assessment_start, | ||
assessment_end | ||
from (SELECT | ||
path, | ||
is_read, | ||
is_write, | ||
source_id, | ||
source_timestamp, | ||
explode(source_lineage) as lineage, | ||
assessment_start_timestamp as assessment_start, | ||
assessment_end_timestamp as assessment_end | ||
FROM inventory.directfs) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.