Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add and populate UCX workflow_runs table #2754

Merged
merged 65 commits into from
Oct 2, 2024
Merged
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
69ddb8f
Add history install
JCZuurmond Sep 25, 2024
02d8a30
Add history install to `WorkspaceContext`
JCZuurmond Sep 25, 2024
26314ea
Run history installation in create UCX catalog
JCZuurmond Sep 25, 2024
786f097
Fix test
JCZuurmond Sep 25, 2024
0180114
Add unit test for creating UCX history schema and table
JCZuurmond Sep 25, 2024
1e7cdac
Test history installation to create history schema
JCZuurmond Sep 25, 2024
50fb6e5
Format
JCZuurmond Sep 25, 2024
a3cd1c5
Create UCX catalog on `MockWorkspaceContext`
JCZuurmond Sep 25, 2024
60389b5
Remove `Record` data class and table creation
JCZuurmond Sep 26, 2024
8aeba40
Make test more robust
JCZuurmond Sep 26, 2024
5161074
Use `SchemaDeployer` to deploy schema
JCZuurmond Sep 26, 2024
4fe1de6
Add `WorkflowRun` dataclass
JCZuurmond Sep 26, 2024
11c3cb9
Make test more robust
JCZuurmond Sep 26, 2024
1ad8bb4
Create `workflow_runs` table
JCZuurmond Sep 26, 2024
68145f9
Rename `HistoryInstallation` to `ProgressTrackingInstaller`
JCZuurmond Sep 26, 2024
9c08d39
Rename progress tracking schem to 'multiworkspace'
JCZuurmond Sep 26, 2024
2f0e019
Move WorkflowRun to separate module
JCZuurmond Sep 26, 2024
302fc1d
Add workflow run recorder
JCZuurmond Sep 26, 2024
8054949
Add start time to workflow
JCZuurmond Sep 26, 2024
c58d9ed
Expect start time on record
JCZuurmond Sep 26, 2024
af119bb
Add attempt to workflow run
JCZuurmond Sep 26, 2024
01b438f
Add job name and start time to task parameters
JCZuurmond Sep 26, 2024
1c9b31d
Add `WorkflowRunRecorder` to `RuntimeContext`
JCZuurmond Sep 26, 2024
48a1a86
Expect workflow start time on initialisation
JCZuurmond Sep 26, 2024
c7d8dff
Record workflow run
JCZuurmond Sep 26, 2024
f967194
Add record workflow run task to migration progress workflow
JCZuurmond Sep 26, 2024
b84e60a
Format
JCZuurmond Sep 26, 2024
9c25c06
Fix reference to catalog
JCZuurmond Sep 26, 2024
fdb643d
Fix TODO
JCZuurmond Sep 26, 2024
34d44e5
Remove start time on Workflow
JCZuurmond Sep 27, 2024
6cf4c97
Record UCX internal workflow name
JCZuurmond Sep 27, 2024
097727c
Remove job name from job parameters
JCZuurmond Sep 27, 2024
cfb44ad
Remove WorkflowRun.status
JCZuurmond Sep 27, 2024
4c8242c
Add missing UCX catalog parameter
JCZuurmond Sep 27, 2024
4721364
Test progress tracking installer creates workflow runs table
JCZuurmond Sep 27, 2024
e249dae
Log when workflow run table is not found
JCZuurmond Sep 27, 2024
01f207a
Test workflow run recorder records a record
JCZuurmond Sep 27, 2024
ea672e1
Make finished at timestamp more precise
JCZuurmond Sep 27, 2024
ee194ea
Fix UNKNOWN typo
JCZuurmond Sep 27, 2024
c60cc31
Test workflow run values
JCZuurmond Sep 27, 2024
34975bf
Use non-escaping characters in UCX test catalog
JCZuurmond Sep 27, 2024
2d83096
Test workflow runs is populated in the during real migration progress
JCZuurmond Sep 27, 2024
da602c9
Use workflow from named parameters at init
JCZuurmond Sep 27, 2024
950d2f0
Add workspace id as task parameter
JCZuurmond Sep 27, 2024
4be9454
Pass new parameters to main
JCZuurmond Sep 27, 2024
08aef94
Remove WorkflowRuns.run_as
JCZuurmond Sep 27, 2024
d33986c
Get workspace id from job parameters
JCZuurmond Sep 27, 2024
44f5f66
Set test local ucx catalog on installation context config
JCZuurmond Sep 27, 2024
965a8fc
Rename progress tracking installer to installation
JCZuurmond Sep 27, 2024
e4e84fc
Add ProgressTrackingInstallation to MockInstallationContext
JCZuurmond Sep 27, 2024
7070bba
Format
JCZuurmond Sep 27, 2024
fe943a0
Add unit test for workflow runs
JCZuurmond Sep 27, 2024
64d8e43
Be saver with finished at test
JCZuurmond Sep 27, 2024
ec2aada
Format README
JCZuurmond Sep 27, 2024
3280e85
Explain workflow runs in the README
JCZuurmond Sep 27, 2024
b3146b1
Fix `workflow_start_time` should be `start_time`
JCZuurmond Sep 27, 2024
9502991
Format
JCZuurmond Sep 27, 2024
57b7d8c
Get workspace id from workspace client
JCZuurmond Sep 27, 2024
c8de2c9
Fix typo's
JCZuurmond Oct 1, 2024
32c1d5e
Add type hint for `WorkflowRunRecorder`
JCZuurmond Oct 2, 2024
931dfeb
Improve comment
JCZuurmond Oct 2, 2024
92fb2fb
Improve docstring
JCZuurmond Oct 2, 2024
5adfc6a
Remove redundant argument
JCZuurmond Oct 2, 2024
b3f9be2
Add assert message
JCZuurmond Oct 2, 2024
d127cec
Get test workspace id from workspace client
JCZuurmond Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Test history installation to create history schema
  • Loading branch information
JCZuurmond committed Sep 27, 2024
commit 1e7cdac089c92ebc7a337982b232af0f48e311ea
7 changes: 7 additions & 0 deletions tests/unit/progress/test_install.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
from databricks.labs.ucx.progress.install import HistoryInstallation


def test_history_installation_run_creates_history_schema(mock_backend) -> None:
installation = HistoryInstallation(mock_backend, "ucx")
installation.run()
assert "CREATE SCHEMA IF NOT EXISTS `ucx`.`history`" == mock_backend.queries[0]