Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add migrate-locations command #1016

Merged
merged 4 commits into from
Mar 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,12 +427,25 @@ databricks labs ucx validate-external-locations
```

Once the [`assessment` workflow](#assessment-workflow) finished successfully, [storage credentials](#migrate-credentials-command) are configured,
run this command to ensure the relevant Unity Catalog external locations are created if they are missing.
run this command to validate and report the missing Unity Catalog external locations to be created.

This command validates and provides mapping to external tables to external locations, also as Terraform configurations.

[[back to top](#databricks-labs-ucx)]


## `migrate-locations` command
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refresh table of contents (there's ide plugin for that)


```text
databricks labs ucx migrate-locations
```

Once the [`assessment` workflow](#assessment-workflow) finished successfully, and [storage credentials](#migrate-credentials-command) are configured,
run this command to have Unity Catalog external locations created. The candidate locations to be created are extracted from guess_external_locations
task in the assessment job. You can run `validate_external_locations` command to check the candidate locations.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Insert valid markdown link and correct command name


[[back to top](#databricks-labs-ucx)]

## `create-table-mapping` command

```text
Expand Down
3 changes: 3 additions & 0 deletions labs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,6 @@ commands:
flags:
- name: workspace_ids
description: List of workspace IDs to create account groups from.

- name: migrate_locations
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: migrate_locations
- name: migrate-locations

Dashes, not underscores

description: Create UC external locations based on the output of guess_external_locations assessment task.
18 changes: 18 additions & 0 deletions src/databricks/labs/ucx/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from databricks.labs.ucx.assessment.aws import AWSResourcePermissions
from databricks.labs.ucx.azure.access import AzureResourcePermissions
from databricks.labs.ucx.azure.credentials import ServicePrincipalMigration
from databricks.labs.ucx.azure.locations import ExternalLocationsMigration
from databricks.labs.ucx.config import WorkspaceConfig
from databricks.labs.ucx.framework.crawlers import StatementExecutionBackend
from databricks.labs.ucx.hive_metastore import ExternalLocations, TablesCrawler
Expand Down Expand Up @@ -344,5 +345,22 @@ def create_uber_principal(w: WorkspaceClient, subscription_id: str):
return


@ucx.command
def migrate_locations(w: WorkspaceClient):
"""This command creates UC external locations. The candidate locations to be created are extracted from guess_external_locations
task in the assessment job. You can run validate_external_locations command to check the candidate locations. Please make sure
the credentials haven migrated before running this command. The command will only create the locations that have corresponded UC Storage Credentials.
"""
if w.config.is_azure:
logger.info("Running migrate_locations for Azure")
installation = Installation.current(w, 'ucx')
service_principal_migration = ExternalLocationsMigration.for_cli(w, installation)
service_principal_migration.run()
if w.config.is_aws:
logger.error("migrate_locations is not yet supported in AWS")
if w.config.is_gcp:
logger.error("migrate_locations is not yet supported in GCP")


if __name__ == "__main__":
ucx()
25 changes: 25 additions & 0 deletions tests/unit/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
installations,
manual_workspace_info,
migrate_credentials,
migrate_locations,
move,
open_remote_config,
principal_prefix_access,
Expand Down Expand Up @@ -357,3 +358,27 @@ def test_create_master_principal(ws):
with patch("databricks.labs.blueprint.tui.Prompts.question", return_value=True):
with pytest.raises(ValueError):
create_uber_principal(ws, subscription_id="12")


def test_migrate_locations_azure(ws):
ws.config.is_azure = True
ws.config.is_aws = False
ws.config.is_gcp = False
migrate_locations(ws)
ws.external_locations.list.assert_called()


def test_migrate_locations_aws(ws, caplog):
ws.config.is_azure = False
ws.config.is_aws = True
ws.config.is_gcp = False
migrate_locations(ws)
assert "migrate_locations is not yet supported in AWS" in caplog.messages


def test_migrate_locations_gcp(ws, caplog):
ws.config.is_azure = False
ws.config.is_aws = False
ws.config.is_gcp = True
migrate_locations(ws)
assert "migrate_locations is not yet supported in GCP" in caplog.messages
Loading