Create EXTERNAL LOCATIONs to map to External Tables (Azure)

[nfx]

Related issues:

• Migrate Azure Service Principals that access storage to UC Storage Credentials #339
• [FEATURE]: Migrate AWS IAM Instance Profiles to UC Storage Credentials #862
• Report AWS Instance Profile to UC Storage Credentials (via query) #338

Jobs to be done:

• use ExternalLocations from https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/hive_metastore/locations.py and tables.TablesCrawler to verify if we have matching external locations configured already - use ws.external_locations.list() API for this.
• validate the logic of ExternalLocations for correctness and corner-cases
• create databricks labs ucx validate-external-locations command to display a number of tables per external location and any HMS table locations that don't belong to an existing external location
• print Terraform resource definitions to have those external locations setup

[nfx]

@zpappa this is all the information about the feature straight from PRD, btw 😉

at this stage, it's more of a one-decision design document specifically for this feature, even though we don't do that for all of the features

We have to aim at expanding the issue description here as explicitly as possible, otherwise people doing the implementation will diverge from logic.

[pohlposition]

Relates to
#339 (Migrate Azure Service Principals that access storage to UC Storage Credentials)
and
#338 (Migrate AWS Instance Profile to UC Storage Credentials)

[zpappa]

So there's a bit of a catch-22 here.

We can try to force create a credential if it hasn't been set up on the cloud provider, but it may not work and you won't be able to sync tables or create external locations with that credential.

I think what needs to happen here is:

1. We gather information from cloud provider about the credentials and what storage accounts they have access to
2. We produce a report with some commands that a customer can run to create credentials
3. OFFLINE, the customer has to go and create these credentials

• What if they renamed the credentials, potential interactivity or mapping step
• What if they collapsed credentials, potential interactivity or mapping step

4. We create credentials in Unity
5. We use the mapping information to register external locations
6. Then we can use storage paths of external tables to map to their containing external locations