feat(ingest/databricks): ingest hive metastore by default, more docs

metadata-ingestion/docs/sources/databricks/unity-catalog_post.md

@@ -1,3 +1,25 @@
+
+
+#### Advanced
+
+##### Multiple Databricks Workspaces
+
+If you have multiple databricks workspaces <b>that point to the same UC metastore</b>, our suggestion is to use separate recipes for ingesting Hive Metastore catalog and Unity Catalog information schema.
+
+To ingest Hive metastore information schema
+- Setup one ingestion recipe per workspace
+- Use platform instance equivalent to workspace name
+- Ingest only hive_metastore catalog in the recipe using config `catalogs: ["hive_metastore"]`
+
+To ingest Unity Catalog information schema
+- Disable hive metastore catalog ingestion in the recipe using config `include_hive_metastore: False`
+- Ideally, just ingest from one workspace
+- To ingest from both workspaces (e.g. if each workspace has different permissions and therefore restricted view of catalogs):
+    - Use same platform instance for all workspaces using same UC metastore
+    - Ingest usage from only one workspace (you lose usage from other workspace)
+    - Use filters to only ingest each catalog once, but shouldn’t be necessary
+
+
 #### Troubleshooting
 
 ##### No data lineage captured or missing lineage

metadata-ingestion/docs/sources/databricks/unity-catalog_pre.md

@@ -13,6 +13,11 @@
     * Ownership of or `SELECT` privilege on any tables and views you want to ingest
     * [Ownership documentation](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/ownership.html)
     * [Privileges documentation](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/privileges.html)
+  + To ingest legacy hive_metastore catalog (`include_hive_metastore` - disabled by default), your service principal must have all of the following:
+    * `READ_METADATA` and `USAGE` privilege on `hive_metastore` catalog
+    * `READ_METADATA` and `USAGE` privilege on schemas you want to ingest
+    * `READ_METADATA` and `USAGE` privilege on tables and views you want to ingest
+    * [Hive Metastore Privileges documentation](https://docs.databricks.com/en/sql/language-manual/sql-ref-privileges-hms.html)
   + To ingest your workspace's notebooks and respective lineage, your service principal must have `CAN_READ` privileges on the folders containing the notebooks you want to ingest: [guide](https://docs.databricks.com/en/security/auth-authz/access-control/workspace-acl.html#folder-permissions).
   + To `include_usage_statistics` (enabled by default), your service principal must have `CAN_MANAGE` permissions on any SQL Warehouses you want to ingest: [guide](https://docs.databricks.com/security/auth-authz/access-control/sql-endpoint-acl.html).
   + To ingest `profiling` information with `method: ge`, you need `SELECT` privileges on all profiled tables.

metadata-ingestion/src/datahub/ingestion/source/unity/config.py

@@ -126,7 +126,7 @@ class UnityCatalogSourceConfig(
         description="SQL Warehouse id, for running queries. If not set, will use the default warehouse.",
     )
     include_hive_metastore: bool = pydantic.Field(
-        default=False,
+        default=True,
         description="Whether to ingest legacy `hive_metastore` catalog. This requires executing queries on SQL warehouse.",
     )
     workspace_name: Optional[str] = pydantic.Field(
@@ -135,7 +135,7 @@ class UnityCatalogSourceConfig(
     )
 
     include_metastore: bool = pydantic.Field(
-        default=True,
+        default=False,
         description=(
             "Whether to ingest the workspace's metastore as a container and include it in all urns."
             " Changing this will affect the urns of all entities in the workspace."