2.3.1

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.3.1

Version 2.3.1

Features

• Rolled back support of Delta Merge statement. There was a false positive in our test suite but additional tests were added and verified that Delta Merge is still not supported due to proprietary implementation from Databricks.

Major Bug Fixes

• Fixed issue with Library definition in Job Tasks causing deserialization error (#177)
• Fixed issue where mount points refer to nested folder (#175)

Other

• Added a utility for removing spaces from OlToPurviewMapping.json ./deployment/util/mappings-remove-spaces.py

Upgrade from 2.3.0 to 2.3.1

OpenLineage / Databricks Installation

• No changes to the OpenLineage / Databricks Installation

Changes to Function App Settings

• Update OlToPurviewMappings with the latest mappings found in the deployment/infra/OlToPurviewMappings.json file.

Changes to Function App

You will need to deploy the updated function code to Azure. You can do so with either of the following:

• Cloning the repo and deploying via VS code's Azure Functions Extension.
• Using the Azure CLI to do a zip push deployment with the 2.3.0 zip release artifact.

az functionapp deployment source config-zip \
-g <RESOURCE_GROUP_NAME> \
-n <FUNCTION_SERVICE_NAME> \
--src <PATH_TO_RELEASE_ZIP_FILE> 

2.3.0

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don't see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.3

Version 2.3

Features

• Support for OpenLineage version 18
• Column level lineage support in parser for Hive and Hive Delta
• Support for new source types:
  • Azure Data Explorer (Kusto)
  • Azure Postgres
  • Postgres
  • Azure MySQL
• ADB managed Hive table lineage support
• Supporting Databricks Notebook orchestrated via Azure Data Factory

Major Bug Fixes

• Improved search and match for Azure Blob and Azure Data Lake Storage Gen 2 resource sets

Other

• Spark 2 has been removed from our test suite. While the solution may continue to work on Spark 2, we do not guarantee it and will not be able to support investigating issues on Spark 2.

Upgrade from 2.2 to 2.3

You will need to deploy the updated function code to Azure. You can do so with either of the following:

• Cloning the repo and deploying via VS code's Azure Functions Extension.
• Using the Azure CLI to do a zip push deployment with the 2.3.0 zip release artifact.

az functionapp deployment source config-zip \ -g <RESOURCE_GROUP_NAME> \ -n <FUNCTION_SERVICE_NAME> \ --src <PATH_TO_RELEASE_ZIP_FILE>

OpenLineage / Databricks Installation

• Remove the openlineage-spark-0.13.0.jar and install the openlineage-spark-0.18.0.jar

Changes to Function App Settings

• Update OlToPurviewMappings with the latest mappings found in the deployment/infra/OlToPurviewMappings.json file.

Full Changelog: 2.2.0...2.3.0

2.1.2

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.1.2

Version 2.1.2

Features

• Improved the ARM template to avoid race condition during deployments.

Major Bug Fixes

• N/A

Other

• N/A

Upgrade from 2.1.1 to 2.1.2

If you have a successfully deployed instance, no change is necessary. The changes are only in the ARM template and affect new deployments.

OpenLineage / Databricks Installation

• There were no changes to the Solution Accelerator Function code

Changes to Function App Settings

• There were no changes to the OlToPurviewMappings file.

2.2.2

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.2.2

Version 2.2.2

Features

• Improved the ARM template to avoid race condition during deployments.

Major Bug Fixes

• N/A

Other

• N/A

Upgrade from 2.2.1 to 2.2.2

If you have a successfully deployed instance, no change is necessary. The changes are only in the ARM template and affect new deployments.

OpenLineage / Databricks Installation

• There were no changes to the Solution Accelerator Function code

Changes to Function App Settings

• There were no changes to the OlToPurviewMappings file.

2.2.1

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.2.1

Version 2.2.1

Features

• Further improved search matching

Major Bug Fixes

• N/A

Other

• Greatly improved test harness

Upgrade from 2.2.0 to 2.2.1

OpenLineage / Databricks Installation

You will need to deploy the updated function code to Azure. You can do so by:

• Cloning the repo and deploying via VS code's Azure Functions Extension.
• Using the Azure CLI to do a zip push deployment with the 2.2.1 zip release artifact.

  az functionapp deployment source config-zip \
  -g <RESOURCE_GROUP_NAME> \
  -n <FUNCTION_SERVICE_NAME> \
  --src <PATH_TO_RELEASE_ZIP_FILE>
  

Changes to Function App Settings

• There were no changes to the OlToPurviewMappings file.

2.2.0

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.2

Version 2.2

Features

• Improved the search matching with PR #119 and PR #124
  • PR #119 prioritize resource sets first regardless of the top search match returned by Microsoft Purview query api.
  • This can be deactivated by setting the app setting prioritizeFirstResourceSet to false.
• For very large spark plans, you can remove them from the Atlas / Microsoft Purview payload by setting the app setting maxQueryPlanSize to the maximum number of bytes you want to allow.
  • The maximum size in bytes for event hub is 1 MB
  • If you commonly have large, complex queries, you should consider setting the maxQueryPlanSize to some value under 1000000.

Major Bug Fixes

• Corrected possible race condition in ARM template with PR #114 (contributed by community member! Thank you!!)
• Removed the ARM template parameter listenToMessagesFromPurviewKafka since it was unused and causing confusion with PR #122 .

Other

• Added documentation for enabling Private Endpoints manually.

Upgrade from 2.1 to 2.2

OpenLineage / Databricks Installation

You will need to deploy the updated function code to Azure. You can do so by:

• Cloning the repo and deploying via VS code's Azure Functions Extension.
• Using the Azure CLI to do a zip push deployment with the 2.2.0 zip release artifact.

  az functionapp deployment source config-zip \
  -g <RESOURCE_GROUP_NAME> \
  -n <FUNCTION_SERVICE_NAME> \
  --src <PATH_TO_RELEASE_ZIP_FILE>
  

Changes to Function App Settings

• There were no changes to the OlToPurviewMappings file.

2.1.1

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.1.1

Version 2.1.1

Features

• No new features in this release.

Major Bug Fixes

• If your mount points overlapped (e.g. /mnt/x and /mnt/x/y) the solution accelerator would find only the shortest path (/mnt/x/`) even if you were specifying the longer path in your code. Now, the solution accelerator will match on the longest (in terms of folder hierarchy) path first.
• When using managed Hive tables in Databricks (i.e. default database pointing to the default DBFS location of /usr/hive/warehouse), the solution accelerator may match to the wrong entity given Purview's inability to handle @ in the query string. Now the solution accelerator can more accurately find match tables or create a placeholder entity if the hive metastore has not been scanned.
  • This requires that the spark.openlineage.namespace is set to the pattern adb-<workspace-id>#<cluster-id>

Other Updates

• The demo environment has been updated to support the hive managed tables scenario out of the box.

Upgrade from 2.1.0 to 2.1.1

You will need to deploy the updated function code to Azure. You can do so by:

• Cloning the repo and deploying via VS code's Azure Functions Extension.
• Using the Azure CLI to do a zip push deployment with the 2.1.1 zip release artifact.

What's Changed

• Add additional troubleshooting guidance from August 2022 by @wjohnson in #94
• Mount Point matching should look for the longest string match by @wjohnson in #108
• Hotfix/tests and output with notebook2 1 by @wjohnson in #109
• Enable Hive Managed Tables in Demo Environment by @wjohnson in #110

Full Changelog: 2.1.0...2.1.1

2.1.0

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.1

Version 2.1

Features

• Support for OpenLineage version 13
• Column level lineage support in parser for Hive and Hive Delta
  • Note column lineage support for other sources will be coming as OpenLineage supports them
    • OpenLineage column support anticipated roadmap
      • ABFSS: September
      • WASBS: September
      • SQL / JDBC: Q4 CY 2022
      • Postgres / JDBC: Q4 CY 2022
      • MySQL / JDBC: Q4 CY 2022
• Support for new source types:
  • MySQL
  • Postgres
• ADB managed Hive table lineage support

Major Bug Fixes

• OpenLineage 13 resolves SaveAsTable driver crash issue

Other

• Improved documentation for custom source configuration
• Created custom source gallery adding ADLS Gen 1

Upgrade from 2.0 to 2.1

OpenLineage / Databricks Installation

• Remove the openlineage-spark-0.11.0.jar and install the openlineage-spark-0.13.0.jar

Changes to Function App Settings

• Update OlToPurviewMappings with the latest mappings found in the deployment/infra/OlToPurviewMappings.json file.

2.0.4

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 2.0.4

Version 2.0.4

Bug Fixes

All fixes are around the demo deployment. If you are using the connector only deployment or have already deployed the solution accelerator, there is no action for you to take in this release.

For those who were having an issue with the demo deployment, please checkout this release and deploy the demo.

• Corrected mistaken az cli command that prevented key vault access policy being applied to deploying user.

Full Changelog: 2.0.3...2.0.4

1.1.1

Release notes

• These release notes provide information about new features, fixes, and updates that are included in the latest update release.
• We often roll out features (and sometimes even fixes) to Current over a period of time. If you don’t see something described below right away, you can expect it soon

ADB to Purview Connector Solution Accelerator - Version 1.1.1

Version 1.1.1

Bug Fixes

All fixes are around the demo deployment. If you are using the connector only deployment or have already deployed the solution accelerator, there is no action for you to take in this release.

For those who were having an issue with the demo deployment, please checkout this release and deploy the demo.

• Added a retry attempt for the ARM template deployment if there is an intermittent error causing the deployment to fail.
• Fixed Key Vault Access Policy Assignment for end user doing deployment.
• Fixed Storage Blob Data Reader role assignment for Purview Managed Service Identity.

Full Changelog: 1.1.0...1.1.1