Release v0.43.0

[nfx]

• Added imageio to known list (#2942). In this release, we have added imageio to our library's known list, which includes all its modules, sub-modules, testing, and typing packages. This change addresses issue #1931, which may have been caused by a dependency or compatibility issue. The imageio library offers I/O functionality for scientific imaging data, and its addition is expected to expand the library's supported formats and functionality. As a result, software engineers can leverage the enhanced capabilities to handle scientific imaging data more effectively.
• Added ipyflow-core to known list (#2945). In this release, the project has expanded its capabilities by adding two open-source libraries to a known list contained in a JSON file. The first library, ipyflow-core, brings a range of modules for the data model, experimental features, frontend, kernel, patches, shell, slicing, tracing, types, and utils. The second library, pyccolo, offers fast and adaptable code transformation using abstract syntax trees, with functionalities including code rewriting, import hooks, syntax augmentation, and tracing, along with various utility functions. By incorporating these libraries into the project, we aim to enhance its overall efficiency and versatility, providing software engineers with access to a broader set of tools and capabilities.
• Added isodate to known list (#2946). In this release, we have added the isodate package to our library's known package list, which resolves part of issue #1931. The isodate package provides several modules for parsing and manipulating ISO 8601 dated strings, including isodate, isodate.duration, isodate.isodates, isodate.isodatetime, isodate.isoduration, isodate.isoerror, isodate.isostrf, isodate.isotime, isodate.isotzinfo, and isodate.tzinfo. This addition enhances our compatibility and integration with the isodate package in the larger system, enabling users to utilize the full functionality of the isodate package in their applications.
• Experimental command for enabling HMS federation (#2939). In this release, we have introduced an experimental feature for enabling HMS (Hive Metastore) federation through a new enable-hms-federation command in the labs.yml file. This command, when enabled, will create a federated HMS catalog synced with the workspace HMS in a hierarchical manner, facilitating migration and integration of HMS models. Additionally, we have added an optional enable_hms_federation constructor argument to the Locations class in the locations.py file. Setting this flag to True enables a fallback mode for AWS resources to use HMS for data access. The HiveMetastoreFederationEnabler class is introduced with an enable() method to modify the workspace configuration and enable HMS federation. These changes aim to provide a more streamlined experience for users working with complex modeling systems, and careful testing and feedback are encouraged on this experimental feature.
• Experimental support for HMS federation (#2283). In this release, we introduce experimental support for Hive Metastore (HMS) federation in our open-source library. A new HiveMetastoreFederation class has been implemented, enabling the registration of an internal HMS as a federated catalog. This class utilizes the WorkspaceClient object from the databricks.sdk library to create necessary connections and handles permissions for successful federation. Additionally, a new file test_federation.py has been added, containing unit tests to demonstrate the functionality of HMS federation, including the creation of federated catalogs and handling of existing connections. As this is an experimental feature, users should expect potential issues and are encouraged to provide feedback to help improve its functionality.
• Fixed InvalidParameterValue failure for scanning jobs running on interactive clusters that got deleted (#2935). In this release, we have addressed an issue where an InvalidParameterValue error was not being handled properly during scanning jobs run on interactive clusters that were deleted. This error has now been added to the exceptions handled in the _register_existing_cluster_id and _register_cluster_info methods. These methods retrieve information about an existing cluster or its ID, and if the cluster is not found or an invalid parameter value is provided, they now yield a DependencyProblem object with an appropriate error message. This DependencyProblem object is used to indicate that there is a problem with the dependencies required for the job, preventing it from running successfully. By handling this error, the code ensures that the job can fail gracefully and provide a clear and informative error message to the user, avoiding any potential confusion or unexpected behavior.
• Improve logging when skipping legacy grant in create-catalogs-schemas (#2933). In this update, the create-catalogs-schemas process has been improved with enhanced logging for skipped legacy grants. This change is a follow-up to previous issue #2917 and progresses issue #2932. The _apply_from_legacy_table_acls and _update_principal_acl methods now include more descriptive logging when a legacy grant is skipped, providing information about the type of grant being skipped and clarifying that it is not supported in the Unity Catalog. Additionally, a new method get_interactive_cluster_grants has been added to the principal_acl object, returning a list of grants specific to the interactive cluster. The hive_acl object is now autospec'd after the principal_acl.get_interactive_cluster_grants call. The test_catalog_schema_acl function has been updated to reflect these changes. New grants have been added to the hive_grants list, including grants for user1 with USE action type on hive_metastore catalog and grants for user2 with USAGE action type on schema3 database. A new grant for user4 with DENY action type on schema3 database has also been added, but it is skipped in the logging due to it not being supported in UC. Skipped legacy grants for DENY action type on catalog2 catalog and 'catalog2.schema2' database are also included in the commit. These updates improve the clarity and usefulness of the logs, making it easier for users to understand what is happening during the migration of grants to UC and ensuring that unsupported grants are not inadvertently included in the UC.
• Notebook linting: ensure path-type is preserved during linting (#2923). In this release, we have enhanced the type safety of the NotebookResolver class in the loaders.py module by introducing a new type variable PathT. This change includes an update to the _adjust_path method, which ensures the preservation of the original file suffix when adding the ".py" suffix for Python notebooks. This addresses a potential issue where a WorkspacePath instance could be incorrectly converted to a generic Path instance, causing downstream errors. Although this change may potentially resolve issue #2888, the reproduction steps for that issue were not provided in the commit message. It is important to note that while this change has been manually tested, it does not include any new unit tests, integration tests, or staging environment verification.