Added ipyflow-core to known list
#2945
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bc34eca to
b8c2175
Compare
nfx
added a commit
that referenced
this pull request
Oct 11, 2024
* Added `imageio` to known list ([#2942](#2942)). In this release, we have added `imageio` to our library's known list, which includes all its modules, sub-modules, testing, and typing packages. This change addresses issue [#1931](#1931), which may have been caused by a dependency or compatibility issue. The `imageio` library offers I/O functionality for scientific imaging data, and its addition is expected to expand the library's supported formats and functionality. As a result, software engineers can leverage the enhanced capabilities to handle scientific imaging data more effectively. * Added `ipyflow-core` to known list ([#2945](#2945)). In this release, the project has expanded its capabilities by adding two open-source libraries to a known list contained in a JSON file. The first library, `ipyflow-core`, brings a range of modules for the data model, experimental features, frontend, kernel, patches, shell, slicing, tracing, types, and utils. The second library, `pyccolo`, offers fast and adaptable code transformation using abstract syntax trees, with functionalities including code rewriting, import hooks, syntax augmentation, and tracing, along with various utility functions. By incorporating these libraries into the project, we aim to enhance its overall efficiency and versatility, providing software engineers with access to a broader set of tools and capabilities. * Added `isodate` to known list ([#2946](#2946)). In this release, we have added the `isodate` package to our library's known package list, which resolves part of issue [#1931](#1931). The `isodate` package provides several modules for parsing and manipulating ISO 8601 dated strings, including `isodate`, `isodate.duration`, `isodate.isodates`, `isodate.isodatetime`, `isodate.isoduration`, `isodate.isoerror`, `isodate.isostrf`, `isodate.isotime`, `isodate.isotzinfo`, and `isodate.tzinfo`. This addition enhances our compatibility and integration with the `isodate` package in the larger system, enabling users to utilize the full functionality of the `isodate` package in their applications. * Experimental command for enabling HMS federation ([#2939](#2939)). In this release, we have introduced an experimental feature for enabling HMS (Hive Metastore) federation through a new `enable-hms-federation` command in the labs.yml file. This command, when enabled, will create a federated HMS catalog synced with the workspace HMS in a hierarchical manner, facilitating migration and integration of HMS models. Additionally, we have added an optional `enable_hms_federation` constructor argument to the `Locations` class in the locations.py file. Setting this flag to True enables a fallback mode for AWS resources to use HMS for data access. The `HiveMetastoreFederationEnabler` class is introduced with an `enable()` method to modify the workspace configuration and enable HMS federation. These changes aim to provide a more streamlined experience for users working with complex modeling systems, and careful testing and feedback are encouraged on this experimental feature. * Experimental support for HMS federation ([#2283](#2283)). In this release, we introduce experimental support for Hive Metastore (HMS) federation in our open-source library. A new `HiveMetastoreFederation` class has been implemented, enabling the registration of an internal HMS as a federated catalog. This class utilizes the `WorkspaceClient` object from the `databricks.sdk` library to create necessary connections and handles permissions for successful federation. Additionally, a new file `test_federation.py` has been added, containing unit tests to demonstrate the functionality of HMS federation, including the creation of federated catalogs and handling of existing connections. As this is an experimental feature, users should expect potential issues and are encouraged to provide feedback to help improve its functionality. * Fixed `InvalidParameterValue` failure for scanning jobs running on interactive clusters that got deleted ([#2935](#2935)). In this release, we have addressed an issue where an `InvalidParameterValue` error was not being handled properly during scanning jobs run on interactive clusters that were deleted. This error has now been added to the exceptions handled in the `_register_existing_cluster_id` and `_register_cluster_info` methods. These methods retrieve information about an existing cluster or its ID, and if the cluster is not found or an invalid parameter value is provided, they now yield a `DependencyProblem` object with an appropriate error message. This `DependencyProblem` object is used to indicate that there is a problem with the dependencies required for the job, preventing it from running successfully. By handling this error, the code ensures that the job can fail gracefully and provide a clear and informative error message to the user, avoiding any potential confusion or unexpected behavior. * Improve logging when skipping legacy grant in `create-catalogs-schemas` ([#2933](#2933)). In this update, the `create-catalogs-schemas` process has been improved with enhanced logging for skipped legacy grants. This change is a follow-up to previous issue [#2917](#2917) and progresses issue [#2932](#2932). The `_apply_from_legacy_table_acls` and `_update_principal_acl` methods now include more descriptive logging when a legacy grant is skipped, providing information about the type of grant being skipped and clarifying that it is not supported in the Unity Catalog. Additionally, a new method `get_interactive_cluster_grants` has been added to the `principal_acl` object, returning a list of grants specific to the interactive cluster. The `hive_acl` object is now autospec'd after the `principal_acl.get_interactive_cluster_grants` call. The `test_catalog_schema_acl` function has been updated to reflect these changes. New grants have been added to the `hive_grants` list, including grants for `user1` with `USE` action type on `hive_metastore` catalog and grants for `user2` with `USAGE` action type on `schema3` database. A new grant for `user4` with `DENY` action type on `schema3` database has also been added, but it is skipped in the logging due to it not being supported in UC. Skipped legacy grants for `DENY` action type on `catalog2` catalog and 'catalog2.schema2' database are also included in the commit. These updates improve the clarity and usefulness of the logs, making it easier for users to understand what is happening during the migration of grants to UC and ensuring that unsupported grants are not inadvertently included in the UC. * Notebook linting: ensure path-type is preserved during linting ([#2923](#2923)). In this release, we have enhanced the type safety of the `NotebookResolver` class in the `loaders.py` module by introducing a new type variable `PathT`. This change includes an update to the `_adjust_path` method, which ensures the preservation of the original file suffix when adding the ".py" suffix for Python notebooks. This addresses a potential issue where a `WorkspacePath` instance could be incorrectly converted to a generic `Path` instance, causing downstream errors. Although this change may potentially resolve issue [#2888](#2888), the reproduction steps for that issue were not provided in the commit message. It is important to note that while this change has been manually tested, it does not include any new unit tests, integration tests, or staging environment verification.
Merged
nfx
added a commit
that referenced
this pull request
Oct 11, 2024
* Added `imageio` to known list ([#2942](#2942)). In this release, we have added `imageio` to our library's known list, which includes all its modules, sub-modules, testing, and typing packages. This change addresses issue [#1931](#1931), which may have been caused by a dependency or compatibility issue. The `imageio` library offers I/O functionality for scientific imaging data, and its addition is expected to expand the library's supported formats and functionality. As a result, software engineers can leverage the enhanced capabilities to handle scientific imaging data more effectively. * Added `ipyflow-core` to known list ([#2945](#2945)). In this release, the project has expanded its capabilities by adding two open-source libraries to a known list contained in a JSON file. The first library, `ipyflow-core`, brings a range of modules for the data model, experimental features, frontend, kernel, patches, shell, slicing, tracing, types, and utils. The second library, `pyccolo`, offers fast and adaptable code transformation using abstract syntax trees, with functionalities including code rewriting, import hooks, syntax augmentation, and tracing, along with various utility functions. By incorporating these libraries into the project, we aim to enhance its overall efficiency and versatility, providing software engineers with access to a broader set of tools and capabilities. * Added `isodate` to known list ([#2946](#2946)). In this release, we have added the `isodate` package to our library's known package list, which resolves part of issue [#1931](#1931). The `isodate` package provides several modules for parsing and manipulating ISO 8601 dated strings, including `isodate`, `isodate.duration`, `isodate.isodates`, `isodate.isodatetime`, `isodate.isoduration`, `isodate.isoerror`, `isodate.isostrf`, `isodate.isotime`, `isodate.isotzinfo`, and `isodate.tzinfo`. This addition enhances our compatibility and integration with the `isodate` package in the larger system, enabling users to utilize the full functionality of the `isodate` package in their applications. * Experimental command for enabling HMS federation ([#2939](#2939)). In this release, we have introduced an experimental feature for enabling HMS (Hive Metastore) federation through a new `enable-hms-federation` command in the labs.yml file. This command, when enabled, will create a federated HMS catalog synced with the workspace HMS in a hierarchical manner, facilitating migration and integration of HMS models. Additionally, we have added an optional `enable_hms_federation` constructor argument to the `Locations` class in the locations.py file. Setting this flag to True enables a fallback mode for AWS resources to use HMS for data access. The `HiveMetastoreFederationEnabler` class is introduced with an `enable()` method to modify the workspace configuration and enable HMS federation. These changes aim to provide a more streamlined experience for users working with complex modeling systems, and careful testing and feedback are encouraged on this experimental feature. * Experimental support for HMS federation ([#2283](#2283)). In this release, we introduce experimental support for Hive Metastore (HMS) federation in our open-source library. A new `HiveMetastoreFederation` class has been implemented, enabling the registration of an internal HMS as a federated catalog. This class utilizes the `WorkspaceClient` object from the `databricks.sdk` library to create necessary connections and handles permissions for successful federation. Additionally, a new file `test_federation.py` has been added, containing unit tests to demonstrate the functionality of HMS federation, including the creation of federated catalogs and handling of existing connections. As this is an experimental feature, users should expect potential issues and are encouraged to provide feedback to help improve its functionality. * Fixed `InvalidParameterValue` failure for scanning jobs running on interactive clusters that got deleted ([#2935](#2935)). In this release, we have addressed an issue where an `InvalidParameterValue` error was not being handled properly during scanning jobs run on interactive clusters that were deleted. This error has now been added to the exceptions handled in the `_register_existing_cluster_id` and `_register_cluster_info` methods. These methods retrieve information about an existing cluster or its ID, and if the cluster is not found or an invalid parameter value is provided, they now yield a `DependencyProblem` object with an appropriate error message. This `DependencyProblem` object is used to indicate that there is a problem with the dependencies required for the job, preventing it from running successfully. By handling this error, the code ensures that the job can fail gracefully and provide a clear and informative error message to the user, avoiding any potential confusion or unexpected behavior. * Improve logging when skipping legacy grant in `create-catalogs-schemas` ([#2933](#2933)). In this update, the `create-catalogs-schemas` process has been improved with enhanced logging for skipped legacy grants. This change is a follow-up to previous issue [#2917](#2917) and progresses issue [#2932](#2932). The `_apply_from_legacy_table_acls` and `_update_principal_acl` methods now include more descriptive logging when a legacy grant is skipped, providing information about the type of grant being skipped and clarifying that it is not supported in the Unity Catalog. Additionally, a new method `get_interactive_cluster_grants` has been added to the `principal_acl` object, returning a list of grants specific to the interactive cluster. The `hive_acl` object is now autospec'd after the `principal_acl.get_interactive_cluster_grants` call. The `test_catalog_schema_acl` function has been updated to reflect these changes. New grants have been added to the `hive_grants` list, including grants for `user1` with `USE` action type on `hive_metastore` catalog and grants for `user2` with `USAGE` action type on `schema3` database. A new grant for `user4` with `DENY` action type on `schema3` database has also been added, but it is skipped in the logging due to it not being supported in UC. Skipped legacy grants for `DENY` action type on `catalog2` catalog and 'catalog2.schema2' database are also included in the commit. These updates improve the clarity and usefulness of the logs, making it easier for users to understand what is happening during the migration of grants to UC and ensuring that unsupported grants are not inadvertently included in the UC. * Notebook linting: ensure path-type is preserved during linting ([#2923](#2923)). In this release, we have enhanced the type safety of the `NotebookResolver` class in the `loaders.py` module by introducing a new type variable `PathT`. This change includes an update to the `_adjust_path` method, which ensures the preservation of the original file suffix when adding the ".py" suffix for Python notebooks. This addresses a potential issue where a `WorkspacePath` instance could be incorrectly converted to a generic `Path` instance, causing downstream errors. Although this change may potentially resolve issue [#2888](#2888), the reproduction steps for that issue were not provided in the commit message. It is important to note that while this change has been manually tested, it does not include any new unit tests, integration tests, or staging environment verification.
|
✅ 43/43 passed, 1 flaky, 2 skipped, 1h15m29s total Flaky tests:
Running from acceptance #6672 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Added
ipyflow-coreto known listLinked issues
Partly resolve #1931