v0.7.0

• Added databricks.labs.blueprint.paths.WorkspacePath as pathlib.Path equivalent (#115). This commit introduces the databricks.labs.blueprint.paths.WorkspacePath library, providing Python-native pathlib.Path-like interfaces to simplify working with Databricks Workspace paths. The library includes WorkspacePath and WorkspacePathDuringTest classes offering advanced functionality for handling user home folders, relative file paths, browser URLs, and file manipulation methods such as read/write_text(), read/write_bytes(), and glob(). This addition brings enhanced, Pythonic ways to interact with Databricks Workspace paths, including creating and moving files, managing directories, and generating browser-accessible URIs. Additionally, the commit includes updates to existing methods and introduces new fixtures for creating notebooks, accompanied by extensive unit tests to ensure reliability and functionality.
• Added propagation of blueprint version into User-Agent header when it is used as library (#114). A new feature has been introduced in the library that allows for the propagation of the blueprint version and the name of the command line interface (CLI) command used in the User-Agent header when the library is utilized as a library. This feature includes the addition of two new pairs of OtherInfo: blueprint/X.Y.Z to indicate that the request is made using the blueprint library and cmd/<name> to store the name of the CLI command used for making the request. The implementation involves using the with_user_agent_extra function from databricks.sdk.config to set the user agent consistently with the Databricks CLI. Several changes have been made to the test file for test_useragent.py to include a new test case, test_user_agent_is_propagated, which checks if the blueprint version and the name of the command are correctly propagated to the User-Agent header. A context manager http_fixture_server has been added that creates an HTTP server with a custom handler, which extracts the blueprint version and the command name from the User-Agent header and stores them in the user_agent dictionary. The test case calls the foo command with a mocked WorkspaceClient instance and sets the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables to test the propagation of the blueprint version and the command name in the User-Agent header. The test case then asserts that the blueprint version and the name of the command are present and correctly set in the user_agent dictionary.
• Bump actions/checkout from 4.1.6 to 4.1.7 (#112). In this release, the version of the "actions/checkout" action used in the Checkout Code step of the acceptance workflow has been updated from 4.1.6 to 4.1.7. This update may include bug fixes, performance improvements, and new features, although specific changes are not mentioned in the commit message. The Unshallow step remains unchanged, continuing to fetch and clean up the repository's history. This update ensures that the latest enhancements from the "actions/checkout" action are utilized, aiming to improve the reliability and performance of the code checkout process in the GitHub Actions workflow. Software engineers should be aware of this update and its potential impact on their workflows.

Dependency updates:

• Bump actions/checkout from 4.1.6 to 4.1.7 (#112).

Contributors: @nfx, @dependabot[bot]

v0.6.3

• fixed Command.get_argument_type bug with UnionType (#110). In this release, the Command.get_argument_type method has been updated to include special handling for UnionType, resolving a bug that caused the function to crash when encountering this type. The method now returns the string representation of the annotation if the argument is a UnionType, providing more accurate and reliable results. To facilitate this, modifications were made using the types module. Additionally, the foo function has a new optional argument optional_arg of type str, with a default value of None. This argument is passed to the some function in the assertion. The Prompts type has been added to the foo function signature, and an assertion has been added to verify if prompts is an instance of Prompts. Lastly, the default value of the address argument has been changed from an empty string to "default", and the same changes have been applied to the test_injects_prompts test function.

Contributors: @nkvuong

v0.6.2

• Applied type casting & remove empty kwarg for Command (#108). A new method, get_argument_type, has been added to the Command class in the cli.py file to determine the type of a given argument name based on the function's signature. The _route method has been updated to remove any empty keyword arguments from the kwargs dictionary, and apply type casting based on the argument type using the get_argument_type method. This ensures that the kwargs passed into App.command are correctly typed and eliminates any empty keyword arguments, which were previously passed as empty strings. In the test file for the command-line interface, the foo command's keyword arguments have been updated to include age (int), salary (float), is_customer (bool), and address (str) types, with the name argument remaining and a default value for address. The test_commands and test_injects_prompts functions have been updated accordingly. These changes aim to improve the input validation and type safety of the App.command method.

Contributors: @nkvuong

v0.6.1

• Made ProductInfo.version a cached_property to avoid failure when comparing wheel uploads in development (#105). In this release, the apply method of a class has been updated to sort upgrade scripts in semantic versioning order before applying them, addressing potential issues with version comparison during development. The implementation of ProductInfo.version has been refactored to a cached_property called _version, which calculates and caches the project version, addressing a failure during wheel upload comparisons in development. The Wheels class constructor has also been updated to include explicit keyword-only arguments, and a deprecation warning has been added. These changes aim to improve the reliability and predictability of the upgrade process and the library as a whole.

Dependency updates:

• Bump actions/checkout from 4.1.5 to 4.1.6 (#106).

Contributors: @dependabot[bot], @nkvuong

v0.6.0

• Added upstream wheel uploads for Databricks Workspaces without Public Internet access (#99). This commit introduces a new feature for uploading upstream wheel dependencies to Databricks Workspaces without Public Internet access. A new flag has been added to upload functions, allowing users to include or exclude dependencies in the download list. The WheelsV2 class has been updated with a new method, upload_wheel_dependencies(prefixes), which checks if each wheel's name starts with any of the provided prefixes before uploading it to the Workspace File System (WSFS). This feature also includes two new tests to verify the functionality of uploading the main wheel package and dependent wheel packages, optimizing downloads based on specific use cases. This enables users to more easily use the package in offline environments with restricted internet access, particularly for Databricks Workspaces with extra layers of network security.
• Fixed bug for double-uploading of unreleased wheels in air-gapped setups (#103). In this release, we have addressed a bug in the upload_wheel_dependencies method of the WheelsV2 class, which caused double-uploading of unreleased wheels in air-gapped setups. This issue occurred due to the condition if wheel.name == self._local_wheel.name not being met, resulting in undefined behavior. We have introduced a cached property _current_version to tackle this bug for unreleased versions uploaded to air-gapped workspaces. We also added a new method, upload_to_wsfs(), that uploads files to the workspace file system (WSFS) in the integration test. This release also includes new tests to ensure that only the Databricks SDK is uploaded and that the number of installation files is correct. These changes have resolved the double-uploading issue, and the number of installation files, Databricks SDK, Blueprint, and version.json metadata are now uploaded correctly to WSFS.

Contributors: @aminmovahed-db, @nfx

v0.5.0

• Added content assertion for assert_file_uploaded and assert_file_dbfs_uploaded in MockInstallation (#101). The recent commit introduces a content assertion feature to the MockInstallation class, enhancing its testing capabilities. This is achieved by adding an optional expected parameter of type bytes to the assert_file_uploaded and assert_file_dbfs_uploaded methods, allowing users to verify the uploaded content's correctness. The _assert_upload method has also been updated to accept this new parameter, ensuring the actual uploaded content matches the expected content. Furthermore, the commit includes informative docstrings for the new and updated methods, providing clear explanations of their functionality and usage. To support these improvements, new test cases test_assert_file_uploaded and test_load_empty_data_class have been added to the tests/unit/test_installation.py file, enabling more rigorous testing of the MockInstallation class and ensuring that the expected content is uploaded correctly.
• Added handling for partial functions in parallel.Threads (#93). In this release, we have enhanced the parallel.Threads module with the ability to handle partial functions, addressing issue #93. This improvement includes the addition of a new static method, _get_result_function_signature, to obtain the signature of a function or a string representation of its arguments and keywords if it is a partial function. The _wrap_result class method has also been updated to log an error message with the function's signature if an exception occurs. Furthermore, we have added a new test case, test_odd_partial_failed, to the unit tests, ensuring that the gather function handles partial functions that raise errors correctly. The Python version required for this project remains at 3.10, and the pyproject.toml file has been updated to include "isort", "mypy", "types-PyYAML", and types-requests in the list of dependencies. These adjustments are aimed at improving the functionality and type checking in the parallel.Threads module.
• Align configurations with UCX project (#96). This commit brings project configurations in line with the UCX project through various fixes and updates, enhancing compatibility and streamlining collaboration. It addresses pylint configuration warnings, adjusts GitHub Actions workflows, and refines the pyproject.toml file. Additionally, the NiceFormatter class in logger.py has been improved for better code readability, and the versioning scheme has been updated to ensure SemVer and PEP440 compliance, making it easier to manage and understand the project's versioning. Developers adopting the project will benefit from these alignments, as they promote adherence to the project's standards and up-to-date best practices.
• Check backwards compatibility with UCX, Remorph, and LSQL (#84). This release includes an update to the dependabot configuration to check for daily updates in both the pip and github-actions package ecosystems, with a new directory parameter added for the pip ecosystem for more precise update management. Additionally, a new GitHub Actions workflow, "downstreams", has been added to ensure backwards compatibility with UCX, Remorph, and LSQL by running automated downstream checks on pull requests, merge groups, and pushes to the main branch. The workflow has appropriate permissions for writing id-tokens, reading contents, and writing pull-requests, and runs the downstreams action from the databrickslabs/sandbox repository using GITHUB_TOKEN for authentication. These changes improve the security and maintainability of the project by ensuring compatibility with downstream projects and staying up-to-date with the latest package versions, reducing the risk of potential security vulnerabilities and bugs.

Dependency updates:

• Bump actions/setup-python from 4 to 5 (#89).
• Bump softprops/action-gh-release from 1 to 2 (#87).
• Bump actions/checkout from 2.5.0 to 4.1.2 (#88).
• Bump codecov/codecov-action from 1 to 4 (#85).
• Bump actions/checkout from 4.1.2 to 4.1.3 (#95).
• Bump actions/checkout from 4.1.3 to 4.1.5 (#100).

Contributors: @dependabot[bot], @nfx, @grusin-db, @nkvuong

v0.4.4

• If Threads.strict() raises just one error, don't wrap it with ManyError (#79). The strict method in the gather function of the parallel.py module in the databricks/labs/blueprint package has been updated to change the way it handles errors. Previously, if any task in the tasks sequence failed, the strict method would raise a ManyError exception containing all the errors. With this change, if only one error occurs, that error will be raised directly without being wrapped in a ManyError exception. This simplifies error handling and avoids unnecessary nesting of exceptions. Additionally, the __tracebackhide__ dunder variable has been added to the method to improve the readability of tracebacks by hiding it from the user. This update aims to provide a more streamlined and user-friendly experience for handling errors in parallel processing tasks.

Contributors: @nfx

v0.4.3

• Fixed marshalling & unmarshalling edge cases (#76). The serialization and deserialization methods in the code have been updated to improve handling of edge cases during marshalling and unmarshalling of data. When encountering certain edge cases, the _marshal_list method will now return an empty list instead of None, and both the _unmarshal and _unmarshal_dict methods will return None as is if the input is None. Additionally, the _unmarshal method has been updated to call _unmarshal_generic instead of checking if the type reference is a dictionary or list when it is a generic alias. The _unmarshal_generic method has also been updated to handle cases where the input is None. A new test case, test_load_empty_data_class(), has been added to the tests/unit/test_installation.py file to verify this behavior, ensuring that the correct behavior is maintained when encountering these edge cases during the marshalling and unmarshalling processes. These changes increase the reliability of the serialization and deserialization processes.

Contributors: @nkvuong

v0.4.2

• Fixed edge cases when loading typing.Dict, typing.List and typing.ClassVar (#74). In this release, we have implemented changes to improve the handling of edge cases related to the Python typing.Dict, typing.List, and typing.ClassVar during serialization and deserialization of dataclasses and generic types. Specifically, we have modified the _marshal and _unmarshal functions to check for the __origin__ attribute to determine whether the type is a ClassVar and skip it if it is. The _marshal_dataclass and _unmarshal_dataclass functions now check for the __dataclass_fields__ attribute to ensure that only dataclass fields are marshaled and unmarshaled. We have also added a new unit test for loading a complex data class using the MockInstallation class, which contains various attributes such as a string, a nested dictionary, a list of Policy objects, and a dictionary mapping string keys to Policy objects. This test case checks that the installation object correctly serializes and deserializes the ComplexClass instance to and from JSON format according to the specified attribute types, including handling of the typing.Dict, typing.List, and typing.ClassVar types. These changes improve the reliability and robustness of our library in handling complex data types defined in the typing module.
• MockPrompts.extend() now returns a copy (#72). In the latest release, the extend() method in the MockPrompts class of the tui.py module has been enhanced. Previously, extend() would modify the original MockPrompts object, which could lead to issues when reusing the same object in multiple places within the same test, as its state would be altered each time extend() was called. This has been addressed by updating the extend() method to return a copy of the MockPrompts object with the updated patterns and answers, instead of modifying the original object. This change ensures that the original MockPrompts object can be securely reused in multiple test scenarios without unintended side effects, preserving the integrity of the original state. Furthermore, additional tests have been incorporated to verify the correct behavior of both the new and original prompts.

Contributors: @pritishpai, @nkvuong

v0.4.1

• Fixed MockInstallation to emulate workspace-global setup (#69). In this release, the MockInstallation class in the installation module has been updated to better replicate a workspace-global setup, enhancing testing and development accuracy. The is_global method now utilizes the product method instead of _product, and a new instance variable _is_global with a default value of True is introduced in the __init__ method. Moreover, a new product method is included, which consistently returns the string "mock". These enhancements resolve issue #69, "Fixed MockInstallation to emulate workspace-global setup", ensuring the MockInstallation instance behaves as a global installation, facilitating precise and reliable testing and development for our software engineering team.
• Improved MockPrompts with extend() method (#68). In this release, we've added an extend() method to the MockPrompts class in our library's TUI module. This new method allows developers to add new patterns and corresponding answers to the existing list of questions and answers in a MockPrompts object. The added patterns are compiled as regular expressions and the questions and answers list is sorted by the length of the regular expression patterns in descending order. This feature is particularly useful for writing tests where prompt answers need to be changed, as it enables better control and customization of prompt responses during testing. By extending the list of questions and answers, you can handle additional prompts without modifying the existing ones, resulting in more organized and maintainable test code. If a prompt hasn't been mocked, attempting to ask a question with it will raise a ValueError with an appropriate error message.
• Use Hatch v1.9.4 to as build machine requirement (#70). The Hatch package version for the build machine requirement has been updated from 1.7.0 to 1.9.4 in this change. This update streamlines the Hatch setup and version management, removing the specific installation step and listing hatch directly in the required field. The pre-setup command now only includes "hatch env create". Additionally, the acceptance tool version has been updated to ensure consistent project building and testing with the specified Hatch version. This change is implemented in the acceptance workflow file and the version of the acceptance tool used by the sandbox. This update ensures that the project can utilize the latest features and bug fixes available in Hatch 1.9.4, improving the reliability and efficiency of the build process. This change is part of the resolution of issue #70.

Contributors: @nfx, @pritishpai