-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes AzureFileShare connection extras #16388
Fixes AzureFileShare connection extras #16388
Conversation
2b27c6d
to
63a71d1
Compare
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
The Azure File Share connection has not been creted in apache#15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes apache#16254
63a71d1
to
3cf66c9
Compare
I also changed the tests to fix the annoying behaviour of having to fix tests after every new connection (I am now just checking if the returned arrays have reasonable size > x) rather than exact content/list. That should be |
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
@potiuk thanks for fixing this error! |
No problem. I am merging it in soon and that's last thing before releasing the providers (it's been delayed due to my unavailability but I am catching up). |
@potiuk an Azure Container Volume connection was also not created in #15159. Wondering if this would cause any similar issues with AzureContainerVolumeHook as it also uses the |
Good point. Yeah I think this needs fixing as well:
It will not crash, but it will not use the extra either |
I also noticed that howtos needs updating |
Added new connection type and updated the howtos in latest fixup @sunkickr - would you mind taking a look ? |
* Fixes AzureFileShare connection extras The Azure File Share connection has not been creted in apache#15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes apache#16254 * fixup! Fixes AzureFileShare connection extras * fixup! fixup! Fixes AzureFileShare connection extras (cherry picked from commit 0c80a7d)
The Azure File Share connection has not been creted in #15159 and it
caused an unexpected side effect as the default Azure Connection
passed service_options dictionary to FileService
with key that was unexpected.
This change fixes two things:
adds AzureFileShare connection that has separate conn_type
and handles the extra_options specific for FileService Hook
available in the Airflow UI.
handles the "deprecated" way of passing keys without UI prefix
but raises a deprecation warning when such key is passed or
when the Wasb connection is used with an empty extras rather
than Azure File Share.
Fixes #16254
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.