-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wasb connection to Azure Fileshareservice Hook not working #16254
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
When we try to enter key-value pairs in JSON format for file service in the wasb connection those key-value pairs are not being stored and get replaced with {"extra__wasb__connection_string": "", "extra__wasb__sas_token": "", "extra__wasb__shared_access_key": "", "extra__wasb__tenant_id": ""} |
When using the UI, you should not configure extras but you can now place the values in the right fields - instead of manually adding "extras" via JSON dict: You have those fields (as you can see in your screenshot):
Which should be used for that purpose. This has been fixed in The Microsoft Azure Provider 2.0.0 (as a breaking change): https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/index.html#id1 The detailed commit: |
Closing for now unless it does not solve your problem (please let us know if it did/did not). |
OK. I see the problem @cvsekhar . It seems that the AzureFileServerHook was wrongly defined and the way extras were used there was wrong. Passing extras this way is not how airlfow works with connection extras - all extras are always prefixed with The AzureFileServerHook has been missing in the change and it does not rally work the same way - on one hand you have no dedicated hook, and on the other hand - the other "Azure" related hooks have some extras defined which are not properly configured. There are few ways you can mitigate the problem:
Let me know if those work. In the meantime I might implement a quick fix - I am just about to release a new version of all providers but I think I can quickly fix this one in that upcoming version. I will ask you to test the RC version then OK ? |
@potiuk, thank you Sure will test the RC |
The Azure File Share connection has not been creted in apache#15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes apache#16254
* Fixes AzureFileShare connection extras The Azure File Share connection has not been creted in #15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes #16254 * fixup! Fixes AzureFileShare connection extras * fixup! fixup! Fixes AzureFileShare connection extras
* Fixes AzureFileShare connection extras The Azure File Share connection has not been creted in apache#15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes apache#16254 * fixup! Fixes AzureFileShare connection extras * fixup! fixup! Fixes AzureFileShare connection extras (cherry picked from commit 0c80a7d)
Apache Airflow version:
Version: v2.1.0
Git Version: .release:2.1.0+304e174674ff6921cb7ed79c0158949b50eff8fe
What happened:
INFO - init() got an unexpected keyword argument 'extra__wasb__connection_string'
What you expected to happen:
We have used the extra field and populated the SAS token according to the documents on how to use wasb connection for fileshare and we should be able to login in using the SAS token. This has been working in 2.0.0
The text was updated successfully, but these errors were encountered: