feat(python): add capability to read unity catalog (uc://) uris #3113
Conversation
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
8fcd7f4 to
0824abe
Compare
| /// Allow http url (e.g. http://localhost:8080/api/2.1/...) | ||
| /// Supported keys: | ||
| /// - `unity_allow_http_url` | ||
| AllowHttpUrl, |
There was a problem hiding this comment.
This allows users to work with a local (non-https) Unity Catalog REST API with delta-rs.
python/src/lib.rs
Outdated
| let (table_path, uc_token) = if UnityCatalogBuilder::is_unity_catalog_uri(table_uri) { | ||
| match rt().block_on(UnityCatalogBuilder::get_uc_location_and_token(table_uri)) { | ||
| Ok(tup) => tup, | ||
| Err(err) => return Err(PyRuntimeError::new_err(err.to_string())), | ||
| } | ||
| } else { | ||
| (table_uri.to_string(), "".to_string()) | ||
| }; | ||
|
|
||
| let mut options = storage_options.clone().unwrap_or_default(); | ||
| if !uc_token.is_empty() { | ||
| options.insert("UNITY_CATALOG_TEMPORARY_TOKEN".to_string(), uc_token); |
There was a problem hiding this comment.
I think this would be a better fit for code inside an object store factory
There was a problem hiding this comment.
Als you might have to wait before @hntd187's code gets merged (#3078)
Yep will have to wait for #3078 to get merged before this one, the temp credentials part is in there :)
I think this would be a better fit for code inside an object store factory
Yes, this code is currently duplicated in new() and is_deltatable(). Shall I move it to a private function? Or object store factory works too, let me know what you have in mind about that.
There was a problem hiding this comment.
Done, I've moved UC specific code to UnityCatalogFactory and added an internal helper method to RawDeltaTable for the rest of the boilerplate. Let me know if this looks fine.
0da4600 to
faf9ba9
Compare
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the `DeltaTable` instance so polars or similar readers can use it. Signed-off-by: Omkar P <45419097+omkar-foss@users.noreply.github.com>
faf9ba9 to
1a810d9
Compare
|
@ion-elgreco I've updated this PR to include the temp credentials functionality from PR #3078. Cheers. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3113 +/- ##
==========================================
- Coverage 72.21% 72.15% -0.06%
==========================================
Files 138 138
Lines 45252 45321 +69
Branches 45252 45321 +69
==========================================
+ Hits 32678 32702 +24
- Misses 10504 10551 +47
+ Partials 2070 2068 -2 ☔ View full report in Codecov by Sentry. |
| if let Some(storage_options) = storage_options { | ||
| builder = builder.with_storage_options(storage_options) | ||
| } | ||
| let (mut builder, table_path, options) = |
There was a problem hiding this comment.
I was thinking more in the lines of this, then you don't have to modify any of the python/lib code
delta-rs/crates/azure/src/lib.rs
Lines 83 to 86 in 9dc8f32
Description
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the
DeltaTableinstance so polars or similar readers can use it.Related Issue(s)
Documentation
N/A