UDFs for Mapping Feed Ranges to Buckets #43092
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
API change check API changes are not detected in this pull request. |
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
...-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/udf/GetFeedRangesForBuckets.scala
Outdated
Show resolved
Hide resolved
...cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/udf/GetBucketForPartitionKey.scala
Outdated
Show resolved
Hide resolved
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
tvaron3
commented
Dec 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Joins with Databricks tables could be inefficient due to the lack of range partitioning. Because there is no range partitioning, customers that want to partition their Databricks tables on feed ranges cannot do it.
Description
Two UDFs were added:
GetFeedRangesForContainerandGetOverlappingFeedRange.GetFeedRangesForContainergives the number of feeds ranges a user needs split from the full range of a container. If no number is specified, it will return the feed ranges corresponding to the physical partitions.GetOverlappingFeedRangegives the feed range that overlaps with the feed range a partition key maps to. A feed range as a string could now be used to partition a Databricks table and using the feed ranges from theGetFeedRangesForContainerUDF.Testing
There already existed an API in the java SDK to split a feed range into multiple sub feed ranges. It explicitly checked for if a partition key was a hierarchical partition key to prevent splitting, but I removed this check. I added tests for splitting feed ranges for hierarchical partition keys to ensure this check could be removed. For the UDFs, I added tests with both normal partition keys and hierarchical partition keys. For these tests, it compares the results of a query filtered by a feed range to the documents with the bucket corresponding to that feed range.
Suggestions
Better names for the udfs ?
Edit: finalized with
GetFeedRangesForContainerandGetOverlappingFeedRange