Add Support for Iceberg table sort orders #21977

evanvdia · 2024-02-21T08:40:37Z

Description

Add Support for Iceberg connector to create sorted files. The sort order can be configured with the sorted_by table property. When creating the table, can specify an array of one or more columns to use for sorting.

Cherry-pick of trinodb/trino#14891

Motivation and Context

Issue : #21978

Test Plan

Added tests for sorted_by operations

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Iceberg Connector Changes
* Support for Iceberg table sort orders. Tables can be created to add a list of `sorted_by` columns which will be used to order files written to the table.

linux-foundation-easycla · 2024-02-21T08:40:45Z

The committers listed above are authorized under a signed CLA.

✅ login: evanvdia / name: Nidhin Varghese (d0a639b)

github-actions · 2024-02-21T08:44:28Z

Codenotify: Notifying subscribers in CODENOTIFY files for diff 8789cd9...d0a639b.

No notifications.

steveburnett

Thanks for the documentation! I had a few suggestions to help readability, and one question about the formatting of one of the two code examples.

Also please sign the Presto CLA by selecting the "Please click here to be authorized" link in the earlier comment.

presto-docs/src/main/sphinx/connector/iceberg.rst

steveburnett

Thanks for the changes! This is better. I found a nit near the beginning, and I ask if you would recheck the formatting of the first code block example.

presto-docs/src/main/sphinx/connector/iceberg.rst

steveburnett

LGTM! (docs)

Pull of updated branch, new local build of docs. Thanks for the changes!

hantangwangd

Take a first look, some problems for discussion.

presto-docs/src/main/sphinx/connector/iceberg.rst

hantangwangd · 2024-02-27T10:52:04Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergNativeMetadata.java

+                icebergTable.sortOrder().fields().stream()
+                        .map(SortField::fromIceberg)
+                        .collect(toImmutableList()));


Seems haven't support IcebergNativeMetadata. Maybe try use transaction.replaceSortOrder() to support sort_by in IcebergNativeMetadata?

We will analyse and work on this.

presto-iceberg/src/test/java/com/facebook/presto/iceberg/hive/TestIcebergDistributedHive.java

hantangwangd · 2024-05-03T11:52:36Z

Hi @evanvdia, just wanted to confirm that are you still working on this?

evanvdia · 2024-05-03T13:28:50Z

Hi @hantangwangd, yes. working on this.

tdcmeehan · 2024-06-05T14:31:44Z

@evanvdia is this ready for review? There is a merge conflict.

evanvdia · 2024-06-05T17:43:49Z

@tdcmeehan Merge conflicts are resolved. But iceberg unit test cases are failing after rebasing.

hantangwangd

By the way, please give us a notice when this PR is ready for review, thanks.

hantangwangd · 2024-06-05T18:53:52Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java

                continue;
            }

-            Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), transformedPage, position);
-            WriteContext writer = createWriter(partitionData);
+            Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), page, position);


Suggested change

Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), page, position);

Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), transformedPage, position);

After check the code, I think the problem is introduced by this change.

Thanks @hantangwangd . that was the issue. done the changes. Will let you know once PR is ready for review.

evanvdia · 2025-02-05T13:44:24Z

@tdcmeehan All changes are done.

tdcmeehan · 2025-02-05T16:10:57Z

presto-docs/src/main/sphinx/connector/iceberg.rst

+        sorted_by = ARRAY['join_date']
+    )
+
+Iceberg Connector does not support sort order transforms.The following sort order transformations are not supported:


A warning is added when these are used, right? Can you add that information to this documentation?

@tdcmeehan Done

evanvdia · 2025-02-06T13:09:41Z

@tdcmeehan Updated the doc. Could you please review.

tdcmeehan · 2025-02-06T15:12:47Z

@steveburnett can you check the docs one last time?

steveburnett

Thanks for the documentation! A few minor suggestions of phrasing for consistency, nothing major.

presto-docs/src/main/sphinx/connector/iceberg.rst

evanvdia · 2025-02-07T04:10:07Z

@steveburnett I have updated the changes. Can you please look into this.

steveburnett

LGTM! (docs)

Pull updated branch, new local doc build, looks good. Thank you!

No attribution

tdcmeehan

Please properly attribute this commit in the commit message. See our contributing guide for instructions.

Cherry-pick of trinodb/trino#14891 Co-authored-by: Alexander Jo <jo.alex2144@gmail.com>

evanvdia requested a review from a team as a code owner February 21, 2024 08:40

evanvdia requested a review from presto-oss February 21, 2024 08:40

evanvdia mentioned this pull request Feb 21, 2024

sorted_by table property not supported in Iceberg connector #21978

Open

steveburnett requested changes Feb 21, 2024

View reviewed changes

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch 2 times, most recently from d7a8899 to b65eae5 Compare February 22, 2024 11:16

steveburnett requested changes Feb 22, 2024

View reviewed changes

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/iceberg.rst Show resolved Hide resolved

tdcmeehan added the iceberg Apache Iceberg related label Feb 22, 2024

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 8c6b521 to 57e5e2a Compare February 23, 2024 14:15

steveburnett previously approved these changes Feb 23, 2024

View reviewed changes

hantangwangd self-requested a review February 24, 2024 01:16

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 57e5e2a to 07ed82a Compare February 27, 2024 10:11

hantangwangd reviewed Feb 27, 2024

View reviewed changes

evanvdia dismissed steveburnett’s stale review via 30ac341 March 12, 2024 04:45

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 07ed82a to 30ac341 Compare March 12, 2024 04:45

tdcmeehan requested a review from hantangwangd May 2, 2024 20:30

tdcmeehan self-assigned this May 2, 2024

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch 2 times, most recently from 7b3bd0f to 3b6903b Compare May 23, 2024 11:43

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 3b6903b to ef6df97 Compare June 5, 2024 16:23

hantangwangd reviewed Jun 5, 2024

View reviewed changes

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from ef6df97 to 32840e1 Compare June 11, 2024 15:29

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 32840e1 to 5e60f75 Compare July 12, 2024 11:45

evanvdia requested a review from elharo as a code owner July 12, 2024 11:45

evanvdia dismissed stale reviews from hantangwangd, agrawalreetika, and ZacBlanco via 0089431 February 5, 2025 06:11

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch 3 times, most recently from f132df0 to 59aa7b6 Compare February 5, 2025 11:39

evanvdia requested review from tdcmeehan, agrawalreetika, hantangwangd and ZacBlanco February 5, 2025 13:38

tdcmeehan reviewed Feb 5, 2025

View reviewed changes

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 59aa7b6 to e3c8cc4 Compare February 5, 2025 17:01

evanvdia requested a review from tdcmeehan February 6, 2025 03:38

steveburnett requested changes Feb 6, 2025

View reviewed changes

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from e3c8cc4 to 938c2e0 Compare February 7, 2025 03:55

evanvdia requested a review from steveburnett February 7, 2025 04:40

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 938c2e0 to 999f8e2 Compare February 7, 2025 05:03

steveburnett approved these changes Feb 7, 2025

View reviewed changes

tdcmeehan previously approved these changes Feb 7, 2025

View reviewed changes

tdcmeehan requested changes Feb 7, 2025

View reviewed changes

Add Support for Iceberg table sort orders

d0a639b

Cherry-pick of trinodb/trino#14891 Co-authored-by: Alexander Jo <jo.alex2144@gmail.com>

evanvdia force-pushed the iceberg_sorted_by_tbl_ppty branch from 999f8e2 to d0a639b Compare February 7, 2025 17:39

tdcmeehan approved these changes Feb 7, 2025

View reviewed changes

ZacBlanco approved these changes Feb 7, 2025

View reviewed changes

ZacBlanco merged commit 003d86a into prestodb:master Feb 7, 2025
55 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support for Iceberg table sort orders #21977

Add Support for Iceberg table sort orders #21977

evanvdia commented Feb 21, 2024 •

edited by ZacBlanco

Loading

linux-foundation-easycla bot commented Feb 21, 2024 •

edited

Loading

github-actions bot commented Feb 21, 2024 •

edited

Loading

steveburnett left a comment •

edited

Loading

steveburnett left a comment

steveburnett left a comment

hantangwangd left a comment

hantangwangd Feb 27, 2024

evanvdia Mar 1, 2024

evanvdia Jul 12, 2024

hantangwangd commented May 3, 2024

evanvdia commented May 3, 2024 •

edited

Loading

tdcmeehan commented Jun 5, 2024

evanvdia commented Jun 5, 2024

hantangwangd left a comment

hantangwangd Jun 5, 2024

evanvdia Jun 12, 2024

evanvdia commented Feb 5, 2025

tdcmeehan Feb 5, 2025

evanvdia Feb 6, 2025

evanvdia commented Feb 6, 2025

tdcmeehan commented Feb 6, 2025

steveburnett left a comment

evanvdia commented Feb 7, 2025

steveburnett left a comment

tdcmeehan left a comment

	Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), page, position);
	Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), transformedPage, position);

Add Support for Iceberg table sort orders #21977

Add Support for Iceberg table sort orders #21977

Conversation

evanvdia commented Feb 21, 2024 • edited by ZacBlanco Loading

Description

Motivation and Context

Test Plan

Contributor checklist

Release Notes

linux-foundation-easycla bot commented Feb 21, 2024 • edited Loading

github-actions bot commented Feb 21, 2024 • edited Loading

steveburnett left a comment • edited Loading

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

hantangwangd left a comment

Choose a reason for hiding this comment

hantangwangd Feb 27, 2024

Choose a reason for hiding this comment

evanvdia Mar 1, 2024

Choose a reason for hiding this comment

evanvdia Jul 12, 2024

Choose a reason for hiding this comment

hantangwangd commented May 3, 2024

evanvdia commented May 3, 2024 • edited Loading

tdcmeehan commented Jun 5, 2024

evanvdia commented Jun 5, 2024

hantangwangd left a comment

Choose a reason for hiding this comment

hantangwangd Jun 5, 2024

Choose a reason for hiding this comment

evanvdia Jun 12, 2024

Choose a reason for hiding this comment

evanvdia commented Feb 5, 2025

tdcmeehan Feb 5, 2025

Choose a reason for hiding this comment

evanvdia Feb 6, 2025

Choose a reason for hiding this comment

evanvdia commented Feb 6, 2025

tdcmeehan commented Feb 6, 2025

steveburnett left a comment

Choose a reason for hiding this comment

evanvdia commented Feb 7, 2025

steveburnett left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

evanvdia commented Feb 21, 2024 •

edited by ZacBlanco

Loading

linux-foundation-easycla bot commented Feb 21, 2024 •

edited

Loading

github-actions bot commented Feb 21, 2024 •

edited

Loading

steveburnett left a comment •

edited

Loading

evanvdia commented May 3, 2024 •

edited

Loading