Improve call performance for `sync_partition_metadata` utility #18384

fgwang7w · 2022-09-22T06:49:43Z

This PR fixes following issues:

Case 1: Current implementation for getting partition names takes partition values directly without taking into account of the actual path. e.g. The use of escaped character maybe encoded in HMS(Hive MetaStore) as partition name that could result incorrectness with current naming convention to fetch partition names.

Solution: Retrieve partition storage location correctly from metastore using correct API calls

Case 2: Sync partition utility lists file status of each partition twice. In case of handling a large of number of partitions for an external table, redundant check file status is a performance bottleneck.

Solution: Avoid file status check after knowing the path name from storage already by the sync partition procedure code path.

Case 3: For create partition call, batch size for number of trunks of partitions to be added to the metastore database is bounded by a fixed number. This limits the performance to update the metastore when the consumer of the underlying database is capable to handle a large number of batch and a large size of update batch.

Solution: Different metastore systems could implement this commit batch size differently based on different underlying database capacity.

== RELEASE NOTES ==

General Changes
* improve performance for procedure sync_partition_metadata 

Hive Changes
* Increase default partition batch size per commit for Hive Metastore up to 10. 
* Set default partition batch size per commit for Glue Metastore up to 10,000

Detail troubleshoot and solution design can be found in this reference page here

# of partitions	Before (min)	after(min)	delta
2,526	5	1	5
12,000	54	6	9
39,493	92	12	7

agrawalreetika

Thank you for working on this.
The Benchmark number looks promising, this would be a good improvement for the sync-partition procedure.
I have taken the first pass on PR and added my comments.
These attached Benchmark results are from glue or HMS? Should we do it for both?

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

...tastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

imjalpreet

@fgwang7w Thank you for the optimisation. I did the first pass and I have a question regarding one change added in SyncPartitionMetadataProcedure.

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

imjalpreet

I have one more suggestion, please let me know what are your views on the same.

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

agrawalreetika

The changes look good to me. Quick question, Are these attached Benchmark results from glue? Could we update the document with the same? Please update the PR release notes too about the partition batch.

fgwang7w · 2022-10-20T22:51:55Z

The changes look good to me. Quick question, Are these attached Benchmark results from glue? Could we update the document with the same? Please update the PR release notes too about the partition batch.

sure, will update the release notes. many thanks for help to review.

ethanyzhang · 2022-10-25T21:44:21Z

@tdcmeehan @prestodb/committers Hi Presto committers, this PR is reviewed internally already, can we have a second round review?

...tastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java

...-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/glue/GlueHiveMetastore.java

yingsu00

@fgwang7w George, great improvement! In addition to the comments in code, there are several other nits:

Commit title for 1 and 3 are too long
Commit title first letter shall be upper case
Can you use full sentences in the PR message?

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

...tastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java

fgwang7w · 2022-12-18T20:52:37Z

Thank you @yingsu00 for review. All comments are addressed, could you please give another pass? Thanks!
Thank you @pranjalssh @agrawalreetika for approving the PR.

fgwang7w · 2022-12-20T19:33:57Z

@prestodb/committers ping!

yingsu00

@fgwang7w Mostly good, just some nits.
In the PR message and commit messages:

without take into account

-> without taking into account

hms

-> HMS(Hive Meta Store)

Case 2: Sync partition utility lists file status of each partition twice. In case of handling a large of number of partitions for an external table, redundant check file status is a performance bottleneck.

Case 2: Sync partition utility lists file status of each partition twice, causing performance bottleneck when processing an external table with a large of number of partitions.

Commit 1 & 3 titles are too long.
Use upper case for the first letter in each sentence, including all commit titles, commit/PR messages, newly added comments.

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java

presto-hive-common/src/main/java/com/facebook/presto/hive/HdfsContext.java

...tastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java

...-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/glue/GlueHiveMetastore.java

Current implementation for getting partition names takes partition values directly without take into account of the actual path. According to the [hive source code](shorturl.at/evT28), this logic may change in the future. This commit updates the partition values with the relative path acquired from the metastore APIs.

Reference to [HMS Repair Utility](shorturl.at/eHIXY), the size for adding partitions can be in a range between 1 and 2,147,483,647. For [Glue metastore](shorturl.at/pqDH9), it’s bounded by 100 for write access.

fgwang7w marked this pull request as ready for review September 22, 2022 07:01

fgwang7w requested a review from a team as a code owner September 22, 2022 07:01

fgwang7w requested review from presto-oss, agrawalreetika and imjalpreet September 22, 2022 07:01

ethanyzhang added the backlog label Sep 23, 2022

agrawalreetika requested changes Sep 27, 2022

View reviewed changes

fgwang7w force-pushed the syncpart-oss branch 2 times, most recently from 60d9881 to 946716a Compare September 29, 2022 04:08

imjalpreet requested changes Sep 29, 2022

View reviewed changes

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java Outdated Show resolved Hide resolved

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java Outdated Show resolved Hide resolved

imjalpreet requested changes Sep 30, 2022

View reviewed changes

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java Outdated Show resolved Hide resolved

fgwang7w force-pushed the syncpart-oss branch from 946716a to b2177f9 Compare September 30, 2022 17:19

fgwang7w requested a review from imjalpreet September 30, 2022 17:24

imjalpreet requested changes Oct 3, 2022

View reviewed changes

presto-hive/src/main/java/com/facebook/presto/hive/SyncPartitionMetadataProcedure.java Outdated Show resolved Hide resolved

fgwang7w force-pushed the syncpart-oss branch from b2177f9 to de12106 Compare October 17, 2022 20:00

fgwang7w requested review from agrawalreetika and imjalpreet October 17, 2022 20:05

agrawalreetika approved these changes Oct 19, 2022

View reviewed changes

fgwang7w force-pushed the syncpart-oss branch 4 times, most recently from 7fe6a7c to ef1376a Compare November 15, 2022 06:56

fgwang7w force-pushed the syncpart-oss branch from ef1376a to 3eec66a Compare November 17, 2022 08:37

fgwang7w force-pushed the syncpart-oss branch from 3eec66a to 3fecbdc Compare December 6, 2022 01:26

rschlussel requested a review from pranjalssh December 6, 2022 15:54

pranjalssh reviewed Dec 7, 2022

View reviewed changes

fgwang7w force-pushed the syncpart-oss branch 2 times, most recently from d1d4e54 to 12d7d8c Compare December 7, 2022 21:50

fgwang7w force-pushed the syncpart-oss branch 3 times, most recently from d1c911f to 9513fd6 Compare December 14, 2022 19:00

pranjalssh approved these changes Dec 15, 2022

View reviewed changes

pranjalssh requested a review from rschlussel December 15, 2022 19:59

yingsu00 reviewed Dec 17, 2022

View reviewed changes

fgwang7w force-pushed the syncpart-oss branch 2 times, most recently from ceb14b8 to 98ec731 Compare December 18, 2022 20:50

fgwang7w changed the title ~~improve sync_partition_metadata performance~~ Improve call performance for sync_partition_metadata utility Dec 18, 2022

fgwang7w force-pushed the syncpart-oss branch from 98ec731 to 2021afc Compare December 18, 2022 20:55

fgwang7w added the in-final-review label Dec 21, 2022

fgwang7w force-pushed the syncpart-oss branch 5 times, most recently from a0caa9a to b9a5dc3 Compare December 27, 2022 19:34

yingsu00 self-requested a review January 4, 2023 01:02

yingsu00 reviewed Jan 4, 2023

View reviewed changes

fgwang7w added 3 commits January 4, 2023 14:22

Bypass file path check for sync partition

08c4ec0

Add a metastore interface to control add partition commit batch size

fa39505

Reference to [HMS Repair Utility](shorturl.at/eHIXY), the size for adding partitions can be in a range between 1 and 2,147,483,647. For [Glue metastore](shorturl.at/pqDH9), it’s bounded by 100 for write access.

fgwang7w force-pushed the syncpart-oss branch from b9a5dc3 to fa39505 Compare January 5, 2023 00:28

fgwang7w requested a review from yingsu00 January 5, 2023 02:09

yingsu00 approved these changes Jan 7, 2023

View reviewed changes

yingsu00 merged commit 4fdd16e into prestodb:master Jan 7, 2023

fgwang7w deleted the syncpart-oss branch January 9, 2023 03:19

wanglinsong mentioned this pull request Jan 12, 2023

Add release notes for 0.279 #18920

Merged

30 tasks

ethanyzhang removed the in-final-review label Apr 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve call performance for `sync_partition_metadata` utility #18384

Improve call performance for `sync_partition_metadata` utility #18384

fgwang7w commented Sep 22, 2022 •

edited

Loading

agrawalreetika left a comment

imjalpreet left a comment

imjalpreet left a comment

agrawalreetika left a comment •

edited

Loading

fgwang7w commented Oct 20, 2022

ethanyzhang commented Oct 25, 2022

yingsu00 left a comment

fgwang7w commented Dec 18, 2022

fgwang7w commented Dec 20, 2022

yingsu00 left a comment •

edited

Loading

Improve call performance for sync_partition_metadata utility #18384

Improve call performance for sync_partition_metadata utility #18384

Conversation

fgwang7w commented Sep 22, 2022 • edited Loading

agrawalreetika left a comment

Choose a reason for hiding this comment

imjalpreet left a comment

Choose a reason for hiding this comment

imjalpreet left a comment

Choose a reason for hiding this comment

agrawalreetika left a comment • edited Loading

Choose a reason for hiding this comment

fgwang7w commented Oct 20, 2022

ethanyzhang commented Oct 25, 2022

yingsu00 left a comment

Choose a reason for hiding this comment

fgwang7w commented Dec 18, 2022

fgwang7w commented Dec 20, 2022

yingsu00 left a comment • edited Loading

Choose a reason for hiding this comment

Improve call performance for `sync_partition_metadata` utility #18384

Improve call performance for `sync_partition_metadata` utility #18384

fgwang7w commented Sep 22, 2022 •

edited

Loading

agrawalreetika left a comment •

edited

Loading

yingsu00 left a comment •

edited

Loading