Reuse metadata and protocol entries while retrieving the active files #19410

findinpath · 2023-10-16T10:30:21Z

Description

This is an incremental improvement in the direction set by #18916

While retrieving the metadata & protocol entries from a multi-part checkpoint file, stop scanning the checkpoint files as soon as the metadata & protocol entries are actually found.

Reuse metadata and protocol entries while retrieving the active files

The metadata & protocol entries are already read (and saved) once
when retrieving the table handle.
Reuse this information while retrieving the active files for the table.

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Delta Lake
* Avoid redundant reading of the checkpoint files. ({issue}`issuenumber`)

findepi

Good find

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

findepi · 2023-10-16T14:07:25Z

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

@@ -742,12 +769,12 @@ private record FileOperation(FileType fileType, String fileId, OperationType ope
    {
        public static FileOperation create(String path, OperationType operationType)
        {
-            Pattern dataFilePattern = Pattern.compile(".*?/(?<partition>key=[^/]*/)?(?<queryId>\\d{8}_\\d{6}_\\d{5}_\\w{5})_(?<uuid>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})");
+            Pattern dataFilePattern = Pattern.compile(".*?/(?<partition>key=[^/]*/)?((?<queryId>\\d{8}_\\d{6}_\\d{5}_\\w{5})_(?<uuid>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})|.*.parquet)");


The .*.parquet is pretty broad.
For example, checkpoint files would also match the pattern.

Maybe instead of this,
let's turn dataFilePattern into an ordinary catch all pattern \Q table_directory \E / ( key=value /)? [^/]+

Modified the pattern to .*?/(?<partition>key=[^/]*/)?[^/]+ and placed it only after matching against metadata files.

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

findepi · 2023-10-16T14:18:12Z

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

                tableSnapshot,
-                ImmutableSet.of(ADD),


remove

if (entryTypes.contains(ADD)) { metadataAndProtocol = Optional.of(getCheckpointMetadataAndProtocolEntries( session, checkpointSchemaManager, typeManager, fileSystem, stats, checkpoint));

from io.trino.plugin.deltalake.transactionlog.TableSnapshot#getCheckpointTransactionLogEntries

it's now dead code

Not sure about this:

trino/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointWriterManager.java

Lines 116 to 123 in b7d7ba8

snapshot.getCheckpointTransactionLogEntries(

session,

ImmutableSet.of(PROTOCOL, TRANSACTION, ADD, REMOVE, COMMIT),

checkpointSchemaManager,

typeManager,

fileSystem,

fileFormatDataSourceStats)

.forEach(checkpointBuilder::addLogEntry);

the linked code place already reads metadata entry.
let it read protocol as well and pass both

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java

Typically metadata files should be accessed once, so prefer `.add(x)` over `.addCopies(x, 1)`, so that `.addCopies` stand out as potentially something to address.

Once the metadata & protocol entries are found, the scanning of multi-part checkpoint files can be stopped.

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java

The `metadata` & `protocol` entries are already read (and saved) once when retrieving the table handle. Reuse this information while retrieving the active files for the table.

findinpath · 2023-10-17T08:07:35Z

@findepi pls test this PR with secrets.

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/transactionlog/TestTableSnapshot.java

findepi · 2023-10-17T11:35:17Z

/test-with-secrets sha=f308b77d8670341f48b98c4503b86baea449575c

…add files

findepi · 2023-10-18T11:56:35Z

🚀

Test accuracy for reading multi-part checkpoint files

92317a1

cla-bot bot added the cla-signed label Oct 16, 2023

findinpath added delta-lake Delta Lake connector and removed cla-signed labels Oct 16, 2023

findinpath requested review from findepi, ebyhr and pajaks October 16, 2023 11:47

cla-bot bot added the cla-signed label Oct 16, 2023

findinpath changed the title ~~Avoid useless scanning of multi-part checkpoint files~~ Reuse metadata and protocol entries while retrieving the active files Oct 16, 2023

findinpath self-assigned this Oct 16, 2023

findepi reviewed Oct 16, 2023

View reviewed changes

findinpath added 3 commits October 16, 2023 21:33

Test multi-part checkpoint table file operations

e3f8bcf

Simplify occurrences builders in TestTransactionLogAccess

e19de6b

Typically metadata files should be accessed once, so prefer `.add(x)` over `.addCopies(x, 1)`, so that `.addCopies` stand out as potentially something to address.

Avoid useless scanning of multi-part checkpoint files

3573dc9

Once the metadata & protocol entries are found, the scanning of multi-part checkpoint files can be stopped.

findinpath force-pushed the findinpath/multipart-checkpoint-files branch from 5736604 to 7c56c69 Compare October 16, 2023 19:39

findinpath requested review from findepi and homar October 17, 2023 07:01

findepi approved these changes Oct 17, 2023

View reviewed changes

...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved

Reuse metadata and protocol entries while retrieving the active files

6e2c8fe

The `metadata` & `protocol` entries are already read (and saved) once when retrieving the table handle. Reuse this information while retrieving the active files for the table.

findinpath force-pushed the findinpath/multipart-checkpoint-files branch from 7c56c69 to 6e2c8fe Compare October 17, 2023 07:54

findinpath force-pushed the findinpath/multipart-checkpoint-files branch from 0061960 to f308b77 Compare October 17, 2023 10:49

findepi reviewed Oct 17, 2023

View reviewed changes

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/transactionlog/TestTableSnapshot.java Outdated Show resolved Hide resolved

Require metadata and protocol entries to be provided when retrieving …

2151f45

…add files

findinpath force-pushed the findinpath/multipart-checkpoint-files branch from f308b77 to 2151f45 Compare October 17, 2023 11:56

findepi approved these changes Oct 17, 2023

View reviewed changes

findinpath requested a review from findepi October 18, 2023 03:51

findinpath added the release-notes label Oct 18, 2023

findepi merged commit f5b1e89 into trinodb:master Oct 18, 2023

github-actions bot added this to the 430 milestone Oct 18, 2023

mosabua mentioned this pull request Oct 20, 2023

Add Trino 430 release notes #19428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reuse metadata and protocol entries while retrieving the active files #19410

Reuse metadata and protocol entries while retrieving the active files #19410

findinpath commented Oct 16, 2023 •

edited

Loading

findepi left a comment

findepi Oct 16, 2023

findinpath Oct 16, 2023

findepi Oct 16, 2023

findinpath Oct 16, 2023

findepi Oct 17, 2023

findinpath commented Oct 17, 2023

findepi commented Oct 17, 2023

findepi commented Oct 18, 2023

	snapshot.getCheckpointTransactionLogEntries(
	session,
	ImmutableSet.of(PROTOCOL, TRANSACTION, ADD, REMOVE, COMMIT),
	checkpointSchemaManager,
	typeManager,
	fileSystem,
	fileFormatDataSourceStats)
	.forEach(checkpointBuilder::addLogEntry);

Reuse metadata and protocol entries while retrieving the active files #19410

Reuse metadata and protocol entries while retrieving the active files #19410

Conversation

findinpath commented Oct 16, 2023 • edited Loading

Description

Additional context and related issues

Release notes

findepi left a comment

Choose a reason for hiding this comment

findepi Oct 16, 2023

Choose a reason for hiding this comment

findinpath Oct 16, 2023

Choose a reason for hiding this comment

findepi Oct 16, 2023

Choose a reason for hiding this comment

findinpath Oct 16, 2023

Choose a reason for hiding this comment

findepi Oct 17, 2023

Choose a reason for hiding this comment

findinpath commented Oct 17, 2023

findepi commented Oct 17, 2023

findepi commented Oct 18, 2023

findinpath commented Oct 16, 2023 •

edited

Loading