Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kernel] Add Snapshot::getTimestamp public API #3791

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions kernel/kernel-api/src/main/java/io/delta/kernel/Snapshot.java
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,15 @@ public interface Snapshot {
* @return a list of partition column names, or an empty list if the table is not partitioned.
*/
List<String> getPartitionColumnNames(Engine engine);

/**
* Get the timestamp (in milliseconds since the Unix epoch) of the latest commit in this snapshot.
*
* @param engine the engine to use for IO operations
* @return the timestamp of the latest commit
*/
long getTimestamp(Engine engine);

/**
* Get the schema of the table at this snapshot.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,44 @@ public SnapshotImpl(
this.inCommitTimestampOpt = Optional.empty();
}

/////////////////
// Public APIs //
/////////////////

@Override
public long getVersion(Engine engine) {
return version;
}

/**
* Get the timestamp (in milliseconds since the Unix epoch) of the latest commit in this Snapshot.
* If the table does not yet exist (i.e. this Snapshot is being used to create the new table),
* this method returns -1. Note that this -1 value will never be exposed to users - either they
* get a valid snapshot for a table or they get an exception.
*
* <p>When InCommitTimestampTableFeature is enabled, the timestamp is retrieved from the
* CommitInfo of the latest commit in this Snapshot, which can result in an IO operation.
*
* <p>For non-ICT tables, this is the same as the file modification time of the latest commit in
* this Snapshot.
*/
@Override
public long getTimestamp(Engine engine) {
if (IN_COMMIT_TIMESTAMPS_ENABLED.fromMetadata(metadata)) {
if (!inCommitTimestampOpt.isPresent()) {
Optional<CommitInfo> commitInfoOpt =
CommitInfo.getCommitInfoOpt(engine, logPath, logSegment.version);
inCommitTimestampOpt =
Optional.of(
CommitInfo.getRequiredInCommitTimestamp(
commitInfoOpt, String.valueOf(logSegment.version), dataPath));
}
return inCommitTimestampOpt.get();
} else {
return logSegment.lastCommitTimestamp;
}
}

@Override
public StructType getSchema(Engine engine) {
return getMetadata().getSchema();
Expand All @@ -79,8 +112,16 @@ public ScanBuilder getScanBuilder(Engine engine) {
return new ScanBuilderImpl(dataPath, protocol, metadata, getSchema(engine), logReplay, engine);
}

public Metadata getMetadata() {
return metadata;
///////////////////
// Internal APIs //
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems misleading. Even if this line says "Internal", 3rd party developers may call the subsequent method as they are public.

///////////////////

public Path getLogPath() {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. put all getters at the top of internal APIs, with more complicated internal APIs below
  2. re-ordered these getters to match the order of member field declarations

return logPath;
}

public Path getDataPath() {
return dataPath;
}

public Protocol getProtocol() {
Expand All @@ -102,6 +143,14 @@ public Map<String, DomainMetadata> getDomainMetadataMap() {
return logReplay.getDomainMetadataMap();
}

public Metadata getMetadata() {
return metadata;
}

public LogSegment getLogSegment() {
return logSegment;
}

public CreateCheckpointIterator getCreateCheckpointIterator(Engine engine) {
long minFileRetentionTimestampMillis =
System.currentTimeMillis() - TOMBSTONE_RETENTION.fromMetadata(metadata);
Expand All @@ -122,47 +171,6 @@ public Optional<Long> getLatestTransactionVersion(Engine engine, String applicat
return logReplay.getLatestTransactionIdentifier(engine, applicationId);
}

public LogSegment getLogSegment() {
return logSegment;
}

public Path getLogPath() {
return logPath;
}

public Path getDataPath() {
return dataPath;
}

/**
* Returns the timestamp of the latest commit of this snapshot. For an uninitialized snapshot,
* this returns -1.
*
* <p>When InCommitTimestampTableFeature is enabled, the timestamp is retrieved from the
* CommitInfo of the latest commit which can result in an IO operation.
*
* <p>For non-ICT tables, this is the same as the file modification time of the latest commit in
* the snapshot.
*
* @param engine the engine to use for IO operations
* @return the timestamp of the latest commit
*/
public long getTimestamp(Engine engine) {
if (IN_COMMIT_TIMESTAMPS_ENABLED.fromMetadata(metadata)) {
if (!inCommitTimestampOpt.isPresent()) {
Optional<CommitInfo> commitInfoOpt =
CommitInfo.getCommitInfoOpt(engine, logPath, logSegment.version);
inCommitTimestampOpt =
Optional.of(
CommitInfo.getRequiredInCommitTimestamp(
commitInfoOpt, String.valueOf(logSegment.version), dataPath));
}
return inCommitTimestampOpt.get();
} else {
return logSegment.lastCommitTimestamp;
}
}

/**
* Returns the commit coordinator client handler based on the table metadata in this snapshot.
*
Expand Down
Loading