[Kernel] Integrate TableFeature framework in read and write path
#4159
Conversation
TableFeature framework in read and write path
kernel/kernel-api/src/main/java/io/delta/kernel/internal/TransactionBuilderImpl.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/TransactionBuilderImpl.java
Outdated
Show resolved
Hide resolved
| } | ||
| } | ||
|
|
||
| // Important note: uses the default implementation of `equals` and `hashCode` methods. |
There was a problem hiding this comment.
Makes sense, but would it make sense to move this comment to the class javadoc?
| private static class CheckpointV2TableFeature extends TableFeature.ReaderWriterFeature | ||
| implements FeatureAutoEnabledByMetadata { | ||
| CheckpointV2TableFeature() { | ||
| super("v2Checkpoint", /* minReaderVersion = */ 3, /* minWriterVersion = */ 7); |
There was a problem hiding this comment.
Nit: would it make sense to use the constants like TABLE_FEATURES_MIN_READER_VERSION etc?
There was a problem hiding this comment.
hmm. I think keeping constants is fine here, given some of the legacy protocols (e.g. invariants ) have no constants.
| * | ||
| * @param <V> | ||
| */ | ||
| public class CaseInsensitiveMap<V> implements Map<String, V> { |
There was a problem hiding this comment.
Would it make sense to call it something like CaseInsensitiveStringMap since the key is always a string?
There was a problem hiding this comment.
Sounds good. Will followup in a separate PR. I am also wondering if this is really needed.
| * | ||
| * @param <V> | ||
| */ | ||
| public class CaseInsensitiveMap<V> implements Map<String, V> { |
There was a problem hiding this comment.
Would it make sense to implement equals and hashCode?
There was a problem hiding this comment.
Same comment as above.
kernel/kernel-api/src/main/java/io/delta/kernel/internal/DeltaErrors.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/DeltaErrors.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/TransactionBuilderImpl.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/tablefeatures/TableFeatures.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/tablefeatures/TableFeatures.java
Outdated
Show resolved
Hide resolved
| .withFeatures(extractAutomaticallyEnabledFeatures(metadata, current)) | ||
| .normalized(); | ||
|
|
||
| if (!required.canUpgradeTo(current)) { |
There was a problem hiding this comment.
wouldn't we want to see if the currProtocol can be upgraded to the requiredProtocol ?
kernel/kernel-api/src/main/java/io/delta/kernel/internal/tablefeatures/TableFeatures.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/tablefeatures/TableFeatures.java
Outdated
Show resolved
Hide resolved
4e60a26 to
745c4a4
Compare
745c4a4 to
63d4376
Compare
| (1, 2), | ||
| (1, 3), | ||
| (1, 4), | ||
| (2, 5), |
There was a problem hiding this comment.
Don't we want to test the full cross product here?
e.g. all possible reader versions X all possible writer versions?
e.g. writer version 5 and reader version 1? Is that possible?
Description
This integrates the end-to-end TableFeatures framework in read and write path.
It also adds support for appending into a table with
deletionVectors,v2Checkpointand 'timestampNtz` are supported.How was this patch tested?
Existing and new tests.