Optionally ignore tombstones for read-only use cases

[dispanser]

Description

In a discussion for PR #420, #420 (comment) an idea came up to have a config option to ignore all tombstone entries entirely. Applications that are only reading do not actually use tombstones.

Use Case

This would reduce memory footprint, in particular for read-only applications.

@houqp writes:

Letting read-only application specifying whether tombstones should be kept at all sounds like a neat trick to reduce memory usage :) I think it's better handled as a separate self-contained PR though because it's an optimization specific to a subset of readers, while the problem we are dealing where applies to both readers and write

I'd like to explore that idea further. We actually have a delta table that is mostly tombstones, which would see a major decrease in memory consumption.

[houqp]

Extending on @dispanser 's idea, I think we could even extend it to all writers that doesn't handle checkpoints. By moving checkpoints to a dedicated writer/lambda function, we could even apply this optimization to all data writers.

[mosyp]

An easy way would be just an additional option/flag. And if set, then we'd keep tomstones list empty, destipe it's writer or reader

[dispanser]

I agree with @mosyp , let the user decide on instantiating the delta table.

However, it would be interesting if we're able to detect the misuse, i.e. panic / warn (?) when a writer attempts to write a checkpoint despite not having tombstones available. Not sure if that's easy to implement.

[mosyp]

@dispanser If we stick with an optional flag, then if it's set, then prevent of creating a checkpoint and telling user a meaningful error

[dispanser]

I'd make a quick foray into that topic if someone assigns this to me.

[houqp]

yep, it should be easy to prevent side effect of this optimization by checking for the flag in both checkpoint and vacuum code path.

[dispanser]

Now that I'm able to load my large tombstone-filled delta table, I plan on revisiting this issue. Still considering myself new to rust, I'm wondering how this feature could be represented in the public API. I assume that pub async fn open_table(table_uri: &str) and related functions form the public interface. If this was scala, I'd just tuck on an additional method parameter ignoreTombstones: Boolen = false, which would be backwards compatible but expose the new feature to everyone who wants to use it. What's the best practice in rust?

[houqp]

How about adding a new config value to https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_config.rs? Then we can pass in the full table config as an optional argument. We will have to either introduce breaking API change or add new functions. I am fine with us introducing breaking API change at the current stage of the project.

Another pattern I have seen from other projects is builder struct. So the code will look something like this:

let table = TableOpener::new(uri).with_config(config).with_version(1).open().await?;

Then if we need to add a new knob in the future, e.g. storage backend, we could add a new method to the TableOpener struct without breaking the existing users:

let table = TableOpener::new(uri).with_config(config).with_storage(storage).open().await?;

I am cool with either way, but I think the builder pattern is likely going to be the long term solution we would adopt.

[dispanser]

Thanks, @houqp ! I'll be looking into the builder pattern. As an added bonus, we can keep the top-level entry points for backwards-compatibility, by having them use that config mechanism.

[houqp]

https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/logical_plan/builder.rs is a good example of how the builder pattern works in real world.