Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorganize Container traits #605

Merged
merged 5 commits into from
Nov 30, 2024

Conversation

frankmcsherry
Copy link
Member

@frankmcsherry frankmcsherry commented Nov 29, 2024

The SizableContainer trait spoke about capacities for containers, meant to allow them to express their capacity, a preference for a capacity, and how to acquire more capacity. It seems like this logic was used to implement some right-sizing of containers, but always in a way that the container could have chosen for itself anyhow.

With that in mind, this PR tries out a new trait that instead looks like

/// A container that can be sized and reveals its capacity.
pub trait SizableContainer: Container {
    /// Indicates that the container is "full" and should be shipped.
    fn at_capacity(&self) -> bool;
    /// Restores `self` to its desired capacity, if it has one.
    ///
    /// The `stash` argument is available, and may have the intended capacity.
    /// However, it may be non-empty, and may be of the wrong capacity. The
    /// method should guard against these cases.
    fn ensure_capacity(&mut self, stash: &mut Option<Self>);
}

Here the container can indicate if it is "at capacity" or in Vec terms if its length equals its capacity, and also it can restore its capacity. Each implementation had seemingly straight-forward implementations, and the place the logic was used seem to work great with these functions.


The commits continue with one that removes PushPartitioned and allows Exchange to be parameterized by CB: ContainerBuilder, rather than by containers.


One VERY IMPORTANT thing to sort out: if folks are allowed to create Exchange edges with their own container builders, they must preserve the counts correctly, otherwise progress tracking breaks. This might be a moment to revisit that, and see if there is a reasonable decoupling of .len() from "number of items in the container". At least, there is a very specific thing that needs to be maintained (count of records accepted), and perhaps it shouldn't be up to the container builder to maintain this?

@frankmcsherry frankmcsherry changed the title Reorganize SizableContainer Reorganize Container traits Nov 30, 2024
@frankmcsherry frankmcsherry merged commit f201abd into TimelyDataflow:master Nov 30, 2024
7 checks passed
@github-actions github-actions bot mentioned this pull request Nov 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant