-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Work towards Batcher
unification
#553
Work towards Batcher
unification
#553
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can land this change, but there's one comment around pre-sizing chunks. At the moment, we can use a different capacity than the one from Timely, with this change we'd fall back to Timely's ensure_capacity
, which may or may not be what we want here. I think this can be mitigated by using a newtype pattern.
1ad5485
to
a609da1
Compare
a609da1
to
291c129
Compare
Co-authored-by: Petros Angelatos <petrosagg@gmail.com>
Performance-wise, this PR seems to maintain parity with
vs on
In discussion with @antiguru we've concluded that it should be ok despite a few quirks that we think we can dial in, around buffer sizing and other nits. We'll make sure that we assess the performance on MZ, and "fail forward" from this work. |
The
MergeBatcher
implementation is shared across multiple implementations, that re-do fairly similar work. This PR attempts to extract their differences into compact support traits they can implement, largely about how to read from and write into containers, while in the process of merging.The PR is meant to be largely behavior preserving, with some light glitches here and there around the elegance with which one can manipulate the general objects. I'm 100% up for discussing those as part of review, and either shipping or prioritizing any fixes for them. Examples include: 1. dropping some containers in the act of merging (the final containers in a chain), 2. calling
unwrap()
far too often, rather than destructuring.No hurry to merge, but did want to put this in place so we can start looking at it.
Although the PR reads as fairly LOC positive, several hundred are a new
columnar.rs
example meant to exercise the non-privileged access to these traits and types. Also, the PR does not yetgit rm
the columnation merger which is now dead code, nor is it yet able to get the flat container merger to build (???) so it cannot be removed.