Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-allocate vectors #370

Merged
merged 4 commits into from
Mar 10, 2021
Merged

Conversation

Kixiron
Copy link
Contributor

@Kixiron Kixiron commented Mar 8, 2021

Pre-allocates more vectors in the hopes of reducing startup times and memory usage

@frankmcsherry
Copy link
Member

Thanks! Generally looks good, in particular the ref and ref mut removal, and the with_capacity calls which much more clearly indicate intent. There is some cargo fmt line noise that it would be great to ditch (the code is formatted for a wide and short display, and cargo fmt's opinion that I shouldn't have that display is ...).

@Kixiron
Copy link
Contributor Author

Kixiron commented Mar 10, 2021

I've been looking into how to speed up dataflow construction (kinda in the vein of #308) and one of the things that I've noticed is that there's just a lot of allocations, but I'm not entirely sure how to go about removing/reducing most of them. Preallocation is a nice low hanging fruit, so I figured I'd try it out and see what happens

@frankmcsherry
Copy link
Member

I'm up for looking in to what is the source of the slowness. I've profiled it for some microbenchmarks, and .. it was smeared out a bit. There are several places where allocation could be more lazy (e.g. data buffers between workers) and where fewer allocations could be used (e.g. in reachability). But, if you end up with some traces that incriminate things, I'm up for talking through what is needed and what is not.

@Kixiron
Copy link
Contributor Author

Kixiron commented Mar 10, 2021

Sure thing, I'll look into it these next few weeks and see if I can't uncover anything!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants