Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory optimized branching operation #1

Merged
3 commits merged into from
Jan 26, 2011
Merged

Memory optimized branching operation #1

3 commits merged into from
Jan 26, 2011

Conversation

simoneb
Copy link

@simoneb simoneb commented Jan 2, 2011

The current branching operation caches all rows in order to supply them to each child operation.
I created a multi-threaded branching operation which does not cache the input rows and supplies one element at a time to child operations by iterating the inner collection one element by one and waiting until all children have consumed that element.
This saves memory avoiding OOME for large volumes of input rows but is more CPU intensive due to thread synchronization.

…peline by using monitor instead of several wait handles

Added check of iteration repetitions correctness on fibonacci tests
@jalchr
Copy link

jalchr commented Jul 1, 2011

I suspect this might lead to transaction dead lock for updates/inserts into a database:
I have this config:
new Multi-threadedBranching().
.Left(new InsertRecords())
.Right(new UpdateRecords())

since the process is transactional, and the two operations might run simultaneously, this will be cause one operation to timeout waiting for the second to complete (which it won't until whole process finishes)

RudyCo pushed a commit that referenced this pull request Mar 1, 2017
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants