Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: allow merger to read from multiple one-block locations #140

Closed
matthewdarwin opened this issue Sep 2, 2020 · 2 comments
Closed

Comments

@matthewdarwin
Copy link

It appears (but is not proven), than in our environment when the s3 storage server is under extreme load, that one-block files go missing. The working theory is that having multiple deep-mind processes writing to the same file to the same location causes some race condition and the file just disappears.

Possible workaround: have each of the mindreaders write to its own location and then have the merger read from all those locations.

@matthewdarwin matthewdarwin changed the title Feature: allow merger to read from multiple locations Feature: allow merger to read from multiple one-block locations Sep 2, 2020
@sduchesneau
Copy link
Contributor

work is on the way: streamingfast/merger#3

doing some tests, awaiting for for PR review, then it should be good to go!

@sduchesneau
Copy link
Contributor

This branch https://github.com/dfuse-io/dfuse-eosio/tree/feature/dedupe-suffixed-oneblockfiles contains the feature.
Works locally.
Mindreader needs new flag --mindreader-oneblock-suffix with its specific suffix, different from the other mindreaders.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants