Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adjust reprocess to handle >10k better for automated systems #135

Open
willkg opened this issue Jun 24, 2024 · 1 comment
Open

adjust reprocess to handle >10k better for automated systems #135

willkg opened this issue Jun 24, 2024 · 1 comment

Comments

@willkg
Copy link
Collaborator

willkg commented Jun 24, 2024

Currently, if the number of crash ids to reprocess is > 10k, the reprocess tool balks with a message like:

Reprocessing 29183 crashes sleeping 5 seconds between groups...
You are trying to reprocess more than 10,000 crash reports at once.
Please let us know on #crashreporting on Matrix before you do this.

That's not wildly helpful and causes automated scripts to fail. For the symbol scraper scripts which find missing symbols and then reprocess crash reports, this case comes up semi-often.

This issue covers figuring out a different behavior where we balance these things:

  1. automated systems can generally work without human intervention
  2. reprocessing requests are spread out over enough time that the processor cluster scales up and it doesn't trigger an alert

Possible ideas:

  1. If more than 10k crash ids are specified and --allow-many isn't passed, reprocess truncates the list of crash ids to 10k and only reprocesses those.
  2. If more than 10k crash ids are specified, reprocess enforces a large sleep interval between reprocess API requests or pauses X seconds every 1,000 crash ids or something like that.
@willkg
Copy link
Collaborator Author

willkg commented Jun 24, 2024

I talked with Relud about this and we're going to hold off on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant