Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

engine/statusmgr: Skip status updates without an associated log entry #3794

Merged
merged 4 commits into from
Apr 4, 2024

Conversation

mastercactapus
Copy link
Member

Description:
This PR introduces logic to "skip" status updates that are missing log entries.

Currently, these updates will result in all status updates failing indefinitely, or until the alert has no changes for 7 days.

The new behavior has the following properties:

  • If a log entry is missing for any reason, an error is logged, the status update is skipped, and the status subscription is updated. This means future updates will still be attempted.
  • Since only log entries from the last hour are considered, that means 1 hour is the max possible delay for a status update (currently, it is indefinite)
  • An "omit" list is kept during each engine cycle, ensuring each subscription is attempted at most once per engine cycle. If a status update fails today, it will be retried 100 times, blocking all others.

Which issue(s) this PR fixes:
Closes #3783

Additional Info:

  • As per testing this, a dev bug was fixed where CTRL+C would fail to cancel unit tests or smoke tests.

@github-actions github-actions bot added the size/s label Apr 3, 2024
@github-actions github-actions bot added size/m and removed size/s labels Apr 3, 2024
@ethan-haynes ethan-haynes merged commit 7c25e0f into master Apr 4, 2024
7 checks passed
@ethan-haynes ethan-haynes deleted the status-update-skip-nolog branch April 4, 2024 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Alert status manager gets stuck with empty log ID
3 participants