Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer envelopes after resolving broken project states #1787

Closed
jan-auer opened this issue Jan 27, 2023 · 1 comment · Fixed by #1856
Closed

Buffer envelopes after resolving broken project states #1787

jan-auer opened this issue Jan 27, 2023 · 1 comment · Fixed by #1856
Assignees
Labels
bug Something isn't working filler Requires little effort to resolve. Ready to be picked up anytime.
Milestone

Comments

@jan-auer
Copy link
Member

jan-auer commented Jan 27, 2023

If the upstream responds with broken states, Relay attempts to reuse old caches as long as possible. However, if there is no cache, Relay immediately drops envelopes. Our stability SLO suggests that we should buffer envelopes instead until they timeout.

To achieve this behavior, we will have to:

  • Leave envelopes in the queue when states are invalid
  • Expire queued envelopes when a timeout has been reached
  • Check whether there should be a timeout on the envelopes or rather on the project state

Created from #1767 (comment)

@jan-auer jan-auer added the bug Something isn't working label Jan 27, 2023
@jan-auer
Copy link
Member Author

jan-auer commented Jan 27, 2023

One an invalid state has been resolved, it's passed back into the project cache here, which triggers a flush of all queues:

// Flush all queued `ValidateEnvelope` messages
while let Some((envelope, context)) = self.pending_validations.pop_front() {
self.flush_validation(envelope, context, state.clone());
}
// Flush all queued `AddSamplingState` messages
while let Some(message) = self.pending_sampling.pop_front() {
self.flush_sampling(message);
}

This internally invokes check_envelope, which eventually rejects the envelope:

// if we recorded an invalid project state response from the upstream (i.e. parsing
// failed), discard the event with a state reason.
if self.invalid() {
return Err(DiscardReason::ProjectState);
}

@jjbayer jjbayer added this to the Maintenance milestone Jan 30, 2023
@jan-auer jan-auer added the filler Requires little effort to resolve. Ready to be picked up anytime. label Feb 3, 2023
@olksdr olksdr self-assigned this Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working filler Requires little effort to resolve. Ready to be picked up anytime.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants