Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry with backoff when rate-limited #35

Merged
merged 6 commits into from
Nov 7, 2018
+1,094 −0
Merged

Retry with backoff when rate-limited #35

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
da4a536
Add backoff dependency
mars Oct 23, 2018
176c52c
WIP RoundTripWithRetryBackoff transport
mars Oct 23, 2018
3382225
RoundTripWithRetryBackoff implements RoundTripper interface
mars Oct 24, 2018
d782d8b
Tune retry-with-backoff timing for Heroku API rate limiting
mars Oct 24, 2018
8db66c1
Log notices when retry-with-backoff is actaully used
mars Oct 24, 2018
4a8c894
Configurable timing for RoundTripWithRetryBackoff
mars Oct 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions v3/round_trip_with_retry_backoff.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package heroku

import (
"fmt"
"net/http"

"github.com/cenkalti/backoff"
)

func RoundTripWithRetryBackoff(req *http.Request) (*http.Response, error) {
var lastResponse *http.Response
var lastError error

retryableRoundTrip := func() error {
lastResponse, lastError = http.DefaultTransport.RoundTrip(req)
Copy link

@sclasen sclasen Oct 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably need to check lastError and return it if not nil?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lastError is checked just a few lines down in the parent function, because this backoff.Retry function should only return an error when it should be retried (status 429 for this rate-limiting use-case).

Copy link
Contributor

@talbright talbright Oct 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could dry it up by dropping lastError and doing something like this:

retryableRoundTrip := func() (err error) {
		lastResponse, err = http.DefaultTransport.RoundTrip(req)
		// Detect Heroku API rate limiting
		// https://devcenter.heroku.com/articles/platform-api-reference#client-error-responses
		if lastResponse.StatusCode == 429 {
			err = fmt.Errorf("Heroku API rate limited: 429 Too Many Requests")
		}
		return err
	}

That would also mean that you would only need to handle the error once below...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we only want this to retry for HTTP status 429 responses.

If that function returns other errors, it will retry for them too.

We only want to retry when rate-limited, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah 😊

// Detect Heroku API rate limiting
// https://devcenter.heroku.com/articles/platform-api-reference#client-error-responses
if lastResponse.StatusCode == 429 {
return fmt.Errorf("Heroku API rate limited: 429 Too Many Requests")
}
return nil
}

err := backoff.Retry(retryableRoundTrip, backoff.NewExponentialBackOff())
// Propagate the rate limit error when retries eventually fail.
if err != nil {
if lastResponse != nil {
lastResponse.Body.Close()
}
return nil, err
}
// Propagate all other response errors.
if lastError != nil {
if lastResponse != nil {
lastResponse.Body.Close()
}
return nil, lastError
}

return lastResponse, nil
}
20 changes: 20 additions & 0 deletions vendor/github.com/cenkalti/backoff/LICENSE
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 30 additions & 0 deletions vendor/github.com/cenkalti/backoff/README.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

66 changes: 66 additions & 0 deletions vendor/github.com/cenkalti/backoff/backoff.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions vendor/github.com/cenkalti/backoff/backoff_test.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

63 changes: 63 additions & 0 deletions vendor/github.com/cenkalti/backoff/context.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 25 additions & 0 deletions vendor/github.com/cenkalti/backoff/context_test.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

72 changes: 72 additions & 0 deletions vendor/github.com/cenkalti/backoff/example_test.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading