Document retry configuration for rate throttling on log group creation

[sam-axiadids]

We are getting a lot of Rate limit exceeded when logging to new log groups that are created dynamically, Would be good to have a sleep/retry on "Rate limit exceeded" exceptions on log group creation - probably should be on:

def _idempotent_create(_callable, *args, **kwargs):
try:
_callable(*args, **kwargs)
except ClientError as e:
if e.response.get("Error", {}).get("Code") != "ResourceAlreadyExistsException":
raise

[kislyuk]

Boto3 already performs retries on these types of errors. The fact that you're running into these errors means that either you're overloading the API and need to redesign your app to avoid the overloading, or tune the Boto3 retry logic for your needs. You can tune the retry logic by passing a custom Boto3 CloudWatch Logs client to Watchtower. See boto/botocore#1260 for details of how to configure the boto3 retry logic.

[sam-axiadids]

Thanks ! will look into it.

[renlon]

It seems that the Class -- "watchtower.CloudWatchLogHandler" only takes boto3_session object. You methioned that "You can tune the retry logic by passing a custom Boto3 CloudWatch Logs client to Watchtower". is there an example of how to pass Boto3 CloudWatch Logs client to Watchtower?

[covatic-john]

@renlon did you work out how to do this?

[kislyuk]

Reopening to track documenting this.