Exceptions coming from boto3/botocore when running boto3.client('sts') too many times simultaneously #1592
Comments
By locking around multiple operations on self._deferred Fixes boto/boto3#1592
|
Workaround for this in the mean time: Put a |
|
You're correct about this behavior. Historically we've said that session methods aren't thread safe, but once you've created a client or resource, we guarantee that those calls are thread safe (http://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing). While we can investigate changing that stance, I suspect there will be more work than just the component logic, though that's a good start and one of the most common content points in sessions. |
|
So it looks like another way around this is boto3.Session().client (or boto3.session.Session() as in that linked page) instead of just boto3.client. |
|
So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit |
|
There is also an outstanding unanswered question in botocore about thread safety. Would be great to know what's the actual correct approach should be. |
|
I'm having the same problem. I'm calling boto3.client('sts') for each thread created and i hit KeyError: 'endpoint_resolver'. |
@zgoda-mobica , Same issue here. I had So now using following in thread invoked method.... and it seems to be ok. |
This solves it, but makes the overall threaded push a bit slow. |
It seems that the boto3 library is not threadsafe. The solution discussed in GitHub issues such as boto/botocore#1246 boto/boto3#1592 and the documentation at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing suggest a very simple change that sems to make things work.
|
The documentation says that it is "recommended" to create a resource per thread but the example code below that recommendation creates a resource AND a session per thread. IOW, it is ambiguous if sessions can be shared between threads or not. Further down, the documentation states that resources are NOT thread-safe. That would imply that it's not merely "recommended" to create a resource per thread, but rather that it's a requirement to do so. I'm confused. |
I second this. There doesn't seem to be a generic best practice for this case. |
The 'boto3.client' method is not thread safe: boto/boto3#1592 Put a lock around initializing an S3 client, so the two threads that might do this in parallel don't stomp on each other.
The 'boto3.client' method is not thread safe: boto/boto3#1592 Put a lock around initializing an S3 client, so the two threads that might do this in parallel don't stomp on each other. Co-authored-by: fpereiro <fpereiro@gmail.com>
|
Looking at just the PR: not really. It says:
and further down:
Correct me if I am wrong, but clients are obtained from sessions. This would imply that if I have two threads, each one needs their own session, but that a client obtained from either of the two sessions can be shared by both threads. That seems odd and deserves an explanation or a correction. Without either, I'd still be confused as a user of boto3. |
`boto3` sessions are not thread safe. When used in the object store logger with `use_procs: False`, the default session was shared across threads, which caused us to run into boto/boto3#1592. To fix, this PR creates a new session within each `S3ObjectStore` instance. Closes https://mosaicml.atlassian.net/browse/CO-651
`boto3` sessions are not thread safe. When used in the object store logger with `use_procs: False`, the default session was shared across threads, which caused us to run into boto/boto3#1592. To fix, this PR creates a new session within each `S3ObjectStore` instance. Closes https://mosaicml.atlassian.net/browse/CO-651
|
This happened to ms as well. What can be the workaround? |
|
I documented the issue based on my own understanding. |
|
Thanks for linking the blog post, @Gatsby-Lee! That's correct that clients are safe to share between threads must be instantiated in the primary thread. We'd be happy to accept a PR updating the client documentation to be clearer if you have ideas. Otherwise, I'll pass this along to our doc writers to make adjustments. |
|
@nateprewitt What is the reason that the "DEFAULT_SESSION" was introduced at the first place? If it makes more issues than benefits, why don't we update the boto3.client code like this? |
Creating a Session is a very resource intensive process as it has to load and deserialize several models before it can create clients. If this was required every time a client was created we'd see unacceptable performance impact. It's a rare use-case customers actually need individual sessions, so we avoid this by creating a single instance by default for reuse. If this is undesired behavior customers are free to instantiate their own Session as you're showing. With the code you've proposed above, it will have the same performance as creating one client currently. However, we see creating a client take ~15x longer once we start creating many clients. This is because we are currently amortizing that Session cost over each client creation by reusing a single DEFAULT_SESSION instance. Performance Impact Exampleimport timeit
import boto3
CLIENT_COUNT = 1_000
global_time = timeit.timeit(
'boto3.client("s3")',
setup='import boto3',
number=CLIENT_COUNT
)
print(f"Average time per client with Global Session: {global_time/CLIENT_COUNT} sec")
unique_time = timeit.timeit(
'boto3.session.Session().client("s3")',
setup='import boto3',
number=CLIENT_COUNT
)
print(f"Average time per client with Unique Session: {unique_time/CLIENT_COUNT} sec")We'll see results over 1000 runs come out as seen below: $ python perf_test.py
Average time per client with Global Session: 0.001492504167 sec
Average time per client with Unique Session: 0.022752786 secAs you can see this quickly becomes cost prohibitive for time sensitive applications. This is why the current documentation instructs users to pass clients to threads rather than create them in the thread. |
|
@nateprewitt I think the decision can be very opinionated.
I guess that there is not a right or wrong answer. And, since the existing behavior in creating client is almost everywhere, it is not even simple to change it. So, the mitigation can be
Thank you!! |
… connector
`boto3.client()` is not thread-safe. We sometimes get the following
error:
KeyError: 'default_config_resolver'
File "pcapi/tasks/ubble_tasks.py", line 23, in store_id_pictures_task
ubble_subscription_api.archive_ubble_user_id_pictures(payload.identification_id)
[...]
File "pcapi/connectors/beneficiaries/outscale.py", line 13, in upload_file
client = boto3.client(
File "__init__.py", line 92, in client
return _get_default_session().client(*args, **kwargs)
[...]
File "botocore/session.py", line 1009, in get_component
del self._deferred[name]
To work around that, we now use a dedicated session. And we keep one
for each thread to lessen the overhead of the session initialization.
References:
- discussion of the issue and possible workarounds: boto/boto3#1592
- similar fix in Sentry client code: getsentry/sentry@f58ca5e
… connector
`boto3.client()` is not thread-safe. We sometimes get the following
error:
KeyError: 'default_config_resolver'
File "pcapi/tasks/ubble_tasks.py", line 23, in store_id_pictures_task
ubble_subscription_api.archive_ubble_user_id_pictures(payload.identification_id)
[...]
File "pcapi/connectors/beneficiaries/outscale.py", line 13, in upload_file
client = boto3.client(
File "__init__.py", line 92, in client
return _get_default_session().client(*args, **kwargs)
[...]
File "botocore/session.py", line 1009, in get_component
del self._deferred[name]
To work around that, we now use a dedicated session. And we keep one
for each thread to lessen the overhead of the session initialization.
References:
- discussion of the issue and possible workarounds: boto/boto3#1592
- similar fix in Sentry client code: getsentry/sentry@f58ca5e
This should fix errors with client auth. See boto/boto3#1592 and https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-or-multiprocessing-with-resources Issue #80
And you get, tested on my Windows 10 machine (boto3 version 1.5.31, botocore version 1.8.45) and also on an Amazon Linux EC2, a bunch of exceptions like this:
or:
Normally seems to be several credential_provider ones followed by several endpoint_resolver ones.
The chance of getting these exceptions seems to increase with the number of threads.
The text was updated successfully, but these errors were encountered: