Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(BSR)[API] fix: Fix non-thread safe usage of boto3 client in outscale connector #5884

Merged
merged 1 commit into from
Apr 4, 2023

Conversation

dbaty
Copy link
Contributor

@dbaty dbaty commented Mar 28, 2023

boto3.client() is not thread-safe. We sometimes get the following
error:

KeyError: 'default_config_resolver'
  File "pcapi/tasks/ubble_tasks.py", line 23, in store_id_pictures_task
    ubble_subscription_api.archive_ubble_user_id_pictures(payload.identification_id)
  [...]
  File "pcapi/connectors/beneficiaries/outscale.py", line 13, in upload_file
    client = boto3.client(
  File "__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)
  [...]
  File "botocore/session.py", line 1009, in get_component
    del self._deferred[name]

To work around that, we now use a dedicated session. And we keep one
for each thread to lessen the overhead of the session initialization.

References:


L'erreur sur Sentry : https://sentry.passculture.team/organizations/sentry/issues/408606/?project=5&query=default_config_resolver&statsPeriod=90d

@dbaty dbaty force-pushed the dbaty/workaround_boto_client branch from 429c700 to 0396add Compare March 28, 2023 16:11
… connector

`boto3.client()` is not thread-safe. We sometimes get the following
error:

    KeyError: 'default_config_resolver'
      File "pcapi/tasks/ubble_tasks.py", line 23, in store_id_pictures_task
        ubble_subscription_api.archive_ubble_user_id_pictures(payload.identification_id)
      [...]
      File "pcapi/connectors/beneficiaries/outscale.py", line 13, in upload_file
        client = boto3.client(
      File "__init__.py", line 92, in client
        return _get_default_session().client(*args, **kwargs)
      [...]
      File "botocore/session.py", line 1009, in get_component
        del self._deferred[name]

To work around that, we now use a dedicated session. And we keep one
for each thread to lessen the overhead of the session initialization.

References:
 - discussion of the issue and possible workarounds: boto/boto3#1592
 - similar fix in Sentry client code: getsentry/sentry@f58ca5e
@dbaty dbaty force-pushed the dbaty/workaround_boto_client branch from 0396add to 84ae82a Compare April 4, 2023 09:56
@dbaty dbaty merged commit 84ae82a into master Apr 4, 2023
@dbaty dbaty deleted the dbaty/workaround_boto_client branch April 4, 2023 09:56
@dbaty dbaty temporarily deployed to testing April 4, 2023 10:12 — with GitHub Actions Inactive
@dbaty dbaty temporarily deployed to testing April 4, 2023 10:13 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants