make event stdout encoding more resilient to UTF-16 surrogate pairs #7918
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Build failed.
|
|
recheck |
jakemcdermott
approved these changes
Aug 17, 2020
|
There's also apparently libraries for actually handling these more gracefully: https://pypi.org/project/surrogates/ ...but I also don't know that I want to add a dependency to deal with something so esoteric. |
|
Build failed.
|
c04c5c3 to
8b2641c
Compare
ryanpetrello
commented
Aug 17, 2020
| class SurrogateEncoder(encoders.JSONEncoder): | ||
|
|
||
| def encode(self, obj): | ||
| ret = super(SurrogateEncoder, self).encode(obj) |
There was a problem hiding this comment.
This is a little gross, but I couldn't find a less gross way to do this.
tl;dr there’s code further down the stack in DRF that ends up calling .encode() (and it blows up), so I’m calling it, seeing if it blows up, and then fixing the problem (so it won’t blow up later)
| try: | ||
| ret.encode() | ||
| except UnicodeEncodeError as e: | ||
| if 'surrogates not allowed' in e.reason: |
There was a problem hiding this comment.
since ret is just a python str this assumption lgtm 👍
|
Build failed.
|
|
recheck |
|
Build failed.
|
|
Build failed.
|
|
recheck |
8b2641c to
f2c9437
Compare
|
Build succeeded.
|
f2c9437 to
33e2c05
Compare
|
Build succeeded.
|
|
Build succeeded (gate pipeline).
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Given that we're returning literally any bytestream that Ansible can muster, it probably makes sense to relax
errors='strict'a bit to avoid 500 errors.see: https://unicodebook.readthedocs.io/unicode_encodings.html#utf-16-surrogate-pairs
Run this adhoc command w/
-m shell:printf '\uD83D\uDE00'after (better than a 500 error):