GFile S3 reader may corrupt files containing multibyte characters or CR/LF

[davidsoergel]

In our stub GFile implementation, the S3 reader obtains chunks of n bytes and then decodes them as strings. This risks corrupting data or at least throwing decoding errors, because the byte chunk may end in the middle of a multibyte character (and correspondingly, the next chunk would begin in the middle of that character).

Thus the chunks cannot be decoded independently. Any partial character must be detected and removed from the end of each chunk, and prepended to the subsequent chunk before decoding.

CRLF (\r\n) is effectively a multibyte character in this context, because Python always "decodes" it to just \n.

See

tensorboard/tensorboard/compat/tensorflow_stub/io/gfile.py

Line 268 in 2b3479b

# TODO(orionr): This endpoint risks splitting a multi-byte

(Followup to #2791 and #2777)

[davidsoergel]

FYI @orionr, @sanekmelnikov, @natalialunova, @lanpa. I can't actually assign you here but please feel free to send a fix if you like. Thanks!