[Fix issue 415] Fallback to In Memory Encoding for JPEG Constructed from Byte Streams

[XiaohanZhangCMU]

Description of changes:

As title says.

Testing code

from streaming.base import MDSWriter
from PIL import Image
from io import BytesIO
import numpy

# Create random images 
for n in range(10):
    a = numpy.random.rand(30,30,3) * 255
    im_out = Image.fromarray(a.astype('uint8')).convert('RGB')
    im_out.save('out%000d.jpg' % n)

# This works even without the fix
 with MDSWriter(out='./mds_out', columns={'jpeg_img': 'jpeg'}) as out:
     for i in range(10):
         img = Image.open(f'./out{i}.jpg')
         out.write({'jpeg_img': img})

# This would have failed without the fix.
with MDSWriter(out='./mds_out_bytes', columns={'jpeg_img': 'jpeg'}) as out:
    for i in range(10):
        with Image.open(f'./out{i}.jpg') as im:
            img_bytes_io = BytesIO()
            im.save(img_bytes_io, format='JPEG')
            image_data = img_bytes_io.getvalue()
            image_bytes_io = BytesIO(image_data)
            img = Image.open(image_bytes_io)
            out.write({'jpeg_img': img})

Comparing index.json from both

mds_out/index.json
{"shards": [{"column_encodings": ["jpeg"], "column_names": ["jpeg_img"], "column_sizes": [null], "compression": null, "format": "mds", "hashes": [], "raw_data": {"basename": "shard.00000.mds", "bytes": 12606, "hashes": {}}, "samples": 10, "size_limit": 67108864, "version": 2, "zip_data": null}], "version": 2}

vs

mds_out_bytes/index.json
{"shards": [{"column_encodings": ["jpeg"], "column_names": ["jpeg_img"], "column_sizes": [null], "compression": null, "format": "mds", "hashes": [], "raw_data": {"basename": "shard.00000.mds", "bytes": 12085, "hashes": {}}, "samples": 10, "size_limit": 67108864, "version": 2, "zip_data": null}], "version": 2}

Issue #, if available:

Fixes #415

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.