md5 hash with upload_fileobj.

[sidheshdivekar29]

Hi,
Does upload_fileobj take care of making sure the md5 of
the file being uploaded matches the md5 of the uploaded file in s3
once upload is over ?

I see since upload_fileobj is done in multipart md5 hash in etag is of the format hash-2
and differs from the original md5 of the file.

How to make sure md5 of the file being uploaded matches the md5 of the file in s3 efficiently.
If upload_fileobj makes sure of taking care of integrity then application can safely assume
that object went to S3 and don't have to implement anything to match the md5's.

[kyleknap]

Yeah so the md5 is only equal to the ETag under certain circumstances (i.e. non multipart upload).

Boto3 does not integrity check the md5 of the entire file for multipart uploads, but it does send an md5 header for each part that is uploaded when doing a multipart upload such that if there is an md5 mismatch in any of the parts, it will retry the request until it is correct.

That is probably the best we can do while still doing the transfer efficiently because determining the MD5 and doing integrity checking would require streaming the entire file upfront into memory.

How large are the files you are uploading? If they are small enough you may be able to increase the multipart threshold so multipart uploads are not used if md5 checking of each individual part is not sufficient.

[sidheshdivekar29]

Thanks for replying. I think we should be good if library is matching md5's of the individual
chunks uploaded and retrying in case of failures.
Is there a command or way to enable debugs/traces to check the no. of tries which happened/happening and if they were successfull.

[joguSD]

You can turn on debug logs by adding: boto3.set_stream_logger('').
The number of retries is also available in the response metadata.
ResponseMetadata.RetryAttempts

[sidheshdivekar29]

Thanks for adding the logging support.