RuntimeError('Memory allocation failed') on appending XMP tags to large JPEG's.

[kolt54321]

Hi! I'm using Pyexiv2 to write XMP tags to a large JPEG file - 1.5gb, 61328 x 61535 pixels. Since the file limit is 2gb this should work. However, I am hitting RuntimeError('Memory allocation failed'). The same image at lower resolution work with the same tages.

The piece of code is simple enough:

    final_image_filename = outfile or (info.image_name + '.jpg')
    img.save(final_image_filename, quality=quality, subsampling=0) ##PIL saving
    dir_path = os.path.dirname(os.path.realpath(__file__)) + '\\' + final_image_filename ##Passing the file 
    print(dir_path)

    for key, value in info.metadata.items():
        try:
            xmp_file_obj.modify_xmp({key: value})
        except RuntimeError as e:
            print(f'Failed to add add XMP tag with key "{key}" with value "{value}"')
            print(repr(e))

Does Pyexiv2 have to open the image in memory in order to add tags to it? I'm not sure where it's gone wrong here.

[LeoHsiao1]

The workflow of class pyexiv2.Image is as follows:

1. Read the image from disk
2. Modify the image in memory
3. Save the image from memory to disk

While class pyexiv2.ImageData is used to open and save images in memory.

How do you create xmp_file_obj in your code?

[LeoHsiao1]

Can your code successfully handle other images that are less than 1.5g?

[kolt54321]

Thanks for the clear rundown. Yes! Lower resolution images work perfectly, and I've appended metadata (XMP tags) to ~700mb JPEG's before. I forgot the line of code for xmp_file_obj (my bad, copied and pasted the pieces separately), so here's the full piece:

    final_image_filename = outfile or (info.image_name + '.jpg')
    img.save(final_image_filename, quality=quality, subsampling=0)
    dir_path = os.path.dirname(os.path.realpath(__file__)) + '\\' + final_image_filename
    print(dir_path)
    
    xmp_file_obj = TaggedImage(dir_path) ##xmp_file_obj defined here

    for key, value in info.metadata.items():
        try:
            xmp_file_obj.modify_xmp({key: value})
        except RuntimeError as e:
            print(f'Failed to add add XMP tag with key "{key}" with value "{value}"')
            print(repr(e))
    shutil.rmtree(tiles_dir)
    print("Saved the result as " + final_image_filename)

Where taggedImage(dir_path) is just the full path, with the image name appended to it. I don't think that's the issue as I tried xmp_file_obj = TaggedImage(final_image_filename) which points to the same file, and returns the same error.

It's worth noting that 1.5gb is after JPEG compression... is it possible that the file is opened in memory uncompressed or something?

[LeoHsiao1]

I'm not familiar with the JPEG image format and don't know if it's greater than 1.5g in memory.
It is recommended that you save the 1.5g image separately to disk and see if you can open it with pyexiv2.

I get an exception when I use pyexiv2 to open images larger than 2g, but not the exception that you get.

[kolt54321]

I just tried with the image saved to disk first, and modifying that directly (class pyexiv2.Image was used).

No go, unfortunately, same error as before. Can I send you the file to check?

[LeoHsiao1]

I want to try opening your image, but how to send it to me?

[kolt54321]

I'll give you a google drive link after it finishes uploading in a few minutes - does that work?

[LeoHsiao1]

How long will it take?If it is too late, I will deal with the problem in 24 hours.

[kolt54321]

Not a problem, it's about halfway done by now. Not sure how late it is for you but it's expecting to be done in around 5-10 minutes.

[LeoHsiao1]

OK. I can wait another hour.

[kolt54321]

The file can be accessed here. Let me know if you need access to the code that generates the XMP tags for testing.

[LeoHsiao1]

It takes me a few hours to download this image, so I'll process it in 20 hours.

[kolt54321]

Sure, looking forward.

[LeoHsiao1]

The problem seems to be that Exiv2 cannot modify images larger than 1G. I am waiting for the confirmation from the Exiv2 developer.
Exiv2/exiv2#1248

[kolt54321]

Thanks for the update! Let's hear what he says.

[LeoHsiao1]

So, I'm assuming pyexiv2 can only read the image less than 2G and modify the image less than 1G. But this feature has not been fully tested.

[kolt54321]

Thanks for testing that out. I'll put a conditional in limiting to files 1gb or less.

[github-actions]

This issue has been automatically closed because there has been no activity for a month.