gain map at start of file?

[jcupitt]

Hello everyone, I work on libvips and I'm looking at adding libultrahdr support.

I have a general question about file layout. My understanding is that at the moment the gain map is written right at the end of the file, after the SDR image data. This is a poor fit for libvips, since our _new_from_file() method has to be quick, and has to return all metadata. With the gain map at the end, we can't get to it (as I understand it) without reading the entire image.

Out of curiosity, does anyone know the reasoning behind this decision? I'd naively expect the gain map to be stored in an APPn marker, just as ICC profiles are. This would a lot more convenient and efficient for streaming libraries like libvips.

[ram-mohan]

This is my take, while it is true small jpeg images like thumbnails are stored directly as part of APP markers. The size limitation of the jpeg image that can be included in an app marker is 64kb. Very large thumbnail images, supplementary images (panorama, multi-angle, gainmap, ...) often exceed 64kb and cannot be included as part of app markers. For this multi picture format (MPF) is used. This is This is an extension to jpeg to store multiple images in a single file.

MPO files use a JPEG APP2 marker segment (specifically labeled "MPF") to provide information about the images within the file. In this information there is start point and length of the images that are part of the file, which can be used for quick access. This can solve the problem you are mentioning?

[jcupitt]

You can have any size object in APPn markers, for example CMYK ICC profiles are often much larger:

$ vipsheader -a Apple_Group_with_Labels_V3.jpg | grep icc
icc-profile-data: 557168 bytes of binary data

Larger objects are split into a set of 64kb chunks and then reassembled by the loader. All JPEG readers already have code to do this, so I'd think it would have been simple to reuse for gain maps.

MPO files use a JPEG APP2 marker segment (specifically labeled "MPF") to provide information about the images within the file.

Ah, nice! But that won't help image streaming, since you can't seek. libvips will have to buffer the entire ultrahdr JPEG in memory before it can start doing anything :(

If you're curious, there was a news post a few years ago about libvips streaming explaining roughly how it works:

https://www.libvips.org/2019/11/29/True-streaming-for-libvips.html

tldr: libvips will start processing pixels and writing output as soon as the first few bytes of the compressed image arrive. It doesn't usually need to buffer the compressed input image in RAM.