Ultra HDR in WebP

[grishka]

This question is mostly about the format itself rather than this library, but...

Since I recently got myself a phone that shoots in ultra HDR, I naturally started wanting everything to support it. Before asking other people about it though, it feels fair to first explore adding support to my own project. And this is where I hit a snag: I make heavy use of WebP in my image processing/storage pipeline. I convert all user-uploaded images to high-quality WebP for storage, to save space, and then I serve smaller WebP and JPEG representations of that one file through imgproxy. Both Smithereen itself and imgproxy use libvips for image processing, which is considering adding support for ultra HDR.

However, even when libvips does add that support, it still doesn't appear that WebP is one of the supported formats:

• The Android documentation straight up ignores everything except JPEG
• And this library only supports JPEG, HEIF, and AVIF:

  libultrahdr/ultrahdr_api.h

  Lines 148 to 153 in ea518da

  	/*!\brief List of supported codecs */
  	typedef enum uhdr_codec {
  	UHDR_CODEC_JPG, /**< Compress {Hdr, Sdr rendition} to an {Sdr rendition + Gain Map} using jpeg */
  	UHDR_CODEC_HEIF, /**< Compress {Hdr, Sdr rendition} to an {Sdr rendition + Gain Map} using heif */
  	UHDR_CODEC_AVIF, /**< Compress {Hdr, Sdr rendition} to an {Sdr rendition + Gain Map} using avif */
  	} uhdr_codec_t; /**< alias for enum uhdr_codec */

So what's the best course of action for me here? I did give AVIF a try a while ago, but was unsatisfied with the glacially slow encoding speed (it took hundreds of milliseconds per image on an Apple M1), so unless something has changed about that, switching to AVIF isn't really a viable option.

[ram-mohan]

This is my understanding.

While in theory gain map technology can be applied to any image standard, to my knowledge there was no initiative towards webp standard. ultrahdr (base image + gain image) format requires the image file standard the ability to signal multiple images in single file. Jpeg standard does not support signalling more than one image in its file, but there is an extension format for jpeg, JPEG MPO, which allows signalling multiple images in single file. Using jpeg mpo, primary image and gain map image are signalled together along with some metadata (gainmap metadata) which are combined at the decoding side to create the final hdr image. Similarly, image file formats that are based on ISO base media file format (ISOBMFF), like heif, avif, jpegxl, ... also have the ability to signal multiple images and hence can support ultrahdr.

I think webp format cannot signal multiple images and hence it may not be possible to achieve ultrahdr like effect with webp.

coming to avif, you can perhaps tune some encoding speed presets and disable few toolsets to achieve your target compression (if the default setting is taking more time than what you desired).

[gregbenz]

@grishka This is very interesting. At the same time, I think swimming upstream here is going to be very tough. If there isn't browser support,the map won't be widely useful.

Have you tested encoding speed for HEIF and JXL? They seem like the most promising options for gain map support on the web beyond AVIF. Both already have working implementations from major platforms. That said, AVIF seems like the only viable option in the near term (support for SDR fallback is getting close to 100% and Chrome has a dev flag with support for HDR decoding already).

[grishka]

@gregbenz Well, I want there to be browser support. What do I do to make that happen? It feels like it should be a fairly small change to both Chrome and Android as nothing else at the moment supports Ultra HDR at all anyway. Just, like, remove the "is it a JPEG" check :D

No, I haven't. HEIF is only supported by Safari and JXL is too recent, I haven't tried it yet. I'm not looking for near term anyway — I need libvips and imgproxy to at least somewhat support gain maps before I could even begin thinking about writing any related code on my side.

[gregbenz]

@grishka Android 16 adds HEIF gain map support, and the stated roadmap for libultra also includes support. So there is movement towards support beyond Apple for it. https://www.theverge.com/news/611975/android-16-public-beta-camera-app-hybrid-auto-exposure-color-temperature-tiny-adjustments

JXL seems less clear to me. Chrome pulled support (though it was working great and decisions like that can be revisited), but Adobe has a lot of support for it and it's a great format.

[grishka]

So I guess with how it's progressing, it's a decade or two until we settle for something that would make ultra HDR universally and easily supported ¯\_(ツ)_/¯

It's almost like everyone involved doesn't want to make it easy, almost like they don't have it as a goal to make it easy to support ultra HDR in existing image processing pipelines and storage solutions.

[gregbenz]

@grishka No, not at all. JPG gain maps are well supported in browsers now (not yet transcoding and that''s what efforts like libultrahdr will help solve). AVIF gain maps should be well supported in about a year, with HEIF and JXL maps being promising.

The ISO standard isn't yet final, but soon will be. There is steady progress towards increased support, just not necessarily towards the specific encoding you may prefer (webp). I'm not suggesting everything will be the same as sharing a standard gain map in two years, but we are fairly quickly moving towards a state where you can safely and easily share these images in a number of channels.

In addition, standard HDR images (without a gain map) are decently supported with AVIF now (Firefox is a mess and Safari tone mapping isn't great - gain maps are always ideal, but this is yet another option). I personally discount HDR-only formats as gain maps have superior image quality, but these do meet the needs of many. For example, WordPress 6.8 will soon offer native support for HDR AVIF and that's very useful even if gain maps aren't there yet.