MakeTx - Why is the alpha premultiplied in output?

[Ophtis]

Hi !

I have a few questions about how OIIO handles the alpha.
I have an image with unpremultiplied alpha (a PNG), and I used maketx.exe from Autodesk Arnold to convert it to Tx.
Then, I tryied to directly use the latest OpenImageIO release, with differents flags.
So I used the make_texture with python, with ignore unassociated alpha flag, but it's still premultiplied in conversion.

Here is my code:

import OpenImageIO as oiio

path = "D:/test_alpha_oiio.png"
output = "D:/test_alpha_oiio_02.tx"

Input = oiio.ImageBuf(path)
config = oiio.ImageSpec()
config.attribute("maketx:filtername", "lanczos3")
config.attribute("maketx:ignore_unassoc", 1)
config.attribute("maketx:v", 1)

result = oiio.ImageBufAlgo.make_texture(oiio.MakeTxTexture, Input,
                                        output, config)
print(result)
if not result:
    print("error:", oiio.geterror())

The result is an image TX premultiplied. I've tried maketx:ignore_unassoc, 0 or 1 but no effects.

The only way I successfully had an unpremultplied result was to load the image with a config oiio:UnassociatedAlpha", 1

So my questions are:

• How can I avoid the premultiplication using the maketx.exe?
• How the unassociated alpha is determined? It's in the header of a file, and without it, does OIIO is assuming a PNG file is premultiplied?

For information, I've tried the same image in exr format to tx, and it worked (without any ignore_unassoc flags).

Thanks! 🙏

[lgritz]

Loading an image with configuration hint oiio:UnassociatedAlpha", 1 suppresses that auto-conversion to associated alpha.

I'm looking at the code for the "maketx:ignore_unassoc", and I think it's not doing everything it's advertised to do. I think some fixes are in order.

[lgritz]

Extended background info that everybody should know...

There are two ways to interpret color + alpha in an image:

• "Associated alpha" is when the color channels store how much light is actually represented by the pixel, and the alpha says how much it hides anything that's composited behind it (you can think of alpha = 1 - transparency). For example, suppose you have an image of an opaque grey spaceship where a pixel that is aimed at the middle of the spaceship hull will cause a camera pointed at it to register 50% of the maximum light it can sense, so we assign that pixel (0.5, 0.5, 0.5, a=1.0). Imagine another pixel on the camera sensor that is aimed and the edge of the spaceship, so in some sense the area of that pixel is 1/2 covered by the middle gray color, and 1/2 aimed at empty space, so the total light hitting that sensor is 1/2 what it would be if the spaceship covered the whole pixel, and we give the pixel value (0.25, 0.25, 0.25, a=0.5) meaning that this pixel shines light onto the sensor of value 0.25, and also the pixel is 50% covered, so if we composite this image on top of something else, 50% of the background in that pixel ought to show through. The specific compositing math is

   result = foreground + (1 - foreground.alpha) * background
  

  This math is the same for all channels, including alpha.

• "Unassociated alpha" is when the color channels store the way the object would look if it were totally covered by the object (even if it is not!), and so alpha is interpreted as a mask saying "how much of this color do you really see?" So the middle of the spaceship hull that totally covers a pixel would still be (0.5, 0.5, 0.5, 1.0), just like for associated alpha, but the pixel that covers the edge in this scheme would have value (0.5, 0.5, 0.5, 0.5). In other words, the color channels no longer hold the amount of light that would arrive at the camera, but instead holds some solid color and the alpha separately gives you a "mask" saying how much of that is visible or hidden. Now if you want to composite this over a background, the math would be

  result.alpha = foreground.alpha + (1 - foreground.alpha) * background.alpha
  result.rgb   = ((foreground.rgb * foreground.alpha) + (1 - foreground.alpha) * (background.rgb * background.alpha)) / result.alpha
  

  I think? So it's more complicated math, harder to think about, and the math is not the same for the alpha channels as for the color channels.
  Moreover, there are pixels that associated alpha can represent but unassociated cannot. Consider the spaceship's exhaust, which is a brightly glowing (emissive) gas that does not obscure what's behind it. It might have an associated alpha value of (0.5, 0.5, 1.0, a=0.0), meaning that it's bright blue-white but doesn't absorb what's behind it. How can that be represented as an unassociated alpha image? It can't!

You can convert from unassociated to associated alpha by multiplying the color channels by alpha. (That's why some people call associated images "pre-multiplied".) You can try to go from associated to unassociated by dividing by alpha, but this doesn't always work (divide by zero?), and you will lose any of those associated pixels where, like I described earlier, you have certain types of glowing or transparent pixels. It's not just pure emission or glows, this can also pop up in other unexpected places. For example, consider a piece of glass that is 99% transparent but catches a very bright specular highlight -- its associated value might be (0.8, 0.8, 0.8, 0.01), and it has no way to be represented as an unasociated pixel in a non-HDR format because normal integer 8-bit or 16-bit pixel could represent values > 1 and the theoretical way to represent this pixel as unassociated would be (80, 80, 80, .01). In fact, for an integer-based format, unassociated alpha cannot represent pixels where the light sensor values are greater than the opacity.

So for these reasons -- easy to reason about, simple and uniform math, and the ability to represent pixels where the light amount is greater than the alpha value -- rendering and compositing software really, really, really want to think entirely in terms of associated alpha.

---

OK, so now that brings us to OIIO.

Most "professional quality" image formats only allow associated alpha, because that's the only way to think about the behavior of light that makes any sense at all. (Not all image formats are so enlightened, let's just leave it at that.)

OIIO by default automatically translates any unassociated alpha it comes across into associated -- both because this is almost always what you want, and also to simplify the task of the software calling it. Otherwise, app software would EVERY TIME have to figure out if the image it was asking OIIO to read was associated or unassociated alpha, and have its own logic to handle both cases. So OIIO does all that for you and lets you assume any pixels you get from OIIO are by default associated alpha.

PNG, in its infinite wisdom, by its own specification, insists that it always stores unassociated alpha.

If you start with a PNG file, then the default behavior of maketx is to convert it to associated alpha. You can use the --ignore-unassoc is supposed to disable this, but apparently it does not. Sorry about that, I will fix.

I don't recommend storing your textures unassociated. I mean, at the very least, that means your shaders need to specifically know which texture are associated and which are unassociated, and handle each case properly.

[lgritz]

OK, so I looked into this more today. Here's what I found:

1. You are right that direct calls to make_texture(), either in Python or C++, did not properly honor the "maketx:ignore_unassoc" option. I've posted make_texture: ensure that "maketx:ignore_unassoc" is honored #3269 as a fix for this.
2. However, the maketx executable did some extra work that, as far as I can tell, did this correctly all along.

[Ophtis]

Thank you so much for these explainations !
And also for the fix, I will test it as soon as I can recompile with vcpkg👍 (since I don't see myself compile oiio from source haha)

About the 2. of your answer, I was using at first the maketx.exe from Autodesk Arnold, wich has not the ignore-unassoc flag, that's why I directly use it through OIIO's Python API.

But indeed, I just tried with the maketx.exe I compiled from OIIO (vcpkg gets version 2.2.10), and yep, it's working with --ignore-assoc. 🙂

Thank you Larry!
Regards