Info on clipBlack setting in chroma effect

[emmanuelgeoffray]

Hey @brianchirls
Thanks for this super lib. The more I use it the more I love it ;)
I'm trying some effects with chroma key and I have some questions on it.
Sorry that's not really an issue but more a forum question on how to use.
Anyway,
The green detection is working great, as seen in this screencap:

(settings used in the bottom right corner)

But when I slide the clipBlack to 0, I'm expecting to see the full body, but I'm still seeing some black pixels on my face :)
I thought clipBlack and clipWhite were tools to setup the chromakey but not used after.
I guess I'm getting it wrong.
How could I slide the clipBlack "below 0" to have no black incrust at all?

Thanks in advance for your answer.
cheers,
e

[brianchirls]

e,

I'm so glad you're pleased with the library so far. I'd love to see the work you've done with it once you're ready to share.

I'm not quite sure exactly what's going on here without knowing how you've set up the composition. It looks like there are at least a couple of compositing passes here - I see the white background with the shiny pink thing behind you and also the black spots on the faces and hands. So I'll try my best to explain how clipBlack and clipWhite work and will hope that it's helpful.

First of all, clipBlack and clipWhite cannot go below 0 or above 1. If you try to set the values outside that range, they'll just be clipped...

chroma.clipBlack = -9001;
console.log(chroma.clipBlack); // 0

chroma.clipBlack = 9001;
console.log(chroma.clipBlack); // 1

These two properties both affect the alpha value of the output. The chroma filter makes an attempt to give a smooth transition between foreground and background. If the dominant color channel of any given pixel is not the same as the dominant channel (RGB) of the screen color, then that pixel is solid (i.e. alpha = 1). If the dominant color component matches and the saturation is at least as high as the screen color, then we can be pretty confident that that pixel is background (alpha = 0). If the dominant component matches, but the saturation is not as high as the screen color, then we fall somewhere in the middle. This is where clipBlack and clipWhite come in.

If the resulting alpha value is less than the value of clipBlack, then it will just be clipped to 0. If it's greater than clipWhite, then it will be clipped to 1. There is some interpolation in between to keep the output smooth.

Let me know if that helps. If not, feel free to share some working code or provide a more detailed description of how your nodes are set up.

Brian

[emmanuelgeoffray]

Hi Brian,

Thank you very much for your prompt and very detailed response.
I really appreciate your help on this, and I will definitely share the project with once I'm proud of it ;)

As to your explanation of clipBlack and clipWhite, I understand more of it now.
But I still don't understand how I have some black blobs on screen.
I check with the original demo

I guess it is pixels that fall in the case of dominant color matches but saturation is not as high as the green background.
But then it should clip to apha = 0, being transparent, ie. showing the background instead of checging the rgb to black.

I guess my green background is not saturate enough.
But I feel it could work because with clipBlack at a value 1 shows a perfect match of the foreground.

That image could be multiplied to the foreground to have the final result.
Could I tweak some parameters to achieve that?

For info, I attached is the original picture to show the saturation of the green background.

cheers! ;)
e

[brianchirls]

If the only effect you're using is chroma and you're not setting the mask flag, then you should not see any black spots at all, no matter what the clipBlack or clipWhite values. I tried it on my machine with the sample image you posted, and I don't have any black spots.

I wonder if there is something specific to your GPU driver and the way it's handling the particular shader functions I'm using. Are you on Linux? Can you share the details on your GPU and OS?

[emmanuelgeoffray]

Yes only effect is chroma. I'm using the examples without any modifications.
And mask is not set. Did you try with color #a2c3af ?

Yes, I'm on linux mint 16, 64bits, with a GeForce GTX 770 and nvidia drivers installed.
Chrome/39.0.2171.95

[brianchirls]

Okay, we're in business. Looks like I'm seeing the problem on my home machine, but not my work machine. Now I can debug it. Stay tuned...

[brianchirls]

Try that. There was a divide-by-zero error. I guess on some machines it was ignoring it because the alpha value of the resulting color was zero anyway.

[emmanuelgeoffray]

Thanks! Checking it right now

[emmanuelgeoffray]

Working!
Thank you very much!

[brianchirls]

Great! Thanks for catching this one.

https://www.youtube.com/watch?v=xBC6zJ6P29A#t=5s

[mostkamp]

Great, brianchirls' fix really does the trick. However, I had to spend a few hours to figure this out, so it would be great if this could be integrated into the downloadable version as soon as possible.

Thanks for the good work!

[mariomka]

I have issues on Android device BQ Aquaris M5.5 (Snapdragon 615, Adreno 405) with black spots.
I think this device treat 0.00001 like zero but 0.0001 works perfectly. I will make a pull request today later.

[javl]

I ran into this exact problem on an iPad 2, Safari 9 (and Chrome 33) on iOS 9.3 today.
The fix suggested by mariomka worked for me, though I had to change the max value all the way up to 0.01 for it to work.
Maybe this value could be a parameter of the chroma effect? (edit: I just created pull request #141 for this).

Of course tweaking this value might produce unexpected results on some other systems (though in this case it still works fine on my PC as well) but it might proof useful when optimizing for a specific device (like I'm doing with this iPad 2: the page I'm making will not be visited on other devices).

0.00001 -> (original) lot of black areas
0.0001 -> still a lot of black areas
0.001 -> less weird areas, but the ones left have become white and yellow
0.001 to 0.01 -> visible banding/low color detail in some areas
0.01 -> perfect

edit:
If my pull request gets merged, changing the precision can be done like this:

chroma = seriously.effect('chroma');
chroma.precision = 0.01;

[brianchirls]

@javl Have you tried setting the precision to 1 / 255.0?

[javl]

@brianchirls Just tried: gives me the odd banding / grouping of colored pixels again.
The values I mentioned before I got from trial and error, but 0.01 really seems to be the lowest I can go on this device.

I can't take a screenshot on the actual device but uploaded a photo of the screen. You can see artifacts on the ceiling and next to the lamp in the picture when the precision is set to 1/255.0. With the original settings these areas are black.

set to 1/255.0 (artifacts on ceiling): https://imgur.com/kCQKxl8
set to 0.01 (no artifacts): https://imgur.com/T4oUOKn

Note the chairs in the image which are keyed and show a dark pattern through them. The actual keying of those chairs is fine in both images and could be even better with some finetuning, it's just the one with 1/255.0 that has those artifacts.