Add an "Invert Shadow Attenuation" option to Light3D nodes

[merlang7]

Describe the project you are working on

A 3D platformer, aiming to emulate a late Gamecube/Wii aesthetic

Describe the problem or limitation you are having in your project

for a 3D platformer, having a vertically aligned drop shadow at all times is a good tool for the player, especially for those who struggle with depth perception. In Godot, there are 3 ways of achieving that:

1. attaching a decal to the player node that "fakes" the shadow. (functionally works, but stands out as ugly in nicer environments)
2. attaching a negative spotlight oriented down to achieve a similar effect (comes with the same problems as option 1)
3. setting the DirectionalLight3D to point straight down, and enabling shadows. This is the solution I'm currently using but it has several limitations. Firstly, it creates a lot of ugly lighting situations due to the angle of the light. On top of that, the shadows only help with platforming when the player is in direct sunlight; otherwise, the shadows from the environment will make the ground beneath them indistinguishable from other shadowed areas.

however, other games clearly use a different solution. take this Mario Odyssey screenshot as an example:

the environment shadows are clearly at an angle, but Mario's shadow is pointed straight down.

also notice the difference in coloration:

• the "proper" shadows are blueish due to ambient light from the sky filling in the absence of yellow sunlight.

• Mario's shadow is tan, as if his shadow is simply subtracting light from the environment, rather than actually occluding sunlight.

in this next screenshot, we can see that the subtractive theory is correct. Mario's shadow darkens the floor, even though its already in shadow. (notice that the tan tinge is gone. since the subtraction darkens the lighting based on the color already there after "proper calculations," the subtractive shadows are blue if applied on top of "proper" shadows, and tan if applied in open sun)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

By having an option to invert the effect that shadows have on light attenuation (light only applied in shadowed areas, rather light only applied in unshadowed areas)...

you can set the light to "negative," and easily get subtractive shadows, creating a "player shadow-only light" that subtracts light in shadowed areas, rather than adding it in unshadowed areas.

the solution I'm proposing is most useful in combination with #85338 , as it would let only certain objects be included in this shadow map calculation. This would let environmental shadows use the occluding shadow system in Godot with light and shadow angles that are visually pleasing, while having the player and other objects cast subtractive shadows straight down for gameplay purposes (just like we saw with the Mario Odyssey example).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

in the example shader i made to get the screenshots in the previous section, I set the diffuse light calculation in the material's light() function to
DIFFUSE_LIGHT += clamp(dot(NORMAL, LIGHT), 0.0, 1.0) * (1.0 - ATTENUATION) * lightColor;

this simple inversion on a per-material basis is impractical, since it would effect all lights, defeating the point. the ideal implementation would be to have a checkbox under the Shadow dropdown on all Light3D nodes called something along the lines of "Invert Shadow Attenuation," similar to the "Negative" option under the Light dropdown.

If this enhancement will not be used often, can it be worked around with a few lines of script?

as previously mentioned, a user-side ShaderMaterial based approach is insufficient.

Is there a reason why this should be core and not an add-on in the asset library?

1. this issue pertains to rendering.
2. as its a single checkbox from the user's perspective, it wouldn't cause any confusion, and could be easily ignored by anyone who doesn't need it (the same way most people ignore "Negative" lights until they need them).

[Calinou]

What you're asking for is quite similar to the subtractive shadowmapping approach mentioned in #2354, but without lightmapping into the mix. (It should be possible for this approach to work in tandem with lightmapping still; the light would be ignored while baking lightmaps.)

It's quite common for games to use stencil shadows for this kind of fake directional shadow, but it's not the only approach (Mario games definitely use a shadow map for this).

The approach you mentioned for this is quite smart (it should require less intrusive changes than having a "true" fake shadow system), but I'm concerned negative blending will not give you the results you want. Negative lighting will slightly affect surfaces that are bright, but strongly affect surfaces that are dark, which can make the shadow look off if the lighting/material conditions vary a lot in the scene. I've already encountered this issue when using a negative SpotLight3D parented to the character that pointed downwards.

Also, care should be taken with performance as using multiple DirectionalLight3Ds with shadow enabled is slow. It will also reduce the shadow map resolution for each DirectionalLight3D, as they share a shadow atlas. Lastly, using multiple DirectionalLight3Ds with shadow enabled is not supported when using the Compatibility rendering method, so this approach will only work for Forward+ and Mobile. You could circumvent this issue by using a SpotLight3D that points straight down instead (placed way above the character model, with a low spot angle).

[merlang7]

but I'm concerned negative blending will not give you the results you want. Negative lighting will slightly affect surfaces that are bright, but strongly affect surfaces that are dark [...] I've already encountered this issue when using a negative SpotLight3D parented to the character that pointed downwards.

I've definitely encountered similar issues when using negative Spotlight3Ds and OmniLight3Ds, but imo the contrast issues are way less egregious in this specific case since there's no distance attenuation term to meddle with things; its either in shadow or it isn't

you can see that it gets to the point of being pitch black when the light level is very low, and while it certainly looks a bit strange its not nearly as bad as... this, and whatever else negative spotlights like to do.

plus, Mario Odyssey has lots of areas with very dark shadows, and i think it looks fine

I've tried to engineer the worst situation possible for a setup like this - dark textures with high contrast lighting (via the ambeint setting of a reflection probe) - and it looks acceptable

using multiple DirectionalLight3Ds with shadow enabled is slow. It will also reduce the shadow map resolution for each DirectionalLight3D

those are good points. the Shadow Caster Masks being added would hopefully help with the performance side of things, but the resolution thing can't really be fixed. personally I think its a worthy trade-off for defined character shadows over the circular blobs of the decal/spotlight options, but it's a trade-off regardless.

It's quite common for games to use stencil shadows for this kind of fake directional shadow

if Godot opened up the stencil buffer that would basically make this problem go away, but that seems like a much bigger undertaking than something like this