Added support for the extension OES_EGL_image_external(_essl3) for GLES2

[HerrKamrat]

Hi!

I added support for the OpenGL ES extension OES_EGL_image_external. We use this on Android to be able to render textures from the camera and video player without copying them to a regular Texture2D.

This adds a function "bgfx::overrideInternal(ptr, target)" to the api, that allows the user to override the sampler target, and a "SAMPLEREXTERNAL" macro to use in the shaders.
I also patched glsl-optimizer to correctly handle the uniform "samplerExternal".
On backends other then OpenGL ES, the SAMPLEREXTERNAL is treated as a regular SAMPLER2D

Any feedback is welcome : )

[bkaradzic]

"2D" already appears in array...

[HerrKamrat]

The glsl-optimizer replaced all texture lookups in the shader that used a "uniform samplerExternalOES" with the function "textureExternal" (which is deprecated) instead of "texture2D" (which also supportes samplerExternal).

e.g. the input:

uniform lowp samplerExternalOES s_texColor;
...
gl_FragColor = texture2D(s_texColor, v_texcoord0)

would become:

uniform lowp samplerExternalOES s_texColor;
...
gl_FragColor = textureExternal(s_texColor, v_texcoord0)

I felt it was easiest to just change the name of the sampler in the array since the rest was handled correctly

[bkaradzic]

What are possible "target" enums? I'm just curious what user should use here?

[HerrKamrat]

In our case when using a texture from the Camera/MediaPlayer/SurfaceTexture on Android we pass GL_TEXTURE_EXTERNAL_OES as the target. And it could be GL_TEXTURE_RECTANGLE when using the textures from CoreVideo on osx with OpenGL (even if that is deprecated)

[bkaradzic]

I would then figure out better way to handle this, and not introduce another API. For example, since this is only GL thing, you could use overrideInternal, and then use GL shader reflection to set GL_TEXTURE_EXTERNAL_OES for sampler correctly.

[HerrKamrat]

Hmm, I am not 100% sure what you mean by "gl shader reflection", but this is a possible alternative solution that does not expose any new api.
I do not like the #else part, but I have not found any other way to get the target from a texture id for older opengl versions / opengl es.

void TextureGL::overrideInternal(uintptr_t _ptr)
{
       destroy();
       m_flags |= BGFX_SAMPLER_INTERNAL_SHARED;
       m_id = (GLuint)_ptr;
        
#if BGFX_CONFIG_RENDERER_OPENGL >= 43
        // The target of a texture can be checked directly
        GLenum target;
        glGetTexParameteriv(m_id, GL_TEXTURE_TARGET, &target);
        m_target = target;
#else
        // Try to bind the texture to all possible targets
        // and selects the one that does not give an error
            
        const GLenum s_targets[] = {
            GL_TEXTURE_2D,
            // ... etc ...
            GL_TEXTURE_EXTERNAL_OES,
        };
        
        for(int i = 0; i < sizeof(s_targets); i++){
            GLenum entry = s_targets[i];

            glBindTexture(target, m_id);

            // NOTE: might need to rebind the previous bound texture here

            if(0 == glGetError()){
                m_target = target;
                break;
            }
        }
#endif
}

[bkaradzic]

I mean, in code you detect that program is using GLSL_TYPE(GL_SAMPLER_EXTERNAL_OES); and at that point you can assume that whatever if bound to that sampler slot will need to use GL_TEXTURE_EXTERNAL_OES.

[HerrKamrat]

I reworked the pullrequest so that the user-api is unchanged and the texture is bound as a GL_SAMPLER_EXTERNAL_OES if the shader uses that type. Do you think it looks better this way? Any other changes that should be done?

[bkaradzic]

Yeah it looks much better.

[oviano]

This looks useful.

I'm currently figuring out Android MediaCodec decoding via ffmpeg and the best way for interop with bgfx.

It's tricky but it seems like I need to:

Setup:

1. create a GL_TEXTURE_EXTERNAL_OES type
2. make a bunch of JNI calls to create a surface texture, surface listener and surface
3. pass the surface into AMediaCodec_configure (well, the ffmpeg equivalent)

Each frame:

1. to get the texture for a frame, release the MediaCodec output buffer (via ffmpeg), so it gets rendered to the surface, using the surface listener to be alerted to when it's completed.
2. make a JNI call updateTexImage.
3. setup a bgfx texture to have this as it's internal texture
4. draw this texture to the screen

PS and for purely selfish reasons, I liked the suggestion of keeping the same API and having this automatic, because my change to propagate overrideInternal through the updateTexture2D system from a while back (#1984) would be able to use this transparently.

[bkaradzic]

If you're already saving target, if there a good reason why you should not save it for all samplers?

[HerrKamrat]

Maybe not. The only reason I did it like that was to only change how the GL_SAMPLER_EXTERNAL_OES was handled and keep the other textures/samplers as before. But the value is only used to bind the texture in TextureGL::commit, so there should not be any problem by saving it for all samplers. I can fix that

[HerrKamrat]

Or one other reason is that we have to map each sampler to their correct target (e.g. GL_SAMPLER_2D -> GL_TEXTURE_2D). I am not sure how samplers with GL_IMAGE_* should be mapped?

[HerrKamrat]

I think something like this could work?

    GLenum glSamplerTarget(GLenum _sampler){
        switch (_sampler)
        {
        case GL_SAMPLER_2D:
        case GL_INT_SAMPLER_2D:
        case GL_UNSIGNED_INT_SAMPLER_2D:
        case GL_SAMPLER_2D_SHADOW:
            return GL_TEXTURE_2D;
        ...  
                
        case GL_IMAGE_1D:
        case GL_IMAGE_2D:
        ...
            return 0;
        }
        
        BX_CHECK(false, "Unrecognized GL sampler type 0x%04x.", _sampler);
        return 0;
    }

[HerrKamrat]

I pushed a solution where the target is saved for each sampler type, and not just for GL_SAMPLER_EXTERNAL_OES.
I noticed that the AppVeyor build failed, but the errors does not look like they originate from my changes. Or what do you think, @bkaradzic ?

[bkaradzic]

Yeah it looks much better.

[bkaradzic]

Can you figure out why CI is failing?

[HerrKamrat]

The CI is failing because the latest version of imgui includes the "windows.h" header without NOMINMAX being defined before, so the min/max functions in bx.h conflicts with the windows macros.
If it is defined somewhere before this line (in the file or in the project settings for examples-common) everything builds fine:

bgfx/3rdparty/dear-imgui/imgui.cpp

Line 1522 in aea0759

#include <windows.h>

[bkaradzic]

Fixed.

[HerrKamrat]

I fetched your fix and the CI succeedes now : )

[HerrKamrat]

Hi again! Just bumping this pr to see if it acceptable or if something more needs to be changed? : )

[bkaradzic]

All good 👍