Android native call

[Rugvip]

This requires the following PR to openwebrtc: android-log-filter device-listing-fixes jni-fixes sdp-candidate, and and a server that uses the patch in this PR

sdp.js is currently linked from openwebrtc, which is assumed to have the same parent directory as openwebrtc-examples.

I've found the following issues so far:

• reregistering a surface doesn't work, but when registering the old surface, things will work again
• when the video source is switched, SRTP encoding fails
• remote view sent from Android falls behind and then catches up every now and then.
• when Firefox initiates the call it doesn't receive any remote video (probably sdpfail by me, will look into it)
• no simple way to get rid of black borders on the video, will look into this as well

Other than that this works for both incoming and outgoing calls to both Firefox, Chrome and OpenWebRTC

[sdroege]

For getting rid of the black borders, assuming this is with glimagesink you only need to set force-aspect-ratio to false... and then ideally make sure that your video area has the correct aspect ratio.

What do you mean with reregistering a surface? When e.g. the orientation of the app changes, or it is brought to the foreground again? What are you doing then and what happens?

[Rugvip]

Setting force-aspect-ratio to false just stretches the image instead, which doesn't really solve the problem. A possible solution would be to be able to get the dimensions of the video, or to make it possible to render with a transparent background, and/or be able to set a property similar to this. I'll have a look at how this works on iOS

It's when the orientation changes, for now I just shut everything down when the app is stopped.
When the orientation changes I recreate the layout, so the old surface is unregistered and then a new surface is registered. I'm not sure if it it works only for the first surface because it is saved in memory, or if it has to do with the size of the surface, or something else.

[Rugvip]

It turns out that the registration works fine when there is a call going, although there's a ~1s delay before it starts rendering in the new view.

[Rugvip]

This now also depends on EricssonResearch/openwebrtc#104

[Rugvip]

Update:

• reregistering a surface doesn't work, but when registering the old surface, things will work again
  • this is only an issue when we're not sending video
• when the video source is switched, SRTP encoding fails
  • probably caused by ssrc switch
• remote view sent from Android falls behind and then catches up every now and then.
  • doesn't seem to fall far behind that often anymore, might have been inet issues. there's still a pretty big delay though
• when Firefox initiates the call it doesn't receive any remote video (probably sdpfail by me, will look into it)
  • fixed in _android, native call: added workaround for Firefox's packetization-mode=1_
• no simple way to get rid of black borders on the video, will look into this as well
  • still looking into this

[stefanalund]

Seems to work fine, I guess we could land this.

[b95505017]

Hope Android example could work with http://demo.openwebrtc.io too.
Now it just open a MediaSource.
Also, does OpenWebRTC could use hardware decoder/encoder with Android? Which is MediaCodec API.

[stefanalund]

I was running the app last week towards demo.openwebrtc.io with audio/video working both ways. Is that not the case for you?

I'll let someone else answer about HW acceleration on Android. We have it working on iOS 8 at least.

[superdump]

From what @sdroege tells me, we can use MediaCodec for hardware decoding but for encoding the API is very under-specified with respect to what video format the encoder wants and how frames are stored in memory which makes it not possible to support any device generically. We don't understand why the MediaCodec API is defined so that it can't be used for hardware encoding without device specific guesswork.

[sdroege]

Someone will have to check if anything has changed there, last time I looked was about a year ago. But if something changed that would of course only mean that it will work on even newer devices.

It seems like some more encoder related API was added in API level 21 (Lollipop!)

[sdroege]

Doesn't look like something obvious has changed, a lot of encoder related new API but nothing to query which color format with which strides/plane offsets the encoder actually wants.

[b95505017]

@stefanalund I could only saw local preview of camera in Android native example, there is no session id could connect with demo site.

[stefanalund]

You put in the same session ID yourself in the app and for example on the web. When two clients has connected to the same session ID the call button should get enabled.

[superdump]

Merged.

[jrosssavant]

@Rugvip Thanks for building this and getting it merged. I'm trying to get this to run correctly on my Nexus 5 running Android 5.0 against demo.openwebrtc.io:38080 and then I will re-write the signaling piece to get it working with my Janus gateway.

But I'm having trouble getting it to work as-is. I built the openwebrtc.jar per the non-cerbero instructions, dropped it into the NativeCall example, and the app built and deployed without trouble. I'm able to input the session ID in the Android app and also in the browser and click "join", and the call buttons become available, but when I click "Call" from either client, video never makes it from one client to the other. If I click call on the Android client, I see this error:

01-22 16:49:44.607  10621-10621/com.ericsson.research.owr.examples.nativecall D/NativeCallExampleActivity﹕ onCallClicked
01-22 16:49:44.611  10621-10621/com.ericsson.research.owr.examples.nativecall I/g_log﹕ owr_payload_set_property: assertion `pt == -1 || pt >= 96' failed
01-22 16:49:44.612  10621-10621/com.ericsson.research.owr.examples.nativecall I/g_log﹕ owr_payload_set_property: assertion `pt == -1 || pt >= 96' failed

If I click call from the browser(Firefox), I see the Android device gathering remote candidates, but it never seems to close the loop.

01-22 16:52:31.988  11393-11393/com.ericsson.research.owr.examples.nativecall E/RemoteMediaDescription﹕ adding remote candidate: {"candidate":"candidate:0 1 UDP 2130379007 10.11.204.15 53681 typ host","sdpMid":"","sdpMLineIndex":0,"candidateDescription":{"foundation":"0","componentId":1,"transport":"UDP","priority":2130379007,"address":"10.11.204.15","port":53681,"type":"host"}}
01-22 16:52:31.991  11393-11393/com.ericsson.research.owr.examples.nativecall E/RemoteMediaDescription﹕ adding remote candidate: {"candidate":"candidate:0 2 UDP 2130379006 10.11.204.15 60752 typ host","sdpMid":"","sdpMLineIndex":0,"candidateDescription":{"foundation":"0","componentId":2,"transport":"UDP","priority":2130379006,"address":"10.11.204.15","port":60752,"type":"host"}}
...

The Android device and desktop machine are on the same WiFi network. Do any of these errors look familiar? Do you have any suggestions for what to look for to identify this issue, or how to set up my test for a greater chance of success?

[stefanalund]

hmm, there seems to be something wrong with demo.openwebrtc.io, I also have problems connecting two devices myself. We will look at it and get back to you!

[jrosssavant]

I'm able to use demo.openwebrtc.io just fine with Bowser on iOS. I'm digging into the Android app, and it seems to never generate local candidates - MediaSessionEventListener.onNewCandidate is never called. I'll keep digging and let you know if I make progress.

[jrosssavant]

After a bit more testing I find that the NativeCall app will generate local candidates and get two-way video set up, but I typically have to let the app sit and wait for about 20 minutes after pressing the Call button before the local candidates are generated and the two-way video starts. @Rugvip have you experienced this? Do you have any suggestions for how to speed up the generation of local candidates?

[Rugvip]

It sounds like you've run into this: EricssonResearch/openwebrtc#126. Try applying the workaround mentioned in the issue and see if that helps.

I'm building OpenWebRTC for Android using cerbero atm, I'll report back with the result.

[jrosssavant]

Ah, thanks for bringing this to my attention. I'll give the workaround a try, and also try the cerbero build for Android per the instructions.

[jrosssavant]

The workaround you provided worked perfectly, thanks. I tried the Android cerbero build but encountered the error below. How the did the cerbero build go for you?

$ ./cerbero-uninstalled -c config/cross-android-armv7.cbc package openwebrtc
...
[(28/35) icu -> configure ]
...
checking how to run the C preprocessor... arm-linux-androideabi-cpp
configure: error: in `/Users/joseph.ross/cerbero/sources/android_armv7/icu/source/host':
configure: error: C preprocessor "arm-linux-androideabi-cpp" fails sanity check
See `config.log' for more details
Running command 'autoreconf -f -i'
Running command '../configure --prefix /Users/joseph.ross/cerbero/dist/darwin_x86_64 --libdir /Users/joseph.ross/cerbero/dist/darwin_x86_64/lib --enable-shared --enable-static         --disable-renaming --disable-samples --disable-debug         --disable-layout --disable-extras --disable-maintainer-mode  --disable-silent-rules  --enable-introspection  --build=x86_64-apple-darwin12'

Recipe 'icu' failed at the build step 'configure'

[Rugvip]

Got the same error, haven't had time to do any thorough investigation yet. The issue is that icu needs to be built for the host platform before being cross compiled, and at least the OS X build fails pretty much instantly.