About audio in html5 mobile

[moisesjpelaez]

I see there's a limitation in Aura.hx with mobile browsers. Is it possible to do the effects support manually or does it completely depend on Kha?

I'm also interested to contribute to this library once I get more experience with Armory and Kha, is there any recommended read where I can learn more about audio implementation? I'd like to understand better this library's structure before contributing with code.

[MoritzBrueckner]

Hi, sorry for my late answer. I guess you are referring to the following specific comment?

aura/Sources/aura/Aura.hx

Lines 73 to 78 in a14a1c9

	if (kha.SystemImpl.mobile) {
	// kha.js.MobileWebAudio doesn't support a custom audio callback, so
	// manually synchronize all tracks here (note that because of this
	// limitation there are no insert effects supported for mobile audio)
	kha.Scheduler.addTimeTask(masterChannel.getMixChannel().synchronize, 0, 1/60);
	}

What I mean with "custom audio callback" in that comment is a custom kha.audio2.Audio.audioCallback function that is (indirectly) called by the audio driver to request more audio samples from the application. So whenever the driver needs more samples for the audio output device, it asks Aura to write them into a buffer in the audio callback.

Unfortunately, the Kha implementation for mobile browsers does not use such a callback, so Aura also can't use it. Instead, the sounds are played via some other browser API (also via a Kha detour) and Aura only regularly updates some properties of the sounds which would otherwise happen in the audio callback (that's what this line is for). I believe you can find the Kha implementation for the mobile html5 audio here. The mobile audio channel is created here, and it looks like something similar might be missing in Aura.createUncompBufferChannel() – it has been a while since I last tested mobile html5 audio so I can't say this for sure...

I can't really tell you the exact reason(s) why Kha does not play audio the same way on mobile browsers as on desktop browsers, I'm not even sure whether I knew it some time ago or whether I never really investigated it. There seems to be a bit of context here.

Nonetheless, I believe there are some potential solutions for your problem, but they are not easy to implement and they have some big drawbacks:

• Try to use a ScriptProcessorNode to implement the audio callback. I believe this is also used for the non-html5 Kha implementation. Unfortunately, I don't know how well this is supported on mobile browsers, so using this might cause the same problems that Kha worked around by providing mobile-specific audio code... And even worse: using ScriptProcessorNodes is deprecated...
• Use AudioWorklets instead. This is the modern replacement of ScriptProcessorNode that works in a separate audio thread, but I also can't tell you how well this is supported on mobile browsers. Sooner or later, Aura should use audio worklets for html5 audio on both mobile and desktop devices, but it would require big changes to Aura (or Kha, if Kha was using it) because the audio worklet code needs to reside in a different JavaScript file to not create a bunch of thread-safety problems. To achieve this, we would need to compile the Haxe code into two files, which could be complicated...

I'm also interested to contribute to this library once I get more experience with Armory and Kha, is there any recommended read where I can learn more about audio implementation? I'd like to understand better this library's structure before contributing with code.

Very cool! There is almost no documentation about the implementation at the moment (other than some code comments and maybe some hints in the wiki), but I'm happy to explain it or to write some documentation when I get time. If you have specific questions, feel free to open more discussions here :)

Some general notes about the implementation:

• You should be able to find pretty much all playback-related code by following the audio callback implementation
• Memory allocation in the audio thread code (all code indirectly called by the audio callback) is a no-go (this can be tricky sometimes). On some targets, garbage collection is even turned off in those parts of the code
• Channels have a handle that resides in the main thread (if the application uses an audio thread) and an actual implementation in the audio thread that communicate via messages. This still needs to be implemented for DSP effects, and the message system probably also isn't completely thread-safe at the moment...
• The package structure is pretty chaotic at the moment, especially aura.types (which does not make much sense) and aura.utils.