audio: add optional decoding buffer

[cspiel1]

This is only for discussion so far.

       If an AEC is in the filter list aufilt, then the aubuf should be
       constant and small. A second audio buffer aubufdec should be added via
       config if network jitter is expected.

       .--------.   .-------.   .--------.   .----------.   .--------.
 |\    |        |   |       |   |        |   |  opt.    |   |        |
 | |<--| auplay |<--| aubuf |<--| aufilt |<--| aubufdec |<--| decode |<--- RTP
 |/    |        |   |       |   |        |   |          |   |        |
       '--------'   '-------'   '--------'   '----------'   '--------'

The aubufdec makes mostly sense, if it runs in adaptive mode to handle changing network jitter best. If running in adaptive mode, an extra thread might not be required.

[sreimers]

If an AEC is in the filter list aufilt, then the aubuf should be constant and small.

With aubuf you mean the auplay aubuf should constant and small for AEC or the aubufdec?

[cspiel1]

The auplay aubuf should be small and constant. Where constant is most important for an AEC.

I tested this PR already with these settings and the rem/docs/aubuf/generate_plots.sh:

audio_buffer   20-260
audio_buffer_mode      fixed
decode_buffer_mode     adaptive
decode_buffer          20-320          # ms

jitter_buffer_type     adaptive
jitter_buffer_delay    0-20

with this jitter simulation:

sudo tc qdisc add dev ifb1 root netem delay 200ms 100ms

Note: If decode_buffer_mode is "adaptive" then there is no extra thread needed for the aufilt processing pipeline. But the thread you meant is for the RTP and decode, right?

The plot looks like this.

[sreimers]

I see, can we put the AEC in a "auplay aufilt" stage?:

auplay <-- auplay aufilt <-- aubuf <-- aufilt <-- decode <-- RTP

If the AEC calculation is fast enough this should archive the best results.

But the thread you meant is for the RTP and decode, right?

Right.

[cspiel1]

I see, can we put the AEC in a "auplay aufilt" stage?:

auplay <-- auplay aufilt <-- aubuf <-- aufilt <-- decode <-- RTP

This was my first draft. When discussing this with our audio experts, we found that indeed AEC has to be very fast because the auplay thread has the strictest real time constraints in the whole decode pipeline. The AEC (e.g. our audio-core) can not guaranty this real time constraints in general.

If the AEC calculation is fast enough this should archive the best results.

But the thread you meant is for the RTP and decode, right?

Right.

The RTP+decode thread is a good idea. It does not overlap with this PR.

[sreimers]

The decode part for AEC is normally not heavy (since the most work is done in encode) but maybe depends on the implementation.

Some quick webrtc_aec aufilt measurements (20ms ptime):

20-85 microseconds (decode)
100-350 microseconds (encode)

So putting webrtc_aec decode in auplay thread should be fast enough for realtime.

By The AEC (e.g. our audio-core) you mean a custom/hardware AEC?
Can we put the aubufdec into a aufilter module?

[cspiel1]

20-85 microseconds (decode) 100-350 microseconds (encode)

I will discuss this with our audio experts. Maybe they could make similar measurements with our audio-core.

By The AEC (e.g. our audio-core) you mean a custom/hardware AEC?

Yes a custom audio-core containing AEC, noise suppression, ...

Can we put the aubufdec into a aufilter module?

Maybe. Then there would not be any change in the baresip core. I'll give it a try.

[cspiel1]

And I just realized that if an decoding aubuf is added another thread is needed for the filter list, otherwise the auplay aubuf will have the underruns. This thread can be put also in the module. I will call this module auajb for "Audio - Adaptive Jitter Buffer" with one config setting decode_buffer 20-160.

[juha-h]

Christian Spielberger writes:

And I just realized that if an decoding aubuf is added another thread is needed for the filter list, otherwise the auplay aubuf will have the underruns. This thread can be put also in the module. I will call this module `auajb` for "Audio - Adaptive Jitter Buffer" with one config setting `decode_buffer 20-160`.

After you have done all these changes, please let non-audio experts like me to know what changes are needed in current config in order to make it optional in common use cases. For example, should the new module be loaded.

[cspiel1]

This is not designed to support asynchronous write/read of audio frames through filters.

	/* Process exactly one audio-frame in reverse list order */
	for (le = rx->filtl.tail; le; le = le->prev) {
		struct aufilt_dec_st *st = le->data;

		if (st->af && st->af->dech)
			err |= st->af->dech(st, &af);
	}

For an AEC setup we need, these three threads:

• An RTP+decoding thread that pushes into an adaptive aubuf "A",
• a filter thread that contains the AEC filter, reads from "A" and pushes into a fixed aubuf "B",
• an auplay thread that reads from "B".

auplay <-- B <-- filters with AEC <-- A <-- decode <-- RTP
|  thread  |          thread          |      thread      |

[alfredh]

when using the new C11 thrd_t api, I think the ifdef is no longer needed ?

[alfredh]

is it possible to use Atomic bool here, instead of mutex ?

[cspiel1]

I think so. This mtx solution is maybe overcautious. I saw this in module ausine.

[alfredh]

le can now be declared inside the for loop (C99 style)

[alfredh]

auframe can be made const ?

[alfredh]

ssrc value 0 is also a valid SSRC. I think there should be a variable ssrc_set that can be used ?

[cspiel1]

okay, we can simplify this:

bool flush = rx->ssrc != hdr->ssrc;

because it doesn't make a difference if we flush at the beginning. The first frame is pushed to aubuf anyway.