Fix pause buffers

[QuantumEntangledAndy]

Some fixes to the buffer after a pause to keep it filled so we can play smoothly

[Veuchez]

For me it is ingiardabile both in substream and in mainstream.
Continuous blocks, video artifacts and image freez.
Attached the log
log.txt

[QuantumEntangledAndy]

Still not sure what's up with your camera. You had the error in steam message which usually means the cameras internal buffer filled up (not neolink but the camera itself).

Could you provide some specs on your model. What's the resolution fps, os, docker/native etc. if I get others with the same issue I'll see if I can find some common ground.

[Veuchez]

Reolink Argus 3 Pro
Ubuntu 22.04 native
That of my camera? But how come from the official app I see it perfectly then?
Mainstream 2560x1440, 15fps, bitrate 3072kbps
Substream 896x512, 15fps, bitrate 672kbps

[QuantumEntangledAndy]

P.s. if you set RUST_LOG to neolink=debug then it will only show debug messages for neolink which is a bit easier to read

[Veuchez]

neolink=debug ./neolink rtsp --config=neolink.toml > log.txt 2>&1
log.txt

[QuantumEntangledAndy]

That of my camera? But how come from the official app I see it perfectly then?

Many possible reasons. Official client dosent need to worry about internal buffers or queues or timestamps or anything needed for rtsp expected by clients it just displays last frame which can reduce artifacts.

Additionally they seem to have a slightly non standard h264 stream (perhaps some approximations in their hardware encoder) that causes some issues like invalid NAL units. Since Reolink writes both decoder and encoder they can accommodate for this. Where as neolink does nothing but pass it on and let the client deal with it.

I also don't know what all of the parameters in the packets do. For example there's an integer value in the acknowledgment packet that seems to update randomly ever 1s. It could be some sort of latency that is used to negotiate dropping frames.

This is a reverse engineering project everything has to be learned by testing live there's no nice docs or standard reference to look up. If you think you can do better please make your own contribution. I cannot test and tinker with cameras I don't own.

[QuantumEntangledAndy]

neolink=debug ./neolink rtsp --config=neolink.toml > log.txt 2>&1

log.txt

Like this:

RUST_LOG=neolink=debug ./neolink rtsp --config=neolink.toml > log.txt 2>&1

[Veuchez]

I really appreciate your work, I'm not here to criticize in any way.
I was just trying to figure out how it could be an internal buffer problem of the camera if on the official app it works fine.
I understand very well the difficulties of this project and I would not know where to start myself, the only way I can help you and be a tester.

[Veuchez]

Here is a short log, if you need longer I'll do it again.
log.txt

[QuantumEntangledAndy]

I just think there's an internal buffer that's filling and that's why we get that error.

Evidence:

• this error starts when a video frame is encountered when one is not expected
• the error occurs more often for larger streams
• after this error the next time stamp on the frame jumps by a fixed time
  • this time depends on the camera model
  • for my E1 it is 15s
  • for my Argus2e it is 9s

This behaviour is akin to a fixed size ring buffer.

• When the camera fills the ring buffer it starts overwriting at the beginning. (Which is why we get a packet mid way through)
• the time stamp jumps (because the ring buffer has overwritten what would be our next frame with the newest one)
• the amount of time jump depends upon the size of the video bigger streams have smaller time jumps because it takes less time to fill the buffer (this is why the E1 jumps 15 and the eco 9s, this also tells us that the buffer is fixed size)

[QuantumEntangledAndy]

Why it fills I haven't quite figured out yet. The camera (and neolink) have to maintain a buffer of packets for UDP reassembly. What this means is camera sends a packet of id 100 then 101 etc when I get the packet I acknowledge it by saying yes I got 100. After which the camera should delete it from its buffer.

We both need to maintain this buffer because UDP is lossy. I might get packet 105 before packets 103 and 104. So in this case I tell the camera that I have packets 100,102 and 105 and request a resend of 103 and 104.

Perhaps for example the camera is not accepting my acknowledge packets in the way it expects and it's over filling it's buffer because of this. I really can't tell since I don't observer this error often enough.

I do know that I get this error more often when there's a congested network for example when the family is streaming a lot of video.

[Veuchez]

If I can help you understand better with other tests I am available... Currently I am alone in the house and I am using the internet only to answer you, no active streaming and no other use of the network.
The congested network should not be my case, at least for now.

[QuantumEntangledAndy]

One thing we could try is a wireshark dump of the packets. Then I could check if the camera is sending the right packets after an acknowledgement or sending old packets or even tell if neolink has sent the wrong acknowledgments.

[QuantumEntangledAndy]

Buts it's kinda a different issue and not related to this PR

[Veuchez]

If you explain to me how to do it I will do it ...

[QuantumEntangledAndy]

Something like this would work I think though might need tinkering as I havent done this in awhile:

sudo apt-get install tcpdump
sudo tcpdump host CAM_IP_ADDRESS -w capture.pcap

Should capture all packets to and from the camera ip and write them into the capture.pcap file. Set your camea password to a dummy one first or change it afterwards because reolinks password encryption is not too strong and it can likely be cracked from these packets.

Press Cntl-C when done to stop the dump

[Veuchez]

I launch this command with neolink working right?

[QuantumEntangledAndy]

Launch in another terminal before neolink starts, start neolink, let neolink run for a while, stop neolink, stop tcpdump

[mmarius623]

I‘m getting the following error since the lastest Build: (Argus 3 pro)

neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.
[2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.
[2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.
[2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.
[2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.
[2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.

is this error related to this?

[Veuchez]

Here are the logs!
https://drive.google.com/file/d/1_Cuey18Hu5hgca98GdYfJo-njrQS7mZ6/view?usp=sharing

[QuantumEntangledAndy]

I‘m getting the following error since the lastest Build: (Argus 3 pro)

neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera. [2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera. [2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera. [2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera. [2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera. [2023-05-15T12:03:34Z ERROR neolink::rtsp::gst::sender] Buffer exhausted. Not enough data from Camera.

is this error related to this?

Possibly I am tracking a few issue with the latest release some I am going to mark them as pre-release until I get them sorted

[Veuchez]

Definitely much better but it still freezes. Here's the log.
log.txt
long_log.txt

[QuantumEntangledAndy]

Thanks. I'm going to get some sleep now and check it out in the morning. Hopefully I can make sense of it.

[Veuchez]

Here's the log.
substream.txt
mainstream.txt
main+sub.txt

[QuantumEntangledAndy]

So from your logs, substream works fine but mainStream does not

[QuantumEntangledAndy]

Just did some math on your logs and in 60s of real time only 50s of video frame is recieved. It's not comming in fast enough to be live

[Veuchez]

Unfortunately at the moment I can't tell you if I see a jerky or fluid stream because I'm away from home... for now only the logs...

[QuantumEntangledAndy]

Could you try perhaps putting your cameras bitrate down to 1536 (using the offical app)

[Veuchez]

main1536.txt

[QuantumEntangledAndy]

The log suggests that it works like that with the lower bitrate

[Veuchez]

Is this a fixable problem on neolink?

[QuantumEntangledAndy]

Have you considered this is a limitation of the network. Unlike the official client where they have a direct connection here we act as a relay we both download and re-upload the data increasing the load on the network.

[Veuchez]

Yes, I thought about it, that's why I asked if it can be solved with neolink ... at the same time, however, I think, I have a ubiquiti u6-lr 2 meters from the camera connected in 5ghz ... it is plausible that the network can't do it ?

[QuantumEntangledAndy]

Things you could try is putting the neolink on Ethernet to the switch. Not sure how easy that would be for you to test. If I can find the time I'll try and look at the timestamps in the tcp dump you sent. See how many packet replays and losses there are etc.

[Veuchez]

The server where Neolink runs is connected via ethernet, only the cameras are wifi.

[QuantumEntangledAndy]

I see then not much else I can recommend to test this

[QuantumEntangledAndy]

Ok I checked your tcpdump and in 2m of real time only 1m53s of video has arrived. This is what arrives on the socket even before neolink attempts ANY processing or even just recieving it. It suggests that the nework can't quite handle the load. It's not the sort of thing I can do to address.

I might be able to setup some sort of dynamic bitrate to adjsut the bitrate down if the load gets too high but that would be in another PR

[QuantumEntangledAndy]

Ok so one last thing I have tried is to increase the MTU, it should give us about 300 extra bytes per packet but not sure if it will be enough,

[QuantumEntangledAndy]

The MTU of a network is usually 1500. Some of this data is used for the IP header so you get about 1450 of usable data. After that we have about 100 bytes for the BCUdp data and the BC header leaving about 1350 bytes to pack the video into. This is the value reolink uses. Before I was using a more conservative estimate of 1050 bytes, so by changing this we get an extra 300bytes per packet

[QuantumEntangledAndy]

I did some testing with the official app and my network cannot stream 2048kbps at 1080p(15fps) without pauses and buffering. Seems to be fine for maybe 15s then it buffers for 20s then plays for another 15 etc with time jumps in stamps.

This was tested lower that your 1440p 15fps, bitrate 3072kbps

[QuantumEntangledAndy]

Further tests with 1536kbps also have occasional pauses and time jumps (but not as much as 2048kbps) (official reolink client)

[Veuchez]

I saw that if I only do the mainstream by removing the stream I can get to 2048kbps without problems.
The question that arises now is this, if I connect another camera to the same access point, the problem comes back since it is a bandwidth problem, right?
Because I have 5 identical cameras and 2 access points, but I haven't done the test yet...

[QuantumEntangledAndy]

Well if you use client pause then you can stream them one at a time. It's the reason I wanted to add that feature.

Currently though client and motion pause turn on both sub and main so My next project will probably be getting it to know when your streaming sub and main stream then only stream the right one.