Error streaming from Argus 3 Pro #38
Labels
bug
Something isn't working
Comments
|
Know issue I'm working on. It's the one where the camera seems to send us data not related to the stream. I think I'll have to add error recovery to the stream and instead advance the bytes one at a time until I can find the next good packet. |
|
Ok this should now be addressed in |
|
Now I am seeing warning messages. But I expect this is what you intend... My VLC client sometimes has trouble connecting to the neolink stream. I have not established a pattern yet, but it is not a new problem. |
|
Yea this is the next issue I'm trying to address only it seems a bit random. Sometimes the packets aren't being pulled off from the stream and fed into gsteamer fast enough and I'm not sure why. |
|
Have you had the |
|
I am not running rtsp much now that image is working. I am not able to test with rtsp until next week. I think I occasionally got it in the past with image but not recently. Is there any way we can get RC codes other than 0 or 1? I have image running every 10 minutes. Occasionally it fails in which case I log the RC and Here is example of failure (recent one where now fails if port specified in config) followed by success after I removed that. You can see logger text ends at "Caused by:" David |
|
Are you grepping this? Because there should be another line underneath caused by |
|
It looks like I need to replace newline characters with a space to get it all into the log.
|
|
The caused by errors don't really help. I added them to see what the chain of errors were but learned that they are just the same information as the error that makes it but broken up into smaller chunks. Ive been considering removing them so that the the logs are on one line anyways. What rc do you want? Some sort of distinction between non retryable? I think currently we consider everything retryable except connection and login errors. |
It turns out my problem is that the syslogd I am using (part of busybox) is truncating messages at ~250 characters. I need to build busybox with larger buffer size to accommodate longer messages. |
|
I have been seeing the
I've been getting this error after a few hours of streaming with the 0.5.5 release on my arm64 raspberrypi. It seems to happen eventually every time (I've ended up setting it to reboot every 6 hours to keep it 'alive' beyond this). I can grab logs or whatever, or try new builds, if you can give me a few details if what you'd like me to do/try. |
|
I'm working on a full rework of the gstreamer part of the code. This happens when the gstreamer part of the code isn't pulling the packets from the camera quickly enough so hopefully I can fix that with this. The buffer is currently set to 100 messages. Where each message contains at maximum and IFrame. Im not sure why it's not pulling that off the queue quickly enough at times. I could increase the size of the buffer but I need to make sure that won't just hide the issue for later. |
|
I agree with the problem, with Argus 3 PRO Additionally what are the requirements to run Neolink on a VM with Ubuntu and 5 Reolink Argus 3 PRO? |
|
I've never really bench marked the code. So not sure about the requirements. It usually runs ok on my rpi 3b though. |
|
With the latest release these errors are gone. |
|
@QuantumEntangledAndy After some time these warnings: and then again.. |
|
I think I have mostly addressed this deadline has elapsed in the latest neolink. I haven't had it in awhile since the new buffer model but perhaps you can confirm? |
|
Actually I still see errors, it disconnects, the video stream is better but still jerky. In the video stream I have the date and time with the seconds, sometimes the seconds go backwards, sometimes they jump forward several seconds... Then I launch a test with GST_DEBUG=3 and I place the result. |
|
|
|
Is this VLC jerkeyness? In vlc with time jumping please see #68. Still working on that bit |
|
@dkerr64 could you test with the latest build? I've reworked the buffer code to help imporve the stream and i think your original issue with the unrecognized bcmedia has been partially addressed even if the underlying issue still needs some work. If you don't get back within a week I'll probably close this issue |
|
@QuantumEntangledAndy sorry I have not commented earlier on your more recent updates. I've been using the image capture feature you added for me and it is working really well. For the RTSP stream, I just synced and built the code as it exists today in master. Things have hugely improved, I especially like that the video stream starts almost instantly. But I still see issues. I am testing with a Reolink Argus 3 pro and VLC as the client (which I am running on macOS). Without pause... If I wait for "buffers prepared" then things are pretty good, but choppy. Definitely not as smooth as the native iOS client. And after a few moments a stream of error messages starts to log (but the VLC is still showing live video)... as I write this it has been running for 20 minutes which it would never do before. With pause... Similar pattern to above. I let Neolink startup and go into pause state then open up the VLC stream. It starts up very quickly and again I get the error messages. But in this case it stops streaming after about 10-15 seconds with no message from Neolink, but the VLC window just closes.... Neolink does not print out any message that it is pausing the stream. But if I look at the network traffic it does seem to have stoped (below you can see the ~20 minute long first test, a short break, then "pause" test which is very brief. |
|
@QuantumEntangledAndy if I run the tests with a hardwired camera then without pause it is working well, no error messages... That last error came when I shutdown the VLC client. But with pause, I have the same problem as the WiFi attached Argus 3 pro... error messages and then the VLC window closes down. |
|
Oh yes we added that message while testing for buffer issues where it would drain. After pause the buffer should drain out since no new frames are comming in I'll see if I can add some way to handle that situation better |
|
Ok this is being addressed in #82 |
|
Ok test ready could you see if this build helps for you? Docker is here if you use that |
|
I recommend you test without motion pause first just client. Also be aware that during a pause VLC is likely to drop the connection for inactivity, use ffplay or something to handle that better |
|
@QuantumEntangledAndy Looking good. Not perfect, but a lot better... two runs here, first using the wired camera, second with the WiFi attached Argus 3 pro... So you can see that I get Buffer exhausted messages, more so with the WiFi attached. But the VLC client stays up the whole time, it does not shut down. When it resumes the audio seems to start first, then video follows. It takes about 10 seconds to fully recover (video starts) from buffer exhausted. I also noticed at one point where video seemed to play faster than realtime (that is the seconds counter on the video ticks up faster than every second). Possibly because the video is trying to catch up with the audio which started playing before the video did. You can see it takes a few attempts to login to the camera, but that is not new. I see this a lot but have not bothered to try and track down why. The Gstreamer error seems to be a side affect of me closing the VLC client, but it does not seem to impact the server, can reconnect. David. |
|
I see, I am trying to track down why the buffer is being exhausted and what can be done with it. From my tests the camera is only sending 46s of video in a real time period of 63s. Theres just not enough frames comming in to be live |
|
Oddly the buffer is ok when I connect via relay but it drains quickly over TCP. TCP should be the more stable direct connection... It's really weird |
|
Can you try again with the latest build in that PR. Found a possible bug in the last one related to the time re-stamping after a pause that might cause the buffer to empty. |
|
@QuantumEntangledAndy looking good. Occasional buffer exhausted message but in general is running well, I only saw it once in a 5 minute run. I only tried with the Argus 3 pro as it is late here. I notice that the playback is delayed by almost 20 seconds from real time, maybe due to the buffering. The native client has under 2 second delay. But congrats on getting it to work way more reliably. |
|
I can try reducing the size of the buffer that should reduce the latency, just can be difficult to find the middle ground between latency and enough buffer for camera issues. I might try some std dev caluclations to find out the variance in the buffer size under normal operations |
|
The log… |
Definitely worth trying and maybe consider making it a config setting… some cameras may be better than others and work well with a smaller buffer. |
|
Still buffer exhausted. Can you run with debug logging? RUST_LOG=neolink=debug ~/github/neolink/target/release/neolink rtsp --config=/home/david/neolink.toml |
|
In the debug logs you can also see |
|
Doing this all from an iPad so my Terminal scroll back is not very big, but I captured this. I see an over full message, perhaps the buffer is not emptying fast enough? |
|
Hmm something is up it goes from overfull to nothing too quickly. I'll check my code again |
👍 |
|
Could you check with latest master? Hopefully I got everything sorted out now |
|
@QuantumEntangledAndy looking good. I do get the buffer exhausted message and then a new message that says to try and reduce maximum bitrate. So I did that, from 3Kbps to 2Kbps and at the lower rate things are working well. It feels odd that 3Kbps bitrate would saturate neolink's buffer... it seems to be restricted to the Argus 3 pro as my wired camera RLC-520 is also set to 3Kbps and does not have this problem. |
|
Also... while in pause state, what handshaking, if any, is taking place between neolink and the camera? The other day after I had finished testing, I noticed sometime later that the battery was draining at a much higher pace than normal. This was even after shutting down neolink. Now it may be completely unrelated and coincidental (I notice higher drain when there is a lot of movement in the frame triggering the cameras motion detection) but I wanted to ask and check what power optimizations you have in place. Thanks. |
|
Maybe we need another issue to open for this, but during pause there is still an awful lot of traffic which is definitely draining the camera's battery... 177 is the camera, 185 is the host running neolink. |
|
I'm aware. Pause is not a real pause because of the need to stay connected to monitor the motion alerts. There's a PR open to address this atm #85 and I'm working on something for client pause too. |
This does seem consistent with the native iOS client. Pause streaming from the camera, or go to settings, or even go to view another camera and those 28-byte length UDP packets just keep coming. The only way to stop them is to send the iOS client to background. But then bringing it back to foreground requires reconnecting (not sure if it has to re-discover as well, probably does). Reconnecting slows down resumption of streaming, but not by a whole lot. Looks like there is a trade off here for the pause mode... keep the connection alive so streaming resumes promptly at expense of power drain, or disconnect and re-connect at expense of few seconds delay in resuming the stream. Opportunity for yet-another user config option! |
|
The macOS official client does, however, stop the 28-byte handshakes if you switch to another camera, and then reconnects on returning to the Argus 3 pro. Delay does not seem to bad. Might be worth sniffing the traffic to see if it really logs out or not. It does something cleanly, because network traffic is very different when you just close down the client mid-streaming... lots of port unreachable until the camera figures out no one is listening anymore. |
|
Yes I'm going to see if I can work something up for it when I have time to cycle into this not pause for a certain time then a full disconnect after some time period. Maybe can check battery too. |
|
Will close this as the original issue seems to be addressed. If you want to discuss the pause thing please open a new issue. It gets hard to keep track of issues if they are all grouped so seperate issues are preffered |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Getting this recoverable error.
The text was updated successfully, but these errors were encountered: