neolink + powered camera - high CPU usage after some hours #61
Comments
|
Interesting. Are you familiar with how to compile with rust and profiling? Running a flamegraph might help get to the bottom of this. I'm currently tracking an issue with some of the threads locking and not being able to send further stream data. Maybe there's a relation |
|
If you can give me some pointers, I will manage to do it. |
|
Ok since you seem good with docker we will try it in there. Hopefully profiling works in a container. Start with this base image docker.io/rust:slim-buster It contains the rust tool chain Install the following packages apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
libgstrtspserver-1.0-dev \
libgstreamer1.0-dev \
libgtk2.0-dev \
libglib2.0-dev \
gstreamer1.0-tools \
gstreamer1.0-x \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
git Clone the repo git clone https://github.com/QuantumEntangledAndy/neolinkNow let's build it to make sure things work cd neolink
cargo buildIf that works let's install the flamegraph. (This step might need work to install other things I'm not sure) cargo install flamegraphNow we run neolink in the flamegraph program cargo flamegraph --bin neolink -- rtsp --config=yourconfig.tomlIt should now build and profile to create a flamegraph called flamegraph.svg in the cwd. That graph should let us know what's going on. But I've never run a flame graph for more than 30 mins so not sure how it will behave long term |
|
Will come back as soon as possible (might be more towards mid of the week). Thanks! |
|
Woulnt be of great help in coding but observed same thing with same config all on proxmox |
|
I'm not sure what you mean by |
|
frigate uses image processing for motion / person detection, with or without hardware acceleration. on my system i have a proxmox with 2 LXC containers :
when motion detection is enabled on frigate, it consumes al ot CPU and i suppose that image processing is slowed, and frames are skipped. Strangely (?) at the same time, even if neolink is inside a separated container it suddenly overload CPU usage, leading to errors / disconnexion, same type of errors reported here. if frigate detection is turned off, CPU charge keeps low and there's no problem with neolink. However using neolink without the ability to post process the rtsp is quite useless. it looks like that neolink overloads when frigate is not able anymore to process the rtsp flow issuing from neolink such reaction is not observed on all other containers i have. They are slowed because of frigate CPU charge on the whole proxmox but they do not overaload themselves. |
|
Ah so your client is not pulling the frames and the buffer overflows. I am actually working on fixing the buffers at the moment by listening to the need/enough data signals from gatreamer. Maybe that will help. |
|
Could you test with this build https://github.com/QuantumEntangledAndy/neolink/actions/runs/4721352049 it has various imporvments to try to address buffers and cpu usage |
Would love to test it as well as with @jsapede running frigate and see a CPU usage of over 167% on xeon v2 8core |
|
@jsapede mind sharing your frigate config? Can't get main stream to work on my B800 |
|
I believe that specific build was already merged into master. Are you seeing issues with current master? |
|
As you Can sée i observed a cpu overload in saturday. Unfortunately i didnt kept the logs and restarted the lxc but part of the problem i had was on frigate side because of permanent transcodig / restreaming. Actually, i turned off this feature and it works a lot better. However there was a CPU overloard on this saturday, dunno the origin. Strangely neolink even at 100% CPU, provided stream to the frigate without any problem. |
|
I've got some more buffer fixes coming soon. Just needs a final set of tests. Maybe it will help more. |
|
Those fixes I mentioned are in perhaps you could try the latest build and see if it helps with frigate |
|
i have the same issue with 0.5.8. i just try to update to v0.5.9 or v0.5.10 but my stream not working on this version. main and sub are no available. (frigate or vlc) i downgrade to 0.5.8 and all work fine again, exept the cpu overload after time (2-3day in my case, i need to reboot) |
|
What exactly do you see with the error? Perhaps you could open a seperate issue and we could discuss it in detail. |
|
Also triple backticks for logs please ```
Log log
Log log
``` |
|
Also, also if you can please try the build from #82 it has some bug fixes in it |
|
Hey Andy, My apologies but I haven't been able to have the time to do the tests you suggested. Will try to do them this weekend. Nevertheless, one of my biggest gripes, still solely using a non-battery camera, was Frigate ffmep memory and CPU run away... I simply had to limit each container available CPU and memory, to prevent overheating of my NUCs CPU and being unable to also even access the server remotely. Yesterday I enabled the second neolink container I have, that I just use for the battery powered camera, in order to test the new commits/versions. The picture below shows the impact, even of a limited 0.5 CPU (NUC has a 4 core CPU) and 1GB of RAM per container. Please note that I don't have a way to show which neolink process is actually the responsible for each spike (Zabbix is doing a summary of both processes CPU and memory consumption).
On the battery powered camera container, I can see a constant warning about max bitrate. Frigate logs show a slew of errors on trying to read the RTSP buffer from the camera: As for the non-battery powered camera container, there are very few errors/logs: I am using the fluent subStream in Frigate Go2RTC server. I will try to conduct the performance tests you asked previously and will also try to get some logs with debug enabled. Cheers, |
|
Does your neolink config use both main and sub stream? I ask because buffer exhaustion usually happens on main stream and rarely on sub stream. But at the moment unless you specify stream = subStream it will always pull from both. Also does the log about frigate crashing stop after the message "buffers prepared" from neolink. Before the buffers are prepared the video only lasts for 500 frames then stops which causes those error messages in my frigate too but can be safely ignored once the buffer is ready. |
Config:
|
|
The PTS warning is interesting. Currently I set the DTS instead but I can set the PTS if ffmpeg prefers it. |
|
What's your frigate config? Maybe I can replicate it and see if I can get these ffmpeg message too. |
|
Here you go! Thanks!! |
|
Fixed :) Never had any CPU perf or memory run away behaviors. |
|
Oh good. I'm still getting a small memory increase over time that looking at allocations in the profiler seems to suggest that the gstreamer buffer is growing too big so I see what I can with that in the next release |
Setup
neolink docker logs:
Frigate docker logs
I've enabled RUST_LOG = debug and am awaiting to see if it reproduces.
The text was updated successfully, but these errors were encountered: