[Support]: Frigate causes my system to crash and reboot #6469
Comments
|
I don't know whether we have the same issue but I also get lots of system crashes with version 0.12 but only when I connect to cameras that are restreamed through go2rtc. I have to use restream because I have a couple of Nest cameras that change their url every 5 mins. I'm now running 2 frigate instances, my direct connection cameras running 0.12 and the restreamed nest cameras running on 0.11. Not ideal because I really like 0.12 but I've gone almost 2 weeks without a system crash like this. It doesn't appear to be the go2rtc part because I'm using the inbuilt go2rtc in 0.12 to do the restreaming for the 0.11 instance. Bit of a ramble but I hope it makes sense. |
|
@Inrego I too am having issues recently since v11 but my system is running HA Core on firstly an Intel NUC Core i7 5th Gen and now a Beelink SEi12 with a 12th Gen Intel Core i5. Full story here https://community.home-assistant.io/t/catastrophic-core-failure-after-add-on-crash-advice-needed-for-logging-and-debug/568668. My problem is, HA doesn't reboot it dies completely which is maddening. It's happened 3 times in the last week. For now, I've disabled the HA Frigate add-on while I await guidance from someone on how to configure logs that are relevant and persistent, because like you, the logs I can access have nothing relevant. |
|
Same Problem here. Since updating to v0.12 Homeassistant crashes and does not reboot. All system ressources seem to be OK, so no memory leak. I am having this issue since the release of V0.12 and have so far not found any solution. I hope there is a solution because V0.12 behaves much better in every other regard |
Are you consuming the restream of any cameras? If so have you tried connecting directly? |
|
I use the home assistant frigate Integration to restream the camera feeds directly from frigate I think 🤔. Would be great to keep using that since it allows for a pretty low delay stream display in Homeassistant Another interesting side effect ist that since reverting to V0.11 no clips are recorded anymore! |
|
I would suggest using 0.12 without any go2rtc configuration to see if things still crash. As is mentioned in the release notes, you can't simply downgrade because the database has been upgraded and is not compatible with 0.11. |
Actually there is no go2rtc in my config as it is not needed in my case for a pretty good experience with Homeassistant. Ok I understand for now it will have to stay at v0.11 because of it crashing the whole Server |
|
@HAuser1234 I'm just stabbing into the dark here since this isn't your issue and you haven't provided any of the information we ask for in a support request. I would recommend opening your own issue so I can suggest some things to try. |
|
Can you please offer some advice for the recommended log configuration that would remain persistent if the hardware is restarted? That way we can all open our own issue and provide the most helpful logs? |
|
The default logs from frigate will suffice if there are errors and well most likely need system logs, htop, etc to understand why the system is hanging. |
|
@Inrego I would suggest removing this from your config since it's no longer doing anything: ffmpeg:
input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000
output_args:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copyWe will start by backing things out to see what is causing the issues. I would recommend trying the following and seeing if it still crashes.
|
Frigate's logs are written to stdout and captured by the standard docker logging mechanisms. You don't need the files stored inside the container. https://stackoverflow.com/questions/36666246/docker-look-at-the-log-of-an-exited-container |
|
You could try adding a volume mount of |
@blakeblackshear in #6477 I have opened a new issue on this |
|
Mind sharing your docker run command to create the container? I have been fighting with a similar issue (#5976) without any success. I understand that the cause of my issue could be completely different from yours. Maybe you can try this. It did not fix mine, instead it makes the issue happening much sooner. But looks it fixed for that user. My guess now is that go2rtc is somehow killed by the host system by some reason. |
I've done as asked. The server rebooted within a minute after starting Frigate. Here's my new config which also crashed: |
|
What do the logs say? |
|
Not much: Worth noting: the crashes started as I am setting Frigate up on a completely new machine. I tried starting Frigate again on my old server (same version, same config), and it doesn't reboot. |
|
Just had a reboot again after about 30 minutes, so that didn't work either. |
|
So strange. With your current config, Frigate is doing basically nothing. Let's try running with the smallest possible config using a dummy camera only. I think this should work. mqtt:
enabled: False
database:
path: /db/dummy.db
cameras:
dummy:
ffmpeg:
inputs:
- path: /dev/zero
input_args: -re -s 640x480 -f rawvideo -pix_fmt rgb24 -r 1
roles:
- detect
detect:
enabled: False
height: 480
width: 640
fps: 1 |
|
It's started with this config now. Hoping to solve it, I've more than 20 other docker containers running on this machine which are disrupted by these reboots. With 10-20 more containers yet to be migrated. |
|
So far that config has been running without issues. |
|
Now let's start adding things back part by part to see what the issue is. Let's just add the detector back so the Coral initializes, but we will leave detect disabled so it doesn't do anything yet. detectors:
coral:
type: edgetpu
device: usb
mqtt:
enabled: False
database:
path: /db/dummy.db
cameras:
dummy:
ffmpeg:
inputs:
- path: /dev/zero
input_args: -re -s 640x480 -f rawvideo -pix_fmt rgb24 -r 1
roles:
- detect
detect:
enabled: False
height: 480
width: 640
fps: 1 |
|
That's been running about an hour now. It isn't necessarily an indication that it's running, because prior to posting this issue, it would sometimes run for a few hours before rebooting. |
|
Nevermind. Just had a reboot. |
|
Let's go back to the config without the detector to be sure it wasn't a fluke before adding other stuff back in. |
|
Been running fine so far. I'm just gonna try my original config with the Coral in another USB port and see what happens. If reboot, I'll go back to basic config and await further instructions |
|
20 hours so far. I'm inclined to chuck it up as a faulty USB port. I'll give it a few days and then close the issue if no more reboots. |
|
Damn, I just had another reboot. I was so sure it was fixed |
|
Can you try running with CPU detectors to see if it's a USB issue? |
|
Just had a reboot again, with the most basic config. Now I'm starting to doubt if it's even caused by Frigate at all. It just seems like too much of a coincidence that the reboots happened shortly after starting Frigate at the beginning, so maybe it accelerated an underlying issue. I'm gonna try stopping the Frigate container and unplug the Coral for a while to see if any reboots still happen. |
|
Reboot again now, without Frigate running. Sorry for wasting your time. |
|
Was this issue fixed ? |
|
@Nisbo Which CPU are you running? My troubleshooting back then led me to realizing it was due to intels new little.BIG core design where some tasks would be off-loaded to a smaller CPU core.
My CPU is a Note that this issue was not particularly related to Frigate, as you can see this issue is closed because the system crash also happened while Frigate wasn't running. I also added the same cpuset to sonarr and radarr containers. |
|
@Inrego |
|
Then I suppose your issue is unrelated to mine. |
|
I have the same problem. Raspberry Pi 4 with SSD, Coral TPU. |
|
Meanwhile I changed from my Intel NUC (with HA installed) to a Raspberry Pi 5. (Standalone, Runing in Portainer) @NikoGrub maybe you can also try a different power supply / hub or try to separate Frigate from HA (if you have enough hardware) because Coral and SSD should take a lot of power |
|
@Nisbo Thank you for the fast reply and information. I'll watch the problem for a while and think about it. |
|
Same Problem. I have a Home Assistant OS version running on Intel NUk. I installed Frigate initially and after 3 days it crashed the first time - completely frozen the whole server. After restart it crashes now every 5 to 50 mins. I use a Coral TPU. Frustrating - no info in the logs. |
No need to report about it. If it is indeed the same issue, then you can apply the fix I provided in this thread. If it's not the same, you should start a new thread. Commenting on a closed thread is very unlikely to get the attention of the devs |
Describe the problem you are having
When running frigate, my system will frequently crash and reboot.
I've recently built a new server, and copied the config from my old server to the new. I'm using a Coral TPU on USB.
There's not really anything in logs to indicate the issue as far as I can see (I've check syslog and many other places before I figured out it was related to Frigate).
Version
latest tag on Docker
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Other Linux
Install method
Docker CLI
Coral version
USB
Network connection
Wired
Camera make and model
Reolink RLC-810A
Any other information that may be helpful
The crash happened at 13:18 in the logs. As you can see, not really much to go from.
I'm unsure if the issue is due to the CPU BIG.little architecture with small (E) and big (P) cores. The CPU is an Intel Core i9-13900K
I know some other software like Sonarr had issues with this, but it just ran very unstable, it didn't crash the whole system.
But, I've seen people mention on reddit using Frigate with 12th gen CPU's without issue, and these also use P and E cores.
For now, I've shut down Frigate and now the server is running fine without reboots. I've also done some stress-testing of my CPU to check for issues under heavy load. But it just runs without issues.
I also have netdata monitoring on the server, and from there I can tell that it doesn't look like an overheating issue.
The text was updated successfully, but these errors were encountered: