Frequent connection drops

[fabiosci]

Hi,
I own two 9FUS with stock firmare 4.2.0*. Inside the camera is written "Y203C_MB_RT2.0 2019/008/08". I successfully flashed both with y25 firmware and everything went fine and the cameras seem to work as expected. At the moment one camera has been flashed with the 0.2.9 version and the other one with the 0.3.0.
After a while (hours or days, randomly) the cameras disconnect from wifi network (they appear not connected in my router page) and the only way to recover them is to unplug the power supply. This behaviour is not synchronized between the two cameras but occurs with both firmware versions. The 0.2.9 seems to be more stable than the 0.3.0.
Another info: I turn off wifi network during the night but in the morning the cameras usually connect normally.

[roleoroleo]

It could be a memory and/or cpu usage problem. Try to disable unnecessary services.

[fabiosci]

I think unnecessary services are already disabled. This was and still is my configuration:

• Disable cloud, RTSP, FTP, NTPD and HTTPD: on. I've just disabled FTP because at the moment I don't record on sd card (not inserted in the cam)
• Recording without cloud, ONVIF, SSH, Legacy FTP, Telnet,: off
• RTSP stream: high
• ONVIF profile: high
• MQTT: on
• Camera setting page: everything on, Detection sensitivity low
  Do you think it could be useful to reduce RTSP stream and ONVIF profile to low?
  Can it be useful to monitor the free memory from the main page?

Another detail (maybe important): I can randomly hear a sound from the cameras as if were reboot (like a click), but they continue working after this "click". Never understood meaning and reason of this random sound.

Thanks for all your work!

[roleoroleo]

Did you noticed the same behavior with the original fw?
Because my fw doesn't apply change to the wifi part.

[fabiosci]

I don't know because I flashed with your firmware 5 min after unboxing and didn't backup the original firmware (at my own risk, I know).
The strange thing is that both cameras act the same way...

[roleoroleo]

A serial log would be useful.
Are you able to do it?

[fabiosci]

do you mean I have to follow the first part of this guide: https://github.com/roleoroleo/yi-hack-MStar/wiki/Dump-your-backup-firmware ?
If not, could you please point me in the right direction?
thanks

[roleoroleo]

Yes.

[fabiosci]

here is the putty log with internet access blocked by firewall:
putty.log

I noticed a different log when internet access is allowed:
putty2.log

both finish with line '[PWM] mstar_pwm_enable', which is the last line even after some minutes.
they seem very different from logs posted in issued opened by other people... don't know if it is ok...
thank you!

[roleoroleo]

The logs are ok.
Is it possible to record the log across a disconnection?
Could you leave the log enabled for a long time?

[fabiosci]

sure!
I'm already logging and I'll stop it the next time it disconnects.
In the meanwhile I took a picture because log in this moment is continually changing and this is a difference with the one attached to the previous post

sorry for this horrible picture...

[fabiosci]

Just an update: after 3 days my cam is still online (unfortunately, I'd say...) and the log still keep changing as shown in the previous picture. On the other cam I applied the new firmware release (0.3.1). I'm monitoring the updated cam to see if anything changes

[fabiosci]

Another update: I noticed that the camera with v0.3.1 (not under log recording) worked fine for some days until I blocked its traffic from firewall. after more or less 24h it stopped working: it disconnected from wifi network and blue led started blinking

[fabiosci]

it finally stopped working under monitoring. I found the led blinking, the cam disconnected from wifi and here there is the serial log:
putty.zip
blinking led.zip

hope this can help in finding the issue
thank you

[roleoroleo]

Unfortunately the log doesn't show any problem.

[fabiosci]

ok, thank you. I'll try by disabling one by one all features and check if one of them can cause the crash

[fabiosci]

Just an update: the cam seems to stop working when it can't communicate with the cloud. I tried the following scenarios:

SCENARIO 1
Option "disable cloud" ON --> the cam stops working after a while (hours or days) (tested with firmware 0.3.3)

SCENARIO 2
Option "disable cloud" OFF AND cam with access to the internet --> the cam is still working after several days (tested with firmware 0.3.3)

SCENARIO 3
Option "disable cloud" OFF AND cam traffic blocked by router --> the cam stops working after a while (hours or days) (tested with firmware 0.3.1, I'm going to test also with 0.3.3)

[roleoroleo]

I have the same behavior regarding scenarios 2 and 3 but not for scenario 1.
My default config is "Disable cloud" ON and my cam works correctly.
Are you blocking the traffic from your router in this scenario?

[fabiosci]

Sorry but I don't remember. I thought it doesn't affect the behaviour and I didn't take note of this.
In the next days I'll try the following scenarios and report the results:
SCENARIO 1.1: Option "disable cloud" ON AND traffic blocked from the router
SCENARIO 1.2: Option "disable cloud" ON AND traffic NOT blocked from the router

[fabiosci]

ok, i've got news regarding the
SCENARIO 1.1: Option "disable cloud" ON AND traffic blocked from the router
the cam stopped working (as expected) just after less than 2 days and in the router's configuration appears not connected. I'm talking about the cam with firmware 0.3.3

regarding the SCENARIO 1.2 the cam (firmware 0.3.1) is still working

[roleoroleo]

Ok. So, when the cam can't access to internet, it stops working.
I don't know why, I will check it.

[fabiosci]

thank you very much!
if it can be useful for furhter tests, i can update the other cam from 0.3.1 to 0.3.4 and monitor its behaviour in the same conditions

[fabiosci]

another update on the following scenario:

SCENARIO 1.2: Option "disable cloud" ON AND traffic NOT blocked from the router

after weeks, the cam stopped working.

EDIT:
I've just noticed that the log.txt tends to become bigger when I block internet connection. I say so because I find in it a lot of error such as:

p2p_tnp.c(state_statistics-6242) check_login fail 0

and some blocks like:

][7/6/14:25:35:588]: p2p_tnp.c(tnp_proc-6389) PPPP_API Version: d2020f04 210.2.15.4
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6398) PPPP_NetworkDetect() ret = 0
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6399) -------------- NetInfo: -------------------
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6400) Internet Reachable : NO
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6401) P2P Server IP resolved : YES
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6402) P2P Server Hello Ack : NO
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6403) Local NAT Type :[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6408) Unknow
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6421) My Wan IP : 0.0.0.0
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6422) My Lan IP : 0.0.0.0
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6424) InitStr(MMFBJPLDIEEPKPHOOIFEPNEHHDCJFMGOHJENKLIHIJBKDABIPMIIAONIPBCLBOKOPIKPDKPFNJDPAIDKBE)
[
][7/6/14:25:40:379]: p2p_tnp.c(tnp_proc-6425) did(TNPUSAI-576296-BRCSC)

this is the output of free -m command:
Mem: total 59, used 52, free 6

In two or three hours the free memory decreased of 1mb.

I'm start suspecting that the cams work fine until they run out of memory, then crash. Could it be possible?

[roleoroleo]

Sorry for the delay in the response.

I tried again your scenario and I think your idea is correct.
There is a memory leak when the connection is blocked.
The cam goes in low memory and crashes.

[fabiosci]

just to let you know my other progresses in investigating this issue: it happens also when the cam is regularly accessing internet both from router (internet connection allowed from firewall) and from firmware (option "disable cloud" off).
this means that the connectivity is not responsible for the problem.
I also found a blog post where someone talks about your firmware and says that it works on yi home camera v1. my cam is v3 (based on the amazon link used to buy it...). i don't know if this can be the reason...

at the moment the cams are useless, hope you'll find the issue soon. If there are tests I can do to help, let me know.
Thanks for your support

[elraro]

Im having exactly the same problem @fabiosci , but with https://github.com/roleoroleo/yi-hack-Allwinner firmware

[roleoroleo]

I need more info.
Try to run this script:
#215 (comment)

I will check if there is a memory leak.

[mhensema]

I'm experiencing the issue as well on the Allwinner firmware/device. I have cloud disabled, but I do see the device making DNS lookups for api.eu.xiaoyi.com. This seems to only happen at boot. Internet access was blocked for the device, but enabled it for now to see if it makes any difference.

[fabiosci]

I run the script from #215 with nohup ./startup.sh & on both cams. Both are configured with "disable cloud" on and internet connection blocked by router. I also see a DNS lookup for api.eu.xiaoyi.com and api.github.com.
One of them stopped working during the last night, probably when wifi was off.
Please ignore the date and time in the log file attached

top-yicam1.log

[roleoroleo]

I don't know how to help you.
I begin to think a kernel driver problem.
Probably you should add a network watchdog that reboot the cam when it's disconnected from the router.

[fabiosci]

thanks anyway.
in your opinion could the kernel driver problem lead me to my issue also in case of internet allowed and cloud enabled?
i can add a watchdog. do you think that a solution like the one described here would work?
in case of firmware upgrades, would the script and crontab setting be deleted?

[roleoroleo]

in your opinion could the kernel driver problem lead me to my issue also in case of internet allowed and cloud enabled?

The problem still exists, but the effect is probably less.

i can add a watchdog. do you think that a solution like the one described here would work?

Yes, but check if all the commands exist.

in case of firmware upgrades, would the script and crontab setting be deleted?

If you add it to the sd card, it will not be deleted.

[fabiosci]

I finally found some time to spend on this issue. I prepared the script and it works (i'm trying to reset only the wifi connection rather than reboot the cam) but it seems that crond is not running. the output of ps command doesn't show crond.

/home/yi-hack/script # ps
PID   USER     TIME  COMMAND
    1 root      0:02 /init
    2 root      0:00 [kthreadd]
    3 root      0:01 [ksoftirqd/0]
    5 root      0:00 [kworker/0:0H]
    6 root      0:00 [kworker/u2:0]
    7 root      0:04 [rcu_preempt]
    8 root      0:00 [rcu_sched]
    9 root      0:00 [rcu_bh]
   10 root      0:00 [watchdog/0]
   11 root      0:00 [khelper]
   12 root      0:00 [writeback]
   13 root      0:00 [crypto]
   14 root      0:00 [bioset]
   15 root      0:00 [kblockd]
   16 root      0:00 [kworker/0:1]
   17 root      0:00 [kswapd0]
   18 root      0:00 [fsnotify_mark]
   31 root      0:00 [SCLDAZA_THREAD]
   32 root      0:00 [VIPDazaTask]
   36 root      0:00 [deferwq]
   39 root      0:00 /bin/ueventd
   40 root      0:01 [jffs2_gcd_mtd3]
   41 root      0:00 [jffs2_gcd_mtd2]
   82 root      0:00 [kworker/0:2]
   83 root      0:00 [mmcqd/0]
   94 root      0:00 [cryptodev_queue]
  130 root      0:00 [spi0]
  161 root      0:00 [cfg80211]
  177 root      0:01 [RTW_CMD_THREAD]
  180 root      0:00 ./log_server
  181 root      0:05 ./dispatch
  645 root     10:10 ./rmm
  648 root      0:01 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -Dnl80211 -iwlan0 -B
  683 root      0:00 [kworker/u2:3]
  763 root      0:00 udhcpc -i wlan0 -b -s /home/app/script/default.script -x hostname:yicam-taverna
 1079 root      0:00 httpd -p 8080 -h /home/yi-hack/www/ -c /tmp/httpd.conf
 1099 root      0:00 dropbear -R
 1104 root      0:00 ipc_multiplexer
 1119 root      0:01 dropbear -R
 1125 root      0:00 ntpd -p 192.168.188.2
 1165 root      0:00 -sh
 1315 root      0:00 ps

do I miss something?

EDIT: crond enabled

[roleoroleo]

I'm working on a new version where you can configure cron through the web interface.

[fabiosci]

I'm working on a new version where you can configure cron through the web interface.

thank you! the possibility to schedule a custom script it's a good news

[roleoroleo]

Release in 0.3.9
Let me know if it works correctly.

[fabiosci]

just upgraded.
first of all i checked if crond was running and ps gave me:
1235 root 0:00 /usr/sbin/crond -c /var/spool/cron/crontabs/

but crontab -l doesn't seems to work:

/home/yi-hack # crontab -l
crontab: can't change directory to '/crontabs': No such file or directory

the root file in /var/spool/cron/crontabs/ exists and its content is just
0 * * * * /home/yi-hack/script/clean_records.sh 5

I tried the new feature by adding in "Configurations" page the following:
*/5 * * * * root /tmp/sd/watchdog/wd_wifi.sh
then saved and rebooted. When the cam come back online, that job doesn't appear neither in Configuration page nor in root file, which is unchanged.

this is my wd_wifi.sh

#/bin/sh
now=$(date +"%Y_%m_%d")
LOGFILE=/tmp/sd/watchdog/log_$now.log

if ping -c4 -q fritz.box > /dev/null ; then
  echo '['$(date)'] Wifi OK' >> $LOGFILE
else
  echo '['$(date)'] CONNECTION KO. Reboot wifi' >> $LOGFILE
  sleep 1
  /sbin/ifconfig wlan0 down
  # Give interface time to reset before bringing back up
  sleep 10
  /sbin/ifconfig wlan0 up
  # Give WAN time to establish connection
  sleep 10
fi

and it is executable. If I manually execute it, it works as expected.

[roleoroleo]

I tested your line and it works.
Did you clear the cache of the browser?
Check il the system.conf is correctly updated and check if cron file contains the line:

cat /home/yi-hack/etc/system.conf
cat /var/spool/cron/crontabs/root

[fabiosci]

thanks, from another browser (so clean cache) it worked and both system.conf and root contains the new job (which is shown also in Configurations page after rebooting the cam).
the only issue now is that the script is not executed. I run a ping for 10 minutes and no packets has been lost (so the wifi connection hasn't been reset) and the log file hasn't been created

[roleoroleo]

Try to remove "root" from the line.

[fabiosci]

it works, thanks!
i will adapt the script, monitor the behaviour for some days and revert back to you

[cchenga]

Hi @fabiosci, I got the same issue with yi-hack-Allwinner. Could you share your solution/workaround?

[fabiosci]

Hi,
the root cause seems to be not related to the custom firmware, the only workaround i've found is to use a script to check if the connection is available or not (it's a simple ping to my router) and, if not, reset the wifi interface.
The script is scheduled to run every morning when my router turns on the wifi network.
I haven't had time to go deeper, but experienced another issues: sometimes the wifi connection drops during the day. this could be fixed with the mentioned workaround.

Before closing the ticket I'd like to investigate on the random drops by reading the logs created by the scheduled script

[fabiosci]

after several months, logs and tests now I can say that the connection is correctly established and maintained day after day.
the crontab feature in the UI allowed me to create and schedule a script to check the wifi connection and, if disconnected, reset the wifi adapter and, if necessary, reboot the cam. this solved my issue, even though it's a workaround.

in case it could help someone else, i post here my current solution:

• crontab configuration:
  */5 07-23 * * * /tmp/sd/watchdog/watchdog.sh --> to check the wifi and reboot the cam
  0 3 * * * find /tmp/sd/watchdog/logs/ -mtime +14 -delete --> to delete older log files

I put the watchdog folder into /tmp/sd to keep it after a firmware update.

This is my watchdog.sh

now=$(date +"%Y_%m_%d")
LOGFILE=/tmp/sd/watchdog/logs/wd_log_$now.log

if ping -c4 -q fritz.box > /dev/null ; then
  echo '['$(date)'] Wifi OK' >> $LOGFILE
else
  echo '['$(date)'] CONNECTION KO. Reboot wifi' >> $LOGFILE
  # Disable wifi interface
  /sbin/ifconfig wlan0 down
  # Give interface time to reset before bringing back up
  sleep 20
  /sbin/ifconfig wlan0 up
  # Give Wifi time to establish connection
  sleep 20
  # Check again the connection
  if ping -c4 -q fritz.box > /dev/null ; then
    echo '['$(date)'] Wifi connection successfully established after reset' >> $LOGFILE
  else
    echo '['$(date)'] Wifi connection cannot be established after reset. Reboot cam' >> $LOGFILE
    sync
    sync
    sync
    killall httpd
    killall mqttv4
    sleep 1
    reboot -f
  fi
fi

Thanks roleoroleo for this firmware and your support!

[torcuat]

after several months, logs and tests now I can say that the connection is correctly established and maintained day after day. the crontab feature in the UI allowed me to create and schedule a script to check the wifi connection and, if disconnected, reset the wifi adapter and, if necessary, reboot the cam. this solved my issue, even though it's a workaround.

in case it could help someone else, i post here my current solution:

* crontab configuration:
  `*/5 07-23 * * * /tmp/sd/watchdog/watchdog.sh` --> to check the wifi and reboot the cam
  `0 3 * * * find /tmp/sd/watchdog/logs/ -mtime +14 -delete` --> to delete older log files

I put the watchdog folder into /tmp/sd to keep it after a firmware update.

This is my watchdog.sh

now=$(date +"%Y_%m_%d")
LOGFILE=/tmp/sd/watchdog/logs/wd_log_$now.log

if ping -c4 -q fritz.box > /dev/null ; then
  echo '['$(date)'] Wifi OK' >> $LOGFILE
else
  echo '['$(date)'] CONNECTION KO. Reboot wifi' >> $LOGFILE
  # Disable wifi interface
  /sbin/ifconfig wlan0 down
  # Give interface time to reset before bringing back up
  sleep 20
  /sbin/ifconfig wlan0 up
  # Give Wifi time to establish connection
  sleep 20
  # Check again the connection
  if ping -c4 -q fritz.box > /dev/null ; then
    echo '['$(date)'] Wifi connection successfully established after reset' >> $LOGFILE
  else
    echo '['$(date)'] Wifi connection cannot be established after reset. Reboot cam' >> $LOGFILE
    sync
    sync
    sync
    killall httpd
    killall mqttv4
    sleep 1
    reboot -f
  fi
fi

Thanks roleoroleo for this firmware and your support!

Hi, thanks for this script. I am trying to do it but it doesn't execute. I've tried changing permissions with chmod via SSH, connected as root (without password) and it doesn't seem to be applying the changes:

`/tmp/sd/watchdog # chmod -c 777 watchdog.sh

mode of 'watchdog.sh' changed to 0777 (rwxrwxrwx)

/tmp/sd/watchdog # ls -la

total 96

drwx------ 2 root 0 32768 Oct 29 11:35 .

drwx------ 7 root 0 32768 Nov 10 13:26 ..

-rwx------ 1 root 0 825 Nov 10 13:51 watchdog.sh`

Do you have any idea on why and what I should be doing? Thanks in advance.

[roleoroleo]

You can't change permission on a fat32 filesystem.
But I think 700 is enough.

[torcuat]

Thank you. The reason it didn't execute was that I wrote the .sh file in Notepad++ for Windows and it doesn't use the "Unix" file format by default. It can be solved in the Edit menu>EOL conversion.

[martineiro23]

Hi,

I have the same problem with freezing of my 3 cameras as described above. I have to restart them to make them work again. Could someone explain to me step by step how to upload the watchdog.sh file to my camera?

Thank you

[grochib]

Uptime is at ~11d

I don't believe Wifi is the issue. The device is still connected. See screenshot. I checked the system log on my router (OpenWRT based), and see no deauth(s) or disconnect(s) from the camera.

After some time the camera does start working again (i.e. the webserver becomes available again) without a power cycle. I'll see if I can get some logging from httpd.

Hey @mhensema did you solve this? as far as I can see I have the same problem: the rtsp stream disappears in motioneye for 5 seconds, but the cam stays up and does not reboot. So there is no wifi issue here, as far as I can see. Did you find out what was behind this problem?

[taurgis]

@roleoroleo So I seem to have found a way to stabilise the connection by having 3 things running:

• Check WiFi by pinging, if it fails restart WiFi
• Check static IP, if it is still the one I configured in startup.sh - if not > reset IP
• Check RTSP process and restart it if it is down

Looking at the logs all three happen at random (sometimes a few times a day, other days OK), but recover by scheduling this script to run every 5 mins.

This is still with the Cloud enabled and almost everything but RTSP (this one only low, have others with both ) enabled. I will let this run for a week or so and then disable cloud again.

now=$(date +"%Y_%m_%d")
LOGFILE=/tmp/sd/watchdog/logs/wd_log_$now.log

if ping -c4 -q google.com > /dev/null ; then
  echo '['$(date)'] Wifi OK' >> $LOGFILE
else
  echo '['$(date)'] CONNECTION KO. Reboot wifi' >> $LOGFILE
  # Disable wifi interface
  /sbin/ifconfig wlan0 down
  # Give interface time to reset before bringing back up
  sleep 20
  /sbin/ifconfig wlan0 up
  # Give Wifi time to establish connection
  sleep 20
  # Check again the connection
  if ping -c4 -q google.com > /dev/null ; then
    echo '['$(date)'] Wifi connection successfully established after reset' >> $LOGFILE
  else
    echo '['$(date)'] Wifi connection cannot be established after reset. Reboot cam manually' >> $LOGFILE
  fi
fi

if ps | grep [r]RTSPServer > /dev/null
then
  echo '['$(date)'] RTSP Running' >> $LOGFILE
else
  echo '['$(date)'] RTSP KO. Reboot RTSP' >> $LOGFILE
  /tmp/sd/watchdog/restart.sh &
fi

if ifconfig wlan0 | grep -oE "192.168.0.227" > /dev/null ; then
  echo '['$(date)'] IP OK' >> $LOGFILE
else
  echo '['$(date)'] IP NOT OK, RESETTING' >> $LOGFILE
  ifconfig wlan0 192.168.0.227 netmask 255.255.255.0 up
  route add default gw 192.168.0.1
  echo "nameserver 192.168.0.1" >> /tmp/resolv.conf
fi

restart.sh

now=$(date +"%Y_%m_%d")
LOGFILE=/tmp/sd/watchdog/logs/wd_log_$now.log

echo '['$(date)'] Restarting RTSP' >> $LOGFILE
killall wd_rtsp.sh
killall rRTSPServer
sleep 60
rRTSPServer -r low -a aac

Note: The killall is redundant, but I keep it in there to manually do the restart and a 60 second wait since the restart fails otherwise.

[roleoroleo]

What's the problem of your cam?
wifi? rtsp?

[taurgis]

What's the problem of your cam? wifi? rtsp?

Generally the WiFi, but this is a problem of my creation with the location of the cameras. It mainly started when I used the “static IP startup.sh” trick that they seemed to disappear. Turned out that Yi restores the original IP, or gives it a different one after the WiFi reconnects, this script fixes that behaviour.

The RTSP one happens every so often, the process dies and this restarts it.

Now for the past week, the camera’s keep running (except for the odd 5 minutes with the WiFi) and I am slowly starting to re-enable some features to see if it breaks again - like disabling the cloud.

[taurgis]

What usually happens one time or twice a day, but this has to do with the WiFi and not so much the hack I think (note: I did not reboot the camera):

[roleoroleo]

Generally the WiFi, but this is a problem of my creation with the location of the cameras. It mainly started when I used the “static IP startup.sh” trick that they seemed to disappear. Turned out that Yi restores the original IP, or gives it a different one after the WiFi reconnects, this script fixes that behaviour.

The new hack version has the "Static IP feature". Try it.

[taurgis]

Generally the WiFi, but this is a problem of my creation with the location of the cameras. It mainly started when I used the “static IP startup.sh” trick that they seemed to disappear. Turned out that Yi restores the original IP, or gives it a different one after the WiFi reconnects, this script fixes that behaviour.

The new hack version has the "Static IP feature". Try it.

I will give it a try after my holidays, I figured out the issue with the IP - it was conflicting with one of my Tuya Wifi accessories that wakes up every so often to update metrics taking the same IP...

That explained the "randomness", it wakes up every x time based on temperature changes reaching a threshold - making it very unpredictable (kinda).