Suddenly all nodes are gone in deCONZ #312
Comments
|
Damn, for the this very exact problem deCONZ 2.04.96 will contain improvements that the channel will not be lost in future. The PANID usually should not have changed?
Then do power-cycle all lights (or any other mains powered devices). They should scan all channels and detect the "new" network (since update id is higher). |
|
Unfortunately this didn't help. Some observations while resetting: Questions remain: How could ZigBee channel and network ID be changed on the fly - shouldn't this be prevented? And why did it change for no (obvious) reason? Thanks for all your good work on RaspBee/ deCONZ! EDIT: Not 100% sure PAN ID changed. I just read the PAN ID from the Philips light connected to RaspBee before and guessed that this was the former PAN ID. |
|
Could you get the network to work again? Sorry for the trouble hope this won't happen with recent version.
To put it simple it's a nasty firmware bug, in early 2018 there will be a proper fix for that, till then deCONZ (since 2.04.97) will handle the problem and make sure the channel stays steady. |
|
Yes I'm almost done, but it took quite a while to recover. Some IKEA lights were really hard to reintegrate into the network. |
|
I was on 2.04.84, I went out of town for 3 weeks, and came back to not having any lights. :( I have all the groups and scene buttons on the web UI, but no lights. I've updated to 2.04.99 but still no lights. Gateway is on channel 15. Touchlink scan finds devices on channel 15, so I don't think the channel has changed. Is there anything I can do to recover the lights? Do I need to start adding the lights back one by one? That would be so much pain. I have 17 groups, each with many (10+) scenes. If I have to redo all my scenes I'd consider throwing both my RPi and myself out the window. I restored an RPi image made back when it should have been working correctly, still no lights. How can I check what lights are in the db file? What else can I do? |
My lights seem to be in the db, why don't I see them in the web UI? Outputs: http://n4te.com/temp/log.txt @manup @ebaauw Any ideas? Sorry to ping you directly, but not being able to control any of the lights makes my wife angry! ;) |
|
Power off all lights, exit deCONZ, reset the RaspBee/ConBee (or power cycle the raspberry), start deCONZ. Power on the lights one by one (or in small batches). See #230 (comment) and #195 (comment). I’m running firmware 261a0500 now, fingers crossed. |
|
Thanks for the links and info! I flashed to 261a0500 via gcfflasher to be sure. When I renovated my place, I did away with all light switches, using Dresden scene switches instead. Many of my lights have a hardwired ZigBee 1-10V controller and always have power. The rest are E27 bulbs, on the same breaker. The best I could do is get on a ladder and unscrew the E27 bulbs to reduce the number of lights that have power. I followed your instructions (without unscrewing E27 bulbs, yet). Should I expect lights to appear in the web UI? This is what I see: Is there anything else I can do to debug the problem? |
|
Can you make a touchlink scan in the webapp under menu settings. Near by lights should be listed there with chnall and panid. Next we can check that deconz settings match these |
|
Sure thing: Is it bad that the scan shows lights with 0x875b and the gateway is 0xbb28? Digging around in the db, is it odd |
Yes and no ;) please try the following:
Hope this helps, still strange why the panid was changed in the deCONZ node, I think the firmware needs more verification checks to prevent such issues. |
|
OK, I hope that will help! I have been running headless, is it possible to set the pan ID without using the UI? I'd rather not install X, etc just for that, but I will if necessary. |
|
I saved a config data file from the web UI, decompressed (it is a I then rebooted the RPi aaaand same result -- no lights. Maybe a reboot isn't equivalent to leave/join? |
|
This is still a very big problem for me. I'd like to know what I should do to further debug the problem and try to recover my lights and scenes, or if I should use TouchLink to get my lights back and then spend hours and days recreating all my scenes. |
|
Same here, see #195 (comment). Did you upgrade your Hue lights to the latest firmware? |
|
@ebaauw I don't think I've ever updated my Hue light firmware. OTAU is disabled and I have never had it enabled. FWIW, I did have some of the lights connected to a Hue hub months ago. I have Hue E27 bulbs, LED strips, and motion sensors. I also have hardwired ZigBee 1-10V modules, an Osram plug, and Dresden scene switches. Why does my RaspBee not find any lights? Is it trying? If so, why does it fail? Here is a log of deCONZ running for 2 hours: I want to recover my lights. Specifically, setting up all my scenes again would take days of effort and I don't want to go through that again. If I have to use TouchLink to get my lights back, will I lose my scenes? If so, OK I can go through the pain or redoing my scenes, but I need to know how to avoid this ever happening again. I have managed to stay civil, but please know my frustration is absolutely through the roof. The RaspBee is crucial for my home to function. It's unacceptable to have my lights gone for 3+ days. As a software engineer, I understand things can go wrong and that is OK, but I need to know how to fix it. Also, fixing it should not require days of my time redoing all my scenes. Can I back up all my lights and scenes so if this happens again I can apply the backup? |
|
Hi the missing step is setting the panid for raspbee to same as the lights
have. Currently this is only possible via deconz gui as described above.
You can temporarily use the sd card image on another card for this. The
panid will be stored on the shield.
|
|
Thanks, I wanted to confirm before installing X. I installed X and used the deCONZ GUI to set pan ID 0x875b, then leave/join. No new devices appeared. I also tried changing to another pan ID and back to 0x875b. Using a TouchLink scan, I still see devices on pan ID 0x875b and channel 15 -- same as the RaspBee. What does it mean that the lights don't show up? :( |
|
Could it be that the RaspBee/ deCONZ sent a kind of network reset unintentional/ intentional? |
I suspect it does, but chokes on what it finds, at a very low level in the ZigBee stack (probably RaspBee firmware). Not my area of expertise (nor my ambition to make it so). I find that the ZigBee network is functioning (lights react to Hue dimmer switches), that the deCONZ REST API is functioning (schedules, rules get executed, CLIP sensors get updated), but ZigBee traffic from the RaspBee is somehow blocked, lost, failing, whatever. Consistently, my remedy for this situation has been to power off all ZigBee routers (lights), stop deCONZ, reset the RaspBee, start deCONZ, power up the lights one by one (or in small batches). This morning I only needed to power off most of my Hue lights (24 out of a total of 42) to recover. Got me thinking it might be related to the latest Hue firmware.
I fully understand and, unfortunately, share your fustration. However, as a software engineer, I do appreciate that I took a risk running my home automation on a beta version of deCONZ. If outages like these are unacceptable, better revert to the latest stable version. |
No the settings will only be set to new values when @NathanSweet can you share a screenshot of the deCONZ network settings, to verify that the other settings are valid? I'm currently traveling (till end of January) so coding time is very limited but in early 2018 the firmware + low level deCONZ code will be refactored so that the above issue can't happen anymore and network parameters will be stored in and verified from the database. I'm still not sure what is the root of the problem, the firmware hadn't have such problems in years, but networks became more complex with various new devices and vendors in the game. |
|
@manup Sure thing: All I really care about is my scenes. Where are scenes stored? Can my scenes by restored or the data extracted so I can apply it after using TouchLink? @ebaauw I feel forced to use the beta since it has such a long development cycle (more than a year?). The last stable release wouldn't be usable for me. Disaster recovery is important, even for betas. |
|
Just a short note, will write more later. Settings look valid, have you tried to power cycle the lights after changing the settings? Do the motion blink blue sometimes? Can you confirm that you didn't use the Reset Gateway function in the app? As recover method we have the backup function in the app Settings / Gateway |
|
Yep, the lights get power cycled pretty often, because the circuit breaker is the only way to turn them on/off. Motion sensors don't seem to ever blink blue. The blink red when detecting motion, otherwise the LED is off. I never clicked Reset Gateway. I'm scared to even use TouchLink, since I expect it will nuke my scenes. If TouchLink doesn't nuke my scenes please let me know and I'll just re-add all my lights that way and stop bothering you. Thanks for helping out even when on holiday! <3 |
|
@NathanSweet did you try and restart deCONZ, while all lights are powered off? And then power-on lights in small batches (if you can only power them off using a single circuit breaker, you might need to unscrew the lights and screw them in one by one).
The scenes are stored in nonvolatile memory in the lights. As long as you don't reset these, they would still be present. However, other than recalling the scene and checking the light state, there's no way to query the light as to which state belongs to the scene. The gateway stores the scene states in its database, so it can update its cache of lightstates when recalling a scene. Otherwise you'd have to wait till the lights are polled (or reported their state attributes) before the API reflects the actual light states. The gateway database is stored in
Same here. I use bash scripts to create/restore my gateway rules, schedules, CLIP sensors, groups, scenes. Unfortunately the scripts for scenes don't work as well as I want them to, so I end up restoring them manually. Also, manually re-pairing all lights, sensors, and switches and hacking the database to make sure they have the right IDs is a bitch. I have yet to try the gateway backup/restore (to my test Raspberry), which would be the recovery procedure for hardware failure. However, all my disasters until now have been software related, so a gateway restore wouldn't have helped. |
|
@ebaauw I did, and tried again just now. My small batch was 4 lights.
What are the ways to reset the lights (so I can avoid doing that)? Does using TouchLink to get my lights back clear the scenes? I'd really like to know that. It's a good idea to use setup scripts. Maybe I do that if I have to redo everything. Ideally I could set it up, then extract the data to a script. Maybe I can dig the data out of the sqllite db, I'll see about that. It seems like it would be a good deCONZ function to be able to extract that data and apply it later, but maybe that's a bit complicated to support everything people can do. FWIW, I wrote a Java API for the deCONZ REST and websocket APIs. This could make for some nice, type safe, cross platform scripts (example usage). |
Yes, a reset through Touchlink would erase the scenes. As would a reset through the power cycle reset sequence (differs per manufacturer, not supported by Philips Hue), or a reset through the Hue dimmer switch (pressing and holding On and Off for 10 seconds - actually issues a touchlink reset). There's also a Reset button in the Cluster Info panel for the Basic cluster in the deCONZ GUI, but in my experience that's sort of hit and miss. |
|
I used TouchLink to get back all my lights. After that, they show up in deCONZ with the correct name and in the correct groups. As expected, the scene information is gone. I wrote some code to apply the scene information from the deCONZ database. This gave me my scenes back, so I'm happy about that. I wonder if this should be a built-in function? Code could be written to restore all the lights, groups, scenes, keep the same IDs, etc but I did only what I needed for the moment. My issue now is that one of my motion sensors reports temp, light, presence, but the other four don't. I deleted them, held the physical setup button to add them back, same result. I noticed this: Sensor 21 works, sensor 9 doesn't. I notice 9 has only one circle on the right while 21 has two. What does that mean? Is it relevant? (BTW, when I mouse wheel in deCONZ it zooms either very far in, or very far out -- frustrating.) |
|
The deCONZ manual says:
I repeatedly removed and re-added the motion sensors and still they had one circle, even after waiting a long time. Now they are showing two circles, but I don't know if it was something I did. Even with two circles only some of them work. Some of them have a black dot, but I couldn't find in the docs what that means (none of these 3 sensors give events): I saw the black dot on switches and it seemed to go away after the first button press (after turning blue momentarily). Anyway, I don't get events for some motion sensors and I can't think of anything else to try. Any ideas? |
The right circle only appears once deCONZ has read the device's clusters. If it doesn't appear, the read of the sensor failed, typically because it fell asleep. This is quite unusual for the Hue motion sensor, however. I would try pairing it close to the gateway, while powering off any nearby lights, to make sure it connects directly to the gateway (instead of through the mesh network).
In my experience, the black dot appears after the device has been reported by one of its neighbours, but deCONZ hasn't yet communicated with the device itself. Again, it's quite unusual for the Hue motion sensors, but it happens to all my other sensors after restarting deCONZ.
This is probably because deCONZ hasn't (yet?) setup attribute reporting for the sensor. It should do so on pairing, or later, when it receives some traffic from the sensor, so it knows the sensor is awake. You might try and read the sensor's clusters in the deCONZ GUI and see if deCONZ takes the hint. Otherwise, you might want to setup attribute reporting manually, see #340 (comment). |
|
Motion sensor was right next to the RaspBee. Thanks for the link, I looked at attribute reporting for occupancy and found it was 65355, 65355. I changed it to 1, 300 and left it overnight. The sensors all work now! Thanks for your help! Resetting all my devices wasn't at all fun, but at least my scenes were salvaged. Maybe someone else finds the ScenesFromSQL code useful. Maybe deCONZ could do per node logging? Currently I have no idea what is going on with a device. Is deCONZ doing something? Trying to fetch cluster descriptors? Waiting for the device to get out of standby? Did it ever setup attribute reporting? Was there some error? Currently I feel really in the dark about what it is doing, both per device and globally. The general logging that can be enabled seems too low level to be of use to me. |
Yeah, the deCONZ logging is somewhat clumsy. The messages are there, but hidden in so much noise that you won't be able to find them. Also each type of message uses a different format, so it's impossible to grep for all messages concerning a single device. I would like to see a consistent pattern for log messages, including the MAC address and/or device name of the device the message concerns. Looking at the current code, that's gonna be a hell of a change to implement... |
|
These past weeks my lights are all OK, but I'm having a lot of trouble with switches and motion sensors. Some switches and motion sensors work, some don't. No amount of resetting or fiddling with occupancy attribute reporting fixes it. Sometimes they don't work for hours or a day, then start working, sometimes they never start working. Sometimes they stop working if deconz is restarted. It's completely FUBAR! I was able to reproduce the PAN ID changing. I wanted to add a Hue bulb which had been used with a Philips hub, so I needed to TouchLink reset it. TouchLink kept finding my other devices, so I used circuit breakers to power off other devices. I accidentally powered off the RPi! It came back, I did a TouchLink scan, found the light, reset it, added it, turned all the circuit breakers back on, and that is when I noticed the only device on my network is the new bulb. Fuuuuuu... I checked the PAN ID and it had changed! Since I have the network settings screenshot above in this issue, I saw the network key also changed and NWK update ID changed from 2 to 0. I changed the PAN ID, network key, and NWK update ID back to what they were and my devices came back (praise be to the FSM!). I was about to bite my keyboard in half there for a second. Maybe before when this happened I didn't have the old network key, it got changed somehow, and so when I changed my PAN ID back it wasn't enough to recover my devices. TLDR; pulling the plug on the RPi seemed to cause PAN ID, network key, and NWK update ID to change. |
|
Be sure to make a backup of your configuration from the Advanced Settings in the web app. #247 (comment). |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
This morning suddenly all nodes (~30) disappeared in deCONZ without changing anything and running without problems before. The nodes consist of different manufacturers and type (IKEA, Philips, OSRAM, Xiaomi).
I noticed that the network settings for the RaspBee changed (PAN ID and ZigBee channel). I checked the Network ID and channel of my former connected Philips lights and changed the network settings accordingly. Unfortunately most of the nodes (including the Philips lights) except for the Xiaomi window and door sensors do not show up after changing the PAN ID and ZigBee channel.
The Philips lights show up as soon as I reset them.
EDIT: The Xiaomi window and door sensors show up, but do not emit any websocket messages when activating/ triggering them. The Philips lights do send websockets and can be controlled via REST API calls.
I'm running RaspBee on Raspi3/ stretch and deCONZ 2.04.84.
I would like to avoid to reset all nodes so any help is appreciated.
The text was updated successfully, but these errors were encountered: