SA-003 Sending reports to the broadcast endpoint

[chriskemp]

I've got a couple of ewelink SA-003 on/off switches. They don't seem to respond to any attempts to configure reporting but instead seem to regulary send a report to the broadcast endpoint (255).

As far as I can tell they do this straight after a reset, and before any configuring has taken place, so I'm not sure there's much that can be done to configure them properly.

The wireshark log shows:

174	111.699902	0x3286	0x0000	ZigBee HA	52	ZCL: Report Attributes, Seq: 134
176	111.731611	0x0000	0x3286	ZigBee HA	50	ZCL: Default Response, Seq: 134
178	111.750677	0x0000	0x3286	ZigBee HA	50	ZCL: Default Response, Seq: 134
180	111.770181	0x0000	0x3286	ZigBee HA	50	ZCL: Default Response, Seq: 134
182	111.789760	0x0000	0x3286	ZigBee HA	50	ZCL: Default Response, Seq: 134

Frame 174: 52 bytes on wire (416 bits), 50 bytes captured (400 bits) on interface 0
IEEE 802.15.4 Data, Dst: 0x0000, Src: 0x3286
ZigBee Network Layer Data, Dst: 0x0000, Src: 0x3286
ZigBee Application Support Layer Data, Dst Endpt: 255, Src Endpt: 1
    Frame Control Field: Data (0x00)
    Destination Endpoint: 255
    Cluster: On/Off (0x0006)
    Profile: Home Automation (0x0104)
    Source Endpoint: 1
    Counter: 135
ZigBee Cluster Library Frame, Command: Report Attributes, Seq: 134
    Frame Control Field: Profile-wide (0x08)
    Sequence Number: 134
    Command: Report Attributes (0x0a)
    Attribute Field
        Attribute: OnOff (0x0000)
        Data Type: Boolean (0x10)
        On/off Control: Off (0x00)

Note that it is sending it to endpoint 255.

The problem is that the CC2531 (or z-stack) then sends this same packet to four endpoints in the controller and so zigbee-herdsman responds with 4 generic responses (as per wireshark above) and zigbee2mqtt receives 4 identical messages:

debug 2019-12-18 01:00:53: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0
debug 2019-12-18 01:00:53: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0
debug 2019-12-18 01:00:53: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0
debug 2019-12-18 01:00:53: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0

Obviously none of this is a major issue, but it does seem unclean. Has anyone any suggestions on a "proper" way to deal with this? I'll happily then code/test and submit a PR.

If my understanding is correct, ideally (but I don't think it does) z-stack should tell us that a packet was sent to the broadcast endpoint and therefore the low levels shouldn't send a default response. Then each "endpoint handler" should decide whether to be interested / accept the packet.

In practice, at the moment, I don't think the controller code distinguishes much on its endpoint. I think the destination endpoint is lost by the time the report arrives at onZclOrRawData (zigbee-herdsman/src/controller/controller.ts:387). Potentially the destination endpoint can be added to the payload structure at adapter/z-stack/adapter/zStackAdapter.ts:500 and then, as a workaround, onZclOrRawData can use device specific info to not send a response and to only pass on up one of the four.

Any thoughts?

Thanks, Chris

[Koenkk]

I think the only way to do this easily is to debounce on the transaction number (https://github.com/Koenkk/zigbee-herdsman/blob/master/src/controller/controller.ts#L501)

[chriskemp]

Something like this?

https://github.com/chriskemp/zigbee-herdsman/commit/464b01937af139bbff679b410439dc6becda2323

Are you ok with the meta data being used in this way or would you prefer a new device property?

[Koenkk]

I think we can achieve this easier, can you try the changes from: Koenkk/zigbee-herdsman-converters@82202e0

[chriskemp]

Apart from the bug Koenkk/zigbee-herdsman-converters@82202e0#r36902229 this stops the multiple updates going out on MQTT but it doesn't stop the multiple "default responses" going out on zigbee, presumably since that has already been sent by the time the message gets to the converter?

Obviously that's an improvement, but it'd be nice to keep the on-air traffic down as I was hoping to have quite a bunch of the devices.

[Koenkk]

Reading the ZCL specification I found the following: No default response to non-unicast message

Can you try adding a check to see if the endpoint != 255 (https://github.com/Koenkk/zigbee-herdsman/blob/master/src/controller/controller.ts#L536)

[chriskemp]

I started trying to do this, but (partly as in my original message) I don't think that zigbee-herdsman knows that it was sent to the 0xff endpoint. I've enabled more debug logging and I get:

2020-01-22T23:27:04.753Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":6,"srcaddr":12934,"srcendpoint":1,"dstendpoint":110,"wasbroadcast":0,"l inkquality":55,"securityuse":0,"timestamp":11723485,"transseqnumber":0,"len":7,"data":{"type":"Buffer","data":[8,196,10,0,0,16,0]}}

2020-01-22T23:27:04.754Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":false,"direction":1,"disableDefaul tResponse":false},"transactionSequenceNumber":196,"manufacturerCode":null,"commandIdentifier":10},"Payload":[{"attrId":0,"dataType":16,"attrData":0}],"Cluster":{"ID":6,"attributes":{"o nOff":{"ID":0,"type":16,"name":"onOff"},"globalSceneCtrl":{"ID":16384,"type":16,"name":"globalSceneCtrl"},"onTime":{"ID":16385,"type":33,"name":"onTime"},"offWaitTime":{"ID":16386,"typ e":33,"name":"offWaitTime"}},"name":"genOnOff","commands":{"off":{"ID":0,"parameters":[],"name":"off"},"on":{"ID":1,"parameters":[],"name":"on"},"toggle":{"ID":2,"parameters":[],"name" :"toggle"},"offWithEffect":{"ID":64,"parameters":[{"name":"effectid","type":32},{"name":"effectvariant","type":32}],"name":"offWithEffect"},"onWithRecallGlobalScene":{"ID":65,"paramete rs":[],"name":"onWithRecallGlobalScene"},"onWithTimedOff":{"ID":66,"parameters":[{"name":"ctrlbits","type":32},{"name":"ontime","type":33},{"name":"offwaittime","type":33}],"name":"onW ithTimedOff"}},"commandsResponse":{}}},"address":12934,"endpoint":1,"linkquality":55,"groupID":0}' zigbee2mqtt:debug 2020-01-22 23:27:04: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0

zigbee2mqtt:info 2020-01-22 23:27:04: MQTT publish: topic 'zigbee2mqtt/OnOff1', payload '{"state":"OFF","linkquality":55}'

2020-01-22T23:27:04.774Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":6,"srcaddr":12934,"srcendpoint":1,"dstendpoint":11,"wasbroadcast":0,"linkquality":55,"securityuse":0,"timestamp":11723485,"transseqnumber":0,"len":7,"data":{"type":"Buffer","data":[8,196,10,0,0,16,0]}}

2020-01-22T23:27:04.775Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false},"transactionSequenceNumber":196,"manufacturerCode":null,"commandIdentifier":10},"Payload":[{"attrId":0,"dataType":16,"attrData":0}],"Cluster":{"ID":6,"attributes":{"onOff":{"ID":0,"type":16,"name":"onOff"},"globalSceneCtrl":{"ID":16384,"type":16,"name":"globalSceneCtrl"},"onTime":{"ID":16385,"type":33,"name":"onTime"},"offWaitTime":{"ID":16386,"type":33,"name":"offWaitTime"}},"name":"genOnOff","commands":{"off":{"ID":0,"parameters":[],"name":"off"},"on":{"ID":1,"parameters":[],"name":"on"},"toggle":{"ID":2,"parameters":[],"name":"toggle"},"offWithEffect":{"ID":64,"parameters":[{"name":"effectid","type":32},{"name":"effectvariant","type":32}],"name":"offWithEffect"},"onWithRecallGlobalScene":{"ID":65,"parameters":[],"name":"onWithRecallGlobalScene"},"onWithTimedOff":{"ID":66,"parameters":[{"name":"ctrlbits","type":32},{"name":"ontime","type":33},{"name":"offwaittime","type":33}],"name":"onWithTimedOff"}},"commandsResponse":{}}},"address":12934,"endpoint":1,"linkquality":55,"groupID":0}'

zigbee2mqtt:debug 2020-01-22 23:27:04: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0

2020-01-22T23:27:04.782Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":6,"srcaddr":12934,"srcendpoint":1,"dstendpoint":8,"wasbroadcast":0,"linkquality":55,"securityuse":0,"timestamp":11723485,"transseqnumber":0,"len":7,"data":{"type":"Buffer","data":[8,196,10,0,0,16,0]}}

2020-01-22T23:27:04.784Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false},"transactionSequenceNumber":196,"manufacturerCode":null,"commandIdentifier":10},"Payload":[{"attrId":0,"dataType":16,"attrData":0}],"Cluster":{"ID":6,"attributes":{"onOff":{"ID":0,"type":16,"name":"onOff"},"globalSceneCtrl":{"ID":16384,"type":16,"name":"globalSceneCtrl"},"onTime":{"ID":16385,"type":33,"name":"onTime"},"offWaitTime":{"ID":16386,"type":33,"name":"offWaitTime"}},"name":"genOnOff","commands":{"off":{"ID":0,"parameters":[],"name":"off"},"on":{"ID":1,"parameters":[],"name":"on"},"toggle":{"ID":2,"parameters":[],"name":"toggle"},"offWithEffect":{"ID":64,"parameters":[{"name":"effectid","type":32},{"name":"effectvariant","type":32}],"name":"offWithEffect"},"onWithRecallGlobalScene":{"ID":65,"parameters":[],"name":"onWithRecallGlobalScene"},"onWithTimedOff":{"ID":66,"parameters":[{"name":"ctrlbits","type":32},{"name":"ontime","type":33},{"name":"offwaittime","type":33}],"name":"onWithTimedOff"}},"commandsResponse":{}}},"address":12934,"endpoint":1,"linkquality":55,"groupID":0}'

zigbee2mqtt:debug 2020-01-22 23:27:04: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0

2020-01-22T23:27:04.791Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":6,"srcaddr":12934,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":55,"securityuse":0,"timestamp":11723485,"transseqnumber":0,"len":7,"data":{"type":"Buffer","data":[8,196,10,0,0,16,0]}}

2020-01-22T23:27:04.792Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false},"transactionSequenceNumber":196,"manufacturerCode":null,"commandIdentifier":10},"Payload":[{"attrId":0,"dataType":16,"attrData":0}],"Cluster":{"ID":6,"attributes":{"onOff":{"ID":0,"type":16,"name":"onOff"},"globalSceneCtrl":{"ID":16384,"type":16,"name":"globalSceneCtrl"},"onTime":{"ID":16385,"type":33,"name":"onTime"},"offWaitTime":{"ID":16386,"type":33,"name":"offWaitTime"}},"name":"genOnOff","commands":{"off":{"ID":0,"parameters":[],"name":"off"},"on":{"ID":1,"parameters":[],"name":"on"},"toggle":{"ID":2,"parameters":[],"name":"toggle"},"offWithEffect":{"ID":64,"parameters":[{"name":"effectid","type":32},{"name":"effectvariant","type":32}],"name":"offWithEffect"},"onWithRecallGlobalScene":{"ID":65,"parameters":[],"name":"onWithRecallGlobalScene"},"onWithTimedOff":{"ID":66,"parameters":[{"name":"ctrlbits","type":32},{"name":"ontime","type":33},{"name":"offwaittime","type":33}],"name":"onWithTimedOff"}},"commandsResponse":{}}},"address":12934,"endpoint":1,"linkquality":55,"groupID":0}'

zigbee2mqtt:debug 2020-01-22 23:27:04: Received Zigbee message from 'OnOff1', type 'attributeReport', cluster 'genOnOff', data '{"onOff":0}' from endpoint 1 with groupID 0

...note that the incoming AREQ from z-stack has the dstendpoint set to the particular endpoint and the wasbroadcast flag isn't set, so I'm out of ideas how to tell! Any ideas?

[chriskemp]

Just had a look at z-stack and I believe that this is the code that "hides" the broadcast endpoint from us:

Components\stack\af\AF.c (line 423 ish):

if ( (aff->ProfileID == epProfileID) ||
     ((epDesc->endPoint == ZDO_EP) && (aff->ProfileID == ZDO_PROFILE_ID)) ||
     ((epDesc->endPoint != ZDO_EP) && ( aff->ProfileID == ZDO_WILDCARD_PROFILE_ID )) )
{
  // Save original endpoint
  uint8 endpoint = aff->DstEndPoint;

  // overwrite with descriptor's endpoint
  aff->DstEndPoint = epDesc->endPoint;

  afBuildMSGIncoming( aff, epDesc, SrcAddress, SrcPanId, sig,
                     nwkSeqNum, SecurityUse, timestamp, radius );

  // Restore with original endpoint
  aff->DstEndPoint = endpoint;
}

This code seems to be the same in z-stack 3.0.2.

[Koenkk]

yes that indeed seems to cause the issue. Can you try commenting aff->DstEndPoint = epDesc->endPoint;

I've also checked the wasbroadcast, it will only be set when the message is broadcasted to all devices, this message is only broadcasted to all endpoints of a certain device (coordinator in this case).

[chriskemp]

I haven't tried compiling / flashing the z-stack code yet. I'll have a go!

I guess the "perfect" solution is to set the wasbroadcast flag (or add a wasbroadcastendpoint) in z-stack and then handle it in herdsman...although it does seem rather an overkill!

[chriskemp]

Hmm, sorry, I hadn't realised you had to get a 30 day licence to compile the z-stack code. I don't really want to do that right now as I might want it for something more useful another time!

How about:

https://github.com/chriskemp/zigbee-herdsman-converters/commit/3722b85cdc4938f3da0f32ed4a5aff2f521eddd1
and
https://github.com/chriskemp/zigbee-herdsman/commit/8659dd95ccbaafc31eba24c2fa4ef4a34355d1c6

?? It doesn't solve the broadcast endpoint for any other devices but it seems a simple / tidy enough fix for now?

[Koenkk]

Although this is a bit "hacky" I think its fine as it only affects this device and is in a device specific part.

Can you make the PR's? Then I will merge them.

[Koenkk]

Merged, thanks!