[RFC] Revising MQTT bridge structure

[Koenkk]

Problem

The current zigbee2mqtt/bridge/# topics are a mess. They are not well structured and don't provide a good reply/response mechanism. This makes it hard for other applications to integrate with Zigbee2mqtt. Therefore I propose the revise the structure while keeping the current working (so no breaking change).

Proposal

I didn't work out the full new structure yet, but this is the general idea I have.
@ anyone, feel free to review this, but I especially would like to request @carldebilly to review this (since I think it can be for great use of Zigbee2mqttAssistant).

The initial implementation has been completed and can be tested in the latest dev branch, to enable the new api set the following in configuration.yaml.

experimental:
  new_api: true

In case you want to disable the legacy api add the following (not mandatory, legacy and new api can exist simultaneously)

advanced:
  legacy_api: false

The documentation can be found here.

Note that the api can still change as it's not officially released yet. The plan is to finalise it on 1 September 2020, any feedback resulting in a breaking change should be submitted before this date.

TODO

• Change ID -> id in payloads (requested by @sjorge [RFC] Revising MQTT bridge structure #3281 (comment)) (breaking)
• Add healthcheck (requested by @jsassner [RFC] Revising MQTT bridge structure #3281 (comment))
• Allow to specify clusters to bind/unbind to binding commands (requested by @sjorge)
• Post logging to MQTT (requested by @CodeFinder2 [RFC] Revising MQTT bridge structure #3281 (comment))
• Enable/disable Home Assistant extension (requested by @bodiroga [RFC] Revising MQTT bridge structure #3281 (comment))
• Publish update information (e.g. {"update": {"progress": 20, "state": "available/updating/idle"}) to device state topics. (requested by @SeLLeRoNe FR - Have "update_available" set to "progress"  #3634 (comment))

[sjorge]

Can we also get a configuration.yaml option to disable the old behavior?

[gdave321]

IMHO To be of any use the "everything" to zigbee2mqtt/bridge should include for the devices the type of the device (this is also missing in the current config/devices/get). If type like lamp/color_temp/color/motionsensor/sensor/switch would be available, it can be used to auto create devices in application. Now this info is only available via homeassistant_discovery_topic but there are plenty applications around not really HA compatible.

[sjorge]

That would be nice, then maybe HA extension can maybe move to a seperate reposed ourside of the core. But there would probably need to be a better way to show what a device supports, sort of like feature flags. Type: bulb, features: on/off, colorxy, brightness,... not sure how feasible that is though. ~ sjorge

…

On 5 Apr 2020, at 13:38, gdave321 ***@***.***> wrote:  IMHO To be of any use the "everything" to zigbee2mqtt/bridge should include for the devices the type of the device (this is also missing in the current config/devices/get). If type like lamp/color_temp/color/motionsensor/sensor/switch would be available, it can be used to auto create devices in application. Now this info is only available via homeassistant_discovery_topic but there are plenty applications around not really HA compatible. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[nldroid]

Just my 2ct: if you publish 'everything' for zigbee2mqtt/bridge, the message could be huuuge. I've seen people on the issue list with 200 devices. Wouldn't it make sense to have a zigbee2mqtt/bridge/devices topic for that?

[Koenkk]

@gdave321 if some new discovery will be implemented, I think it should be an existing one, don't want to create yet another discovery mechanism.

@nldroid it can indeed grow large, but what would be the problem of this? It's not really meant for humans.

[sjorge]

On my earlier point, it doesn’t make sense for extensions to expose this over mqtt, the should be able to get this info some otherwise so disregard that remark.

[jsassner]

What i've been missing for a while is a Health Check topic, and that would fit really well in the Request/Response structure.
I have systems that want to check every x minutes that z2m is up and running (else it will warn me), and posting to zigbee2mqtt/bridge/request/healthcheck getting a response would be really good.

[nldroid]

@nldroid it can indeed grow large, but what would be the problem of this? It's not really meant for humans.

Yes but i still use MQTT explorer sometimes to check things,

Imho there should be a good balance in message size and number of topics. They didn't invent the hierarchy for nothing.

For software it's not a big deal to do two requests to get information about the bridge and the devices.

[Koenkk]

@nldroid indeed a valid point, so I think we should have zigbee2mqtt/bridge and zigbee2mqtt/bridge/entities (groups + devices). Just for my understanding, what information do you typically want to get from zigbee2mqtt/bridge?

[sjorge]

@nldroid @Koenkk we already have a zigbee2mqtt/bridge/group/ end point, maybe add a list to that?

Then we can perhaps move some of the bind / device stuff to zigbee2mqtt/bridge/device/<list|bind|unbind> ?

[carldebilly]

Sorry I was late to respond... I'm over busy those days... a full time job at home + a full time daddy is asking a lot.

Love the request id... I was about to suggest it when I read it :-). Love the idea of this refactoring.

But my concern is about the migration from the old to the new topics. Z2MA is not the only tool right now... Do you think we should have a hybrid mode for the migration time?

And... what is the timeline for that? I want to be ready on time for the release. Maybe @magpern can help too :-) (he's very helpful!).

[sjorge]

But my concern is about the migration from the old to the new topics. Z2MA is not the only tool right now... Do you think we should have a hybrid mode for the migration time?

From the commits I'm seeing on dev, yes there is a legacy mode which is on by default

[Koenkk]

@sjorge it will by zigbee2mqtt/bridge/entities so no need to worry about that. Bind unbind stuff will move to something like zigbee2mqtt/bridge/request/bind.

@carldebilly there won't be a breaking change so the old API will still work. Timeline: not really soon, first want to have a proper discussion + prepare zigbee2mqtt internally for this.

[magpern]

@carldebilly

Maybe @magpern can help too :-) (he's very helpful!).

You make me blush!

Yes, count me in. I have tonnes of day job disturbing my spare time right now, but it will calm down in 30 days or so.

[nldroid]

Just for my understanding, what information do you typically want to get from zigbee2mqtt/bridge?

For me, the most important thing is the version. Best would be with latest commit number so you'll never have any doubt on what version people are.

[CodeFinder2]

I would also suggest to split "everything" of zigbee2mqtt/bridge, maybe to zigbee2mqtt/bridge (version, commit hash, permit_join, etc), zigbee2mqtt/bridge/devices (all device related information) and zigbee2mqtt/bridge/groups (for group related information). This structure reduces the network load, simplifies dealing with tools like MQTT Explorer, and (hopefully) increases the performance of JSON parsing.

Another thing: what about sending all logging output to some (separate) MQTT topic? IIRC, this is another desired feature of Z2MA.

(I was thinking to develop a Qt based desktop application as a UI for Z2M.)

[Koenkk]

@CodeFinder2 good idea will include this.

[sjorge]

perhaps rename <prefix>/bridge to <prefix>/coordinator ? That would also allow legacy and the new stuff to both be active.

[Koenkk]

@sjorge not sure, it's not only about the coordinator but just the whole stuff itself. Currently we didn't get into problems regarding conflicts with the old api.

[CodeFinder2]

@Koenkk great, thanks! Maybe also split the various "attributes" into a dedicated JSON message (and not only provide a single string as printed in the log), like (at least)

{
	"type": "info",
	"time": "2020-04-22 22:48:10",
	"message": "... message as printed in Z2M's log ..."
}

as this allows for a more fine-grained visualization (of the log). I don't know (yet) if it makes sense to even split (the content of) message into more attributes (possibly depending on the concrete message).

(In a UI frontend, I am also thinking about live-filtering the log based on some input pattern, e.g., to show only log entries for a specific device. For instance, this might be helpful in case of a pairing process where we are only interessted in observing pairing-device related messages. Also, showing all errors in the recent past would then be very easy.)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Koenkk]

@kirovilya @nurikk I've added permit joining via a specific device. See https://github.com/Koenkk/zigbee2mqtt.io/blob/new_api/docs/information/mqtt_topics_and_message_structure.md#general

[Koenkk]

New api is now enabled by default in the dev branch.

[rekcodocker]

So my standard topic format is:
MSG_TYPE/MSG_SOURCE/DEVICE_TYPE/[/DEVICE_LOCATION[/...]]/DEVICE_NAME[/:VALUE_NAME]

I like the idea of using the power of the MQTT structure. To me that means smaller Jsons - and fewer of them!

So instead of receiving a Json like {"temperature":20,"humidity":50}, you could subscribe to
/zigbee2mqtt/devices/FriendlyNameOfSensor/humidity and receive the value '50'.

I would not recommend to include the type of device in its MQTT-topic. The type is an attribute, not an identifier. So I recommend:
"DEVICE_NAME/device_type" as a topic, and the actual type is the payload of that topic.
Example:
/zigbee2mqtt/devices/FriendlyNameOfSensor/type and receive "temp and humidity sensor".
If you use that structure, every device will have a couple of default topics, like capabilities, link status, power-type etcetera.

That way you can subscribe to exactly the information you want, instead of receiving a Json that may vary from one device to another.

Still, I think there are still very good usecases for posting a massive Json with the entire device-list.
Some software (like dashboard-software) will want to process a complete list at once, and it needs to be certain it has all devices in that list.
I don't know how to do that with micro-MQTT topics; so perhaps there needs to be a large 'blurb all devices' topic.

[sjorge]

Something less advanced that hat is already available

https://www.zigbee2mqtt.io/information/configuration.html

# Optional: experimental options
experimental:
  # Optional: MQTT output type: json, attribute or attribute_and_json (default: shown below)
  # Examples when 'state' of a device is published
  # json: topic: 'zigbee2mqtt/my_bulb' payload '{"state": "ON"}'
  # attribute: topic 'zigbee2mqtt/my_bulb/state' payload 'ON"
  # attribute_and_json: both json and attribute (see above)
  output: 'json'

[isom1266]

Hello and thank you for a great job. Is it possible to add Sonoff mini 01MINIZB module support to the plug?
The plugin detects this module but no support
Of course, it's about Domoticz

[timoline]

@Koenkk
I have an idea about a new device that is pairing to z2m.
Currently they will show up in the root of the z2m topic structure, you give it a friendly name and that topic will be created....that's okey, but the unfriendly name still shows up in the root.
Maybe it's an idea that new paired devices are showing up in their own topic (zigbee2mqtt/unfriendlyname or something)

This will be speed up the cleanup unfriendly names topic proccess, or you justl leave it in its own topic, but then the advantage is that it is isolated in it's own topic and not floating around in the root... ;-)

cuurently

[Koenkk]

@timoline were does this visualisation come from? (shouldn't it be fixed there?)

[timoline]

@timoline were does this visualisation come from? (shouldn't it be fixed there?)

no its just a visualisation of the realtime mqtt structure

New paired device publish itself in the topic (e.g.)
zigbee2mqtt/0xbc33acfffe5d1137
zigbee2mqtt/0xbc33acfffe5d1189
zigbee2mqtt/0xbc33acfffe5d1140

my request is to publih itself in: (or just some other defined topic)
zigbee2mqtt/paired/0xbc33acfffe5d1137
zigbee2mqtt/paired/0xbc33acfffe5d1189
zigbee2mqtt/paired/0xbc33acfffe5d1140

so it's easier to cleanup (after adding a friendly name/ or isolated in its own topic

extra:
the visualisation is just a fork
https://github.com/timoline/d3-MQTT-Topic-Tree

[timoline]

@Koenkk
I have an idea about a new device that is pairing to z2m.
Currently they will show up in the root of the z2m topic structure, you give it a friendly name and that topic will be created....that's okey, but the unfriendly name still shows up in the root.
Maybe it's an idea that new paired devices are showing up in their own topic (zigbee2mqtt/unfriendlyname or something)

This will be speed up the cleanup unfriendly names topic proccess, or you justl leave it in its own topic, but then the advantage is that it is isolated in it's own topic and not floating around in the root... ;-)

cuurently

this is fixed by 9b7ecff