Multiple EMS Devices share same id when using HA & MQTT

[Rainson12]

Before creating a new issue please check that you have:

Bug description
We operate 2 EMS-ESP-32 systems for 2 houses conntected to the same MQTT broker. The heating systems connected to the ems-esps are the same model. I have configured unique network hostnames and unique client ids for both ems-esps. However when running HA & MQTT, home assistant only detects one of the ems-esps and it seems like all entities are overwritten by each other meaning if I add the detected entites to the overview, the displayed values keep switching between the first and seconds ems system (overwriting each others values).

Steps to reproduce
Connect two EMS-ESP-32 to the same heating system model, set different hostnames and clientids and run Home Assistant and enable auto discovery.

Expected behavior
The detected devices and entities have unique ids.

Screenshots
EMS-ESP #1

EMS-ESP #2

HA logs:

HA MQTT detected devices:

Device information
Device #1

{
  "System Status": {
    "version": "3.5.0b4",
    "uptime": "000+00:19:32.997",
    "freemem": 117,
    "free_app": 115,
    "reset reason": "Software reset CPU / Software reset CPU"
  },
  "Network Status": {
    "connection": "WiFi",
    "hostname": "hausa",
    "RSSI": -72,
    "IPv4 address": "192.168.100.20/255.255.255.0",
    "IPv4 gateway": "192.168.100.1",
    "IPv4 nameserver": "192.168.100.1",
    "static ip config": true,
    "enable IPv6": false,
    "low bandwidth": false,
    "disable sleep": true,
    "AP provision mode": "disconnected",
    "AP security": "wpa2",
    "AP ssid": "ems-esp"
  },
  "NTP Status": {
    "network time": "connected",
    "enabled": true,
    "server": "time.google.com",
    "tz label": "Europe/Amsterdam"
  },
  "OTA Status": {
    "enabled": true,
    "port": 8266
  },
  "MQTT Status": {
    "MQTT status": "connected",
    "MQTT publishes": 756,
    "MQTT publish fails": 0,
    "enabled": true,
    "client_id": "hausa",
    "keep alive": 60,
    "clean session": false,
    "base": "hausa",
    "discovery prefix": "homeassistant",
    "nested format": 2,
    "ha enabled": true,
    "mqtt qos": 0,
    "mqtt retain": false,
    "publish time boiler": 10,
    "publish time thermostat": 10,
    "publish time solar": 10,
    "publish time mixer": 10,
    "publish time other": 10,
    "publish time sensor": 10,
    "publish single": false,
    "publish2command": false,
    "send response": false
  },
  "Syslog Status": {
    "enabled": false
  },
  "Sensor Status": {
    "temperature sensors": 0,
    "temperature sensor reads": 0,
    "temperature sensor fails": 0,
    "analog sensors": 0,
    "analog sensor reads": 0,
    "analog sensor fails": 0
  },
  "API Status": {
    "API calls": 0,
    "API fails": 0
  },
  "Bus Status": {
    "bus status": "connected",
    "bus protocol": "Buderus",
    "bus telegrams received (rx)": 1224,
    "bus reads (tx)": 310,
    "bus writes (tx)": 0,
    "bus incomplete telegrams": 0,
    "bus reads failed": 1,
    "bus writes failed": 0,
    "bus rx line quality": 100,
    "bus tx line quality": 100
  },
  "Settings": {
    "board profile": "CUSTOM",
    "locale": "de",
    "tx mode": 2,
    "ems bus id": 11,
    "shower timer": false,
    "shower alert": false,
    "rx gpio": 16,
    "tx gpio": 17,
    "dallas gpio": 18,
    "pbutton gpio": 0,
    "led gpio": 2,
    "hide led": false,
    "notoken api": false,
    "readonly mode": false,
    "fahrenheit": false,
    "dallas parasite": false,
    "bool format": 1,
    "bool dashboard": 1,
    "enum format": 1,
    "analog enabled": true,
    "telnet enabled": true
  },
  "Devices": [
    {
      "type": "Boiler",
      "name": "GBx72/Trendline/Cerapur/Greenstar Si/27i",
      "device id": "0x08",
      "product id": 123,
      "version": "07.00",
      "entities": 64,
      "handlers received": "0x10 0x11 0x15 0x1C 0x18 0x19 0x34 0x2A",
      "handlers fetched": "0x14 0x16 0x33 0x26",
      "handlers pending": "0xBF 0xC2 0x1A 0x35 0xD1 0xE3 0xE4 0xE5 0xE6 0xE9 0xEA"
    },
    {
      "type": "Thermostat",
      "name": "RC300/RC310/Moduline 3000/1010H/CW400/Sense II",
      "device id": "0x10",
      "product id": 158,
      "version": "74.02",
      "entities": 48,
      "handlers received": "0x06 0xA2 0x02BA 0x02BB 0x02BC 0x02BD 0x02BE 0x02BF 0x02C0 0x031D 0x0267",
      "handlers fetched": "0x02A5 0x02B9 0x02AF 0x029B 0x02F5 0x031B 0x023A 0x0240",
      "handlers pending": "0xA3 0x12 0x13 0x0471 0x02A6 0x02B0 0x029C 0x0472 0x02A7 0x02B1 0x029D 0x0473 0x02A8 0x02B2 0x029E 0x0474 0x02A9 0x02B3 0x029F 0x0475 0x02AA 0x02B4 0x02A0 0x0476 0x02AB 0x02B5 0x02A1 0x0477 0x02AC 0x02B6 0x02A2 0x0478 0x02CC 0x02CE 0x02D0 0x02D2 0x031E",
      "handlers ignored": "0x35 0x23 0x1A 0xBF"
    },
    {
      "type": "Controller",
      "name": "BC25",
      "device id": "0x09",
      "product id": 125,
      "version": "03.05",
      "entities": 0
    }
  ]
}

Device #2:

{
  "System Status": {
    "version": "3.5.0b4",
    "uptime": "000+00:20:08.494",
    "freemem": 117,
    "free_app": 115,
    "reset reason": "Software reset CPU / Software reset CPU"
  },
  "Network Status": {
    "connection": "WiFi",
    "hostname": "hausb",
    "RSSI": -71,
    "IPv4 address": "192.168.100.21/255.255.255.0",
    "IPv4 gateway": "192.168.100.1",
    "IPv4 nameserver": "192.168.100.1",
    "static ip config": true,
    "enable IPv6": false,
    "low bandwidth": false,
    "disable sleep": true,
    "AP provision mode": "disconnected",
    "AP security": "wpa2",
    "AP ssid": "ems-esp"
  },
  "NTP Status": {
    "network time": "connected",
    "enabled": true,
    "server": "time.google.com",
    "tz label": "Europe/Amsterdam"
  },
  "OTA Status": {
    "enabled": true,
    "port": 8266
  },
  "MQTT Status": {
    "MQTT status": "connected",
    "MQTT publishes": 736,
    "MQTT publish fails": 0,
    "enabled": true,
    "client_id": "hausb",
    "keep alive": 60,
    "clean session": false,
    "base": "hausb",
    "discovery prefix": "homeassistant",
    "nested format": 2,
    "ha enabled": true,
    "mqtt qos": 0,
    "mqtt retain": false,
    "publish time boiler": 10,
    "publish time thermostat": 10,
    "publish time solar": 10,
    "publish time mixer": 10,
    "publish time other": 10,
    "publish time sensor": 10,
    "publish single": false,
    "publish2command": false,
    "send response": false
  },
  "Syslog Status": {
    "enabled": false
  },
  "Sensor Status": {
    "temperature sensors": 0,
    "temperature sensor reads": 0,
    "temperature sensor fails": 0,
    "analog sensors": 0,
    "analog sensor reads": 0,
    "analog sensor fails": 0
  },
  "API Status": {
    "API calls": 0,
    "API fails": 0
  },
  "Bus Status": {
    "bus status": "connected",
    "bus protocol": "Buderus",
    "bus telegrams received (rx)": 1278,
    "bus reads (tx)": 322,
    "bus writes (tx)": 0,
    "bus incomplete telegrams": 0,
    "bus reads failed": 2,
    "bus writes failed": 0,
    "bus rx line quality": 100,
    "bus tx line quality": 100
  },
  "Settings": {
    "board profile": "CUSTOM",
    "locale": "de",
    "tx mode": 2,
    "ems bus id": 11,
    "shower timer": false,
    "shower alert": false,
    "rx gpio": 16,
    "tx gpio": 17,
    "dallas gpio": 18,
    "pbutton gpio": 0,
    "led gpio": 2,
    "hide led": false,
    "notoken api": false,
    "readonly mode": false,
    "fahrenheit": false,
    "dallas parasite": false,
    "bool format": 1,
    "bool dashboard": 1,
    "enum format": 1,
    "analog enabled": true,
    "telnet enabled": true
  },
  "Devices": [
    {
      "type": "Boiler",
      "name": "GBx72/Trendline/Cerapur/Greenstar Si/27i",
      "device id": "0x08",
      "product id": 123,
      "version": "07.00",
      "entities": 64,
      "handlers received": "0x10 0x11 0x15 0x1C 0x18 0x19 0x34 0x2A",
      "handlers fetched": "0x14 0x16 0x33 0x26",
      "handlers pending": "0xBF 0xC2 0x1A 0x35 0xD1 0xE3 0xE4 0xE5 0xE6 0xE9 0xEA"
    },
    {
      "type": "Thermostat",
      "name": "RC300/RC310/Moduline 3000/1010H/CW400/Sense II",
      "device id": "0x10",
      "product id": 158,
      "version": "74.03",
      "entities": 48,
      "handlers received": "0x06 0xA2 0x02BA 0x02BB 0x02BC 0x02BD 0x02BE 0x02BF 0x02C0 0x031D 0x0267",
      "handlers fetched": "0x02A5 0x02B9 0x02AF 0x029B 0x02F5 0x031B 0x023A 0x0240",
      "handlers pending": "0xA3 0x12 0x13 0x0471 0x02A6 0x02B0 0x029C 0x0472 0x02A7 0x02B1 0x029D 0x0473 0x02A8 0x02B2 0x029E 0x0474 0x02A9 0x02B3 0x029F 0x0475 0x02AA 0x02B4 0x02A0 0x0476 0x02AB 0x02B5 0x02A1 0x0477 0x02AC 0x02B6 0x02A2 0x0478 0x02CC 0x02CE 0x02D0 0x02D2 0x031E",
      "handlers ignored": "0x35 0x1A 0x23 0xBF"
    },
    {
      "type": "Controller",
      "name": "BC25",
      "device id": "0x09",
      "product id": 125,
      "version": "03.05",
      "entities": 0
    }
  ]
}

I would appreciate any suggestions or ideas to resolve this issue

[proddy]

EMS-ESP was never really designed to run as multiple units, or even tested, but it should be possible. What we need to do is append the base to the Home Assistant object_ids and test it.

[tp1de]

Does using 2 different base settings in mqtt does not work?

[proddy]

Does using 2 different base settings in mqtt does not work?

it works for MQTT, but not HA Discovery which generates the unique entity IDs without the base or hostname. So to use your last issue as an example in HA sensor.analog_sensor_analog1 would have to become sensor.<base>.analog_sensor_analog1 and then it should work (in theory).

[proddy]

Code is in the PR (Michael to double check as I'm sloppy these days), then it needs testing properly

[proddy]

I'm really starting to hate this enhancement!

[proddy]

@Rainson12 think it's implemented, if you would like to test it;

[tp1de]

@proddy Have you implemented a parameter in the settings to choose new or old version to keep compatibility to older versions?

[proddy]

@proddy Have you implemented a parameter in the settings to choose new or old version to keep compatibility to older versions?

no, and not intending to either. The change is only in the MQTT Discovery config topics where the unique_id and object_id have changed to use the mqtt base and shortname. If HA users used these to reference the states of sensors then it needs changing, otherwise, I doubt anyone will notice. It shouldn't break your iobroker stuff but let me know otherwise.

As an example, looking at the topic homeassistant/switch/ems-esp/boiler_wwcirc/config

old payload in v3.5.0b10 was {"~":"ems-esp","uniq_id":"boiler_wwcirc","command_topic":"~/boiler/wwcirc","stat_t":"~/boiler_data_ww","name":"dhw Circulation active","object_id":"boiler_dhw_circulation active","val_tpl":"{{value_json.wwcirc}}","payload_on":"on","payload_off":"off","state_class":"measurement","dev":{"ids":["ems-esp-boiler"]}}

new payload is v3.5.0b11 {"uniq_id":"ems-esp_boiler_wwcirc","object_id":"ems-esp_boiler_wwcirc","command_topic":"ems-esp/boiler/wwcirc","stat_t":"ems-esp/boiler_data_ww","name":"dhw Circulation active","val_tpl":"{{value_json.wwcirc}}","payload_on":"on","payload_off":"off","state_class":"measurement","dev":{"ids":["ems-esp-boiler"]}}

[Rainson12]

HA users used these to reference the states of sensors then it needs changing, otherwise, I doubt anyone will notice. It shouldn't break your iobroker stuff but let me know otherwise.

As an example, looking at the topic homeassistant/switch/ems-esp/boiler_wwcirc/config

old payload in v3.5.0b10 was {"~":"ems-esp","uniq_id":"

thank you for the very quick implementation, this is amazing! I will flash and test the new software next weekend!

[tp1de]

@proddy

no, and not intending to either. The change is only in the MQTT Discovery config topics where the unique_id and object_id have changed to use the mqtt base and shortname

Changing unique_id is a breaking change and all HA users need to adjust their dashboards and or own automations, node-red processing etc as far as I understand. Any history / statistics data is lost as well ... or not?

Is this what you want for the very few users with multiple EMS-ESP devices or are I am wrong to consider this to be a breaking change?

[tp1de]

BTW: ioBroker can handle multiple ems-esp devices by running multiple adapter instances parallel. I hope that the API-interfaces is unchanged .... or not?

[proddy]

@proddy

no, and not intending to either. The change is only in the MQTT Discovery config topics where the unique_id and object_id have changed to use the mqtt base and shortname

Changing unique_id is a breaking change and all HA users need to adjust their dashboards and or own automations, node-red processing etc as far as I understand. Any history / statistics data is lost as well ... or not?

Is this what you want for the very few users with multiple EMS-ESP devices or are I am wrong to consider this to be a breaking change?

good point. it will remove all historic data. I think I'll need to add a new flag in the Settings to enable multiple EMS-ESP's which then adds the base to all HA unique_ids.

BTW: ioBroker can handle multiple ems-esp devices by running multiple adapter instances parallel. I hope that the API-interfaces is unchanged .... or not?

no, API stays the same. It's unique in itself because of the IP address and hostname.

[astoessel]

It is a great feature to have multiple MQTT base (I have 2 boilers in my house and it was a major issue to equip both with EMS-ESP). However, would it be possible to add a toggle in the UI so that only people using such feature would have the mqtt base "ems-esp" added in their entity names... It is a major work in HA to rebuild dashboards and automations with the new entity names for a feature not used by most people.

[proddy]

It is a great feature to have multiple MQTT base (I have 2 boilers in my house and it was a major issue to equip both with EMS-ESP). However, would it be possible to add a toggle in the UI so that only people using such feature would have the mqtt base "ems-esp" added in their entity names... It is a major work in HA to rebuild dashboards and automations with the new entity names for a feature not used by most people.

yes, you're right. an oversight from my side. I'll add the flag.

[proddy]

I added the flag, did a quick test and it's backward compatible with previous versions. Thanks @tp1de Thomas.

[tp1de]

I added the flag, did a quick test and it's backward compatible with previous versions.

Is this change already within latest b11 bin-file? If yes I can test too ...

[proddy]

It's a PR #793

[tp1de]

@proddy I did a first test with b11. For me it doesn't work the b10 entity ids are not filled with the right data in HA. There seems to be changes within object_id. I moved back to b10. I might do some further tests tomorrow.

Example: b10 vs. b11 object_id // unique_id

b10 : sensor.boiler_outside_temperature // boiler_outdoortemp
b11 : boiler_outdoortemp // boiler_outdoortemp

[proddy]

strange, it works for me. I'll test this weekend when I'm back home. Did you compile yourself or take the binary?

[tp1de]

Did you compile yourself or take the binary?

I took the binary.

[HolgerHU]

I used the binary file and it finally worked for me as far as I could tell so far. I use 2x GB172 with RC310 and HA on a Rspi 4.

[HolgerHU]

Many thanks to proddy. Great job.

[proddy]

I'll close this now