Issues v3.4.0b9

[harvey637]

Hi,
after updating from 3.4.0b8 to 3.4.0b9 system crashes repeatitly within very few (4~5] minutes.
I get a lot of network errors, possible due to reboots.
I get no scanned devices
When I revert to 3.4.0b8 everything is working fine again - but missing the new customisations length :-)
I tried severeal download of EMS-ESP-3_4_0b9-ESP32.bin, all same size 1.714 KB - same system reboots
In the logs - in the short time of running - I see ntp-requests, that are positive, date/time is correct.

If this is not a "easy" problem (like bad build, minimal typo, ...), please ask for logs and configuration - thank you!
Harvey

[proddy]

me too! Something went wrong in the build and I'm looking into what it could possibly be. Thanks for reporting it.

stay tuned....

[harvey637]

Hi - SUCCESS !!!!!
I removed ALL customisations=exclusions from "everything" -> now working!
Looks like no update for customisation from b8 -> b9
sadly this was (unclear to me what to do) mentioned in the changes:

## **BREAKING CHANGES:**

- Settings:
  - order of Boolean Format has changed in Application Settings - check your settings
  - Dallas Format setting removed. Now customize name of each Dallas sensor via the UI
- MQTT/API
  - Boiler `wwheat` renamed to `ww3wayon` [#211](https://github.com/emsesp/EMS-ESP32/issues/211)
  - Boiler `ww` tag renamed to `dhw`. Any custom Home Assistant lovelace dashboards will need updating.
  - Renamed description of `wwtapactivated` to "turn on/off DHW". Otherwise would have looked like "boiler_dhw_turn_on_off_dhw" in HA.
  - `/api/system/info` endpoint has updated keys. Now lowercase, no underscores and not capitalized. Replace "handlers" with "handlers received", "handlers fetched" and "handlers pending".
  - exclude list changed from 3.4.0b8 to 3.4.0b9 and has to be updated by user.

[harvey637]

some minor issues are left after a few minutes of correct working:
some entries cannot be excluded, e.g. "dhw 3way valve active". not really important, but as I dont have any warmwater (dhw) I would like to remove everything with "dhw", but this is very minor.

[proddy]

deleting all the customizations fixed it for me too. I'll make a change to the code to bypass older configs.

As for the dhw issue, could you copy & paste that into #339 so @MichaelDvP can look at it?

[MichaelDvP]

I havn't reproduced yet, but i think the crash happes here for numbers <10 (now single char string).
I suggest to insert before

    if (entity_id.length() < 4) {
        return;
    }

filtering all old numbers out.

[MichaelDvP]

Excluding ww3wayvalve works for me, is it a too small buffer again? It is one of the last boiler entities. @harvey637 does is work if you exclude a lower number of entities?

[proddy]

I havn't reproduced yet, but i think the crash happes here for numbers <10 (now single char string). I suggest to insert before

    if (entity_id.length() < 4) {
        return;
    }

filtering all old numbers out.

I was thinking of just renaming the json object 'exclude_entities' to something else so the old <b9 customization file is ignored..

[harvey637]

Hi,
no, ww3wayvalve and two more are different, not a length problem.
They are shown als entities in the dashboard/boiler, but NOT shown on the customisation page to be excluded.
see picture ....

You can also see the "last" line with empty name, but a value=0, nothhing to scroll down anymore there. no scrolbar.
the last line (looking completly black) CAN be selected for exclusion (than marked in pink), I was afraid to save :-)

The "old" length problem was when selecting too many entities for exclusion, these last three values can not even benn selected :-.)

[MichaelDvP]

As metioned here it's a too small buffer to create the exclude page.

[proddy]

I'll make the change now and test b8 -> b9

[MichaelDvP]

My suggestion did not work, but checking here:

                if (exclude_entity_id.is<std::string>()) {
                    new_entry.entity_ids.push_back(exclude_entity_id.as<std::string>()); // add entity list
                }

works.

[proddy]

ok, I've just renamed everything from exclude_entities to masked_entities and sending the whole mask to the front-end. Will run some tests and push a PR

[proddy]

As metioned here it's a too small buffer to create the exclude page.

another thing to consider is messagepacking the write back to help reduce the package size if the Save function becomes an issue with large data blocks. Or we make it write back a record at a time when it's modified in the table instead of one go. All things to consider for later depending on how it performs.

[MichaelDvP]

This is not the write back, this is the buffer with all data to the exclude page, it is a bit bigger than dashboard, cause it contains full-name, short-name, values and entities without values. This is already Msgpacked.

[proddy]

This is not the write back, this is the buffer with all data to the exclude page, it is a bit bigger than dashboard, cause it contains full-name, short-name, values and entities without values. This is already Msgpacked.

yes I know, just saying we may need to compress the write backs too at some point

[proddy]

I made a PR @MichaelDvP - apart from renaming the json object from exclude_entities to masked_entities so it doesn't get mixed up from previous builds (see http://ems-esp.local/api/system/customizations) I also now send the complete state mask to the UI which is prep work for #317

[proddy]

@harvey637 can you try with the latest b10?

[harvey637]

Hi,
tried b10 ...... everything is working !!!!! Very good job !!!!
I tried hiding and favourites, looks great. Saving now all entities is ok.

So all my "problems" are gone for now.

some VERY MINOR suggestions:

• in dashboard mark the lines from favourites in a very subtle different color scheme, e.g. both a little bit brighter
• the on/off color scheme has a little bit too much difference (bright/dark), but this is my very personal taste
• the popup hover help over the exlude buttons blocks the NEXT line, so I need to move mouse around to enter next line

stay healthy and warm, have a hot shower and a cool summer
Harvey

[proddy]

thanks, it wasn't hard to implement as @MichaelDvP had already done most of the heavy lifting in the backend. (thanks!)
To your further suggestions

• I was also thinking on how to mark the 'favorite' entities in the Dashboard. Maybe a light green color. I'll try and a few things
• I can use light blue instead of red (less harsh) but the color behavior is fixed to the MUI ToggleButton component so not much I can change unless writing a custom CSS
• Yes, I don't like ToolTips much and had removed them all in v3.4. But can image the buttons are confusing if you don't know what they mean. So I can either change the placement of the ToolTip so its on the left or right and not below, or remove them and add a legend at the top of the page explaining what the options are

thoughts?

[harvey637]

Hi,

• slight greenish color sounds good, same color as used in the customization heart symbol?
• hmmmm, not blue/red, I was thinking of the dashboard alternating lines for the entities (with the soon greenish favourites :-)
• ToolsTips on top of "option" caption, or use the icons in the orange "You can mark ..." line on top

by the way (minor): the word "total" is easier to recognize at a fixed position, eg. always in front. Sometimes it is at the end like in "auxiliary electrical heater energy consumption total".
so the (long) description is inconsistent to "total energy supplied cooling" which is nice in my opinion.
< total | nothing >< energy supplied | energy consumption >< heating | dhw | cooling | auxilary electrical heater | pool | ... >
Everything is ok, but a consistent order of keywords make thing even more beautiful (short names are what they are :-)

This is very minor, but possibly enhances future sorting of rows .... just an idee

Thk
Harvey

[harvey637]

UPPS - with my last saving customization ALL customisation is lost ???????
did work before, I was just adding two more favoutirs :-((

trying again ...

[proddy]

oh dear. Check every now and again with http://ems-esp.local/api/system/customizations to see if all the entities are present.

[harvey637]

Yes SIR, I will do :-)
Its only some editing, as I dont have pool, warm water, gas, only bosch 7400 air heatpump. So a lot of entities are useless FOR ME.
I have about 100 boiler entities, 50 thermostat entities.
And removed lots of about not existing stuff. So customisation ist really GREAT!

By the way: can i WRITE customisation from saved http://ems-esp.local/api/system/customizations in some way?

This is my stuff:

{
  "label": "customizations",
  "Customizations": {
    "sensors": [],
    "analogs": [],
    "masked_entities": [
      {
        "product_id": 200,
        "device_id": 56,
        "entities": [
          "08dewtemperature"
        ]
      },
      {
        "product_id": 158,
        "device_id": 16,
        "entities": [
          "03floordry",
          "08dampedoutdoortemp",
          "03floordrytemp",
          "03building",
          "03minexttemp",
          "03wwsettemp",
          "03wwmode",
          "03wwsettemplow",
          "03wwcircmode",
          "03wwchargeduration",
          "03wwcharge",
          "03wwextra1",
          "03wwextra2",
          "03wwdisinfecting",
          "03wwdisinfectday",
          "03wwdisinfecttime",
          "03hc1/climate",
          "02hc1/ecotemp",
          "02hc1/manualtemp",
          "02hc1/comforttemp",
          "02hc1/summertemp",
          "02hc1/designtemp",
          "02hc1/minflowtemp",
          "02hc1/maxflowtemp",
          "03hc1/roominfluence",
          "03hc1/roominflfactor",
          "03hc1/curroominfl",
          "02hc1/nofrosttemp",
          "03hc1/heatingtype",
          "03hc1/tempautotemp",
          "03hc1/fastheatup"
        ]
      },
      {
        "product_id": 172,
        "device_id": 8,
        "entities": [
          "03hptr6",
          "03hptr5",
          "03hpbrineout",
          "03hptr7",
          "03poolsettemp",
          "03wwtapactivated",
          "03wwsettemp",
          "08hpcoolingon",
          "08nrgsuppheating",
          "03nrgsuppww",
          "08nrgsuppcooling",
          "03nrgsupppool",
          "08hppower",
          "08hpcompon",
          "08hpactivity",
          "08hpheatingon",
          "03wwseltemp",
          "03hpwwon",
          "03hppoolon",
          "03hpbrinepumpspd",
          "03hpswitchvalve",
          "03hpbrinein",
          "03wwtempok",
          "03wwcurflow",
          "03wwstoragetemp1",
          "03wwstoragetemp2",
          "03wwactivated",
          "03wwonetime",
          "03wwdisinfecting",
          "03wwcharging",
          "03wwrecharging",
          "03wwcurtemp2",
          "03wwactive",
          "03ww3wayvalve",
          "03wwsetpumppower",
          "03wwmixertemp",
          "03wwcylmiddletemp",
          "03wwstarts",
          "03wwworkm",
          "03nrgsupptotal",
          "03wwcurtemp",
          "03wwcirc",
          "03wwcircmode",
          "03wwdisinfectiontemp",
          "03wwhystoff",
          "03wwhyston",
          "03wwchargetype",
          "03wwcircpump",
          "03wwmaxpower",
          "03wwflowtempoffset",
          "03wwcomfort",
          "03wwtype",
          "03wwseltempsingle",
          "03wwseltempoff",
          "03wwseltemplow",
          "03pumpdelay",
          "03heatingpump",
          "03fanwork",
          "03ignwork",
          "03oilpreheat",
          "03pumpmodmax",
          "03pumpmodmin",
          "03flamecurr",
          "03burnminperiod",
          "03burnminpower",
          "03burnmaxpower",
          "03boilhyston",
          "03boilhystoff",
          "03setflowtemp",
          "03setburnpow",
          "03curburnpow",
          "03burngas2",
          "03burngas",
          "03exhausttemp",
          "03boiltemp",
          "03syspress",
          "03switchtemp",
          "03heatingpump2mod",
          "03heatingpumpmod",
          "03selburnpow",
          "03tapwateractive",
          "03uptimecomppool",
          "03wwstarts2",
          "03poolstarts",
          "03uptimecompww",
          "03nrgconscompww",
          "03nrgconscomppool",
          "02auxelecheatnrgconsheating",
          "03auxelecheatnrgconsww",
          "03auxelecheatnrgconspool",
          "02reset",
          "02maintenancedate",
          "02maintenancetime",
          "02maintenance",
          "02maintenancemessage",
          "03heatworkmin",
          "03burn2workmin",
          "03burnworkmin",
          "03burnstarts"
        ]
      }
    ]
  }
}

I can see the "02" or "03 or "08" in front - very nice!
thx
Harvey

[proddy]

By the way: can i WRITE customisation from saved http://ems-esp.local/api/system/customizations in some way?

It's coming in #256

[MichaelDvP]

Marking a lot of entities as favorite crashes the esp in the sort here .
But here the customization is not deleted, after reboot the crash occures clicing on the same device in dashboard.

BTW: there is also a button on the help page to download to a file doing the same as http://ems-esp.local/api/system/customizations

[proddy]

I'll replace the sort now with Michaels code

[harvey637]

ok - wainting for the b10 :-)

also I see some problems with api calls: not all entities are received, At first sight it has nothing to do with customization,
as changing options does not modify misbehaviour.

{
  "hc1": {
    "controlmode": 1,
    "offsettemp": 0,
    "program": 0,
    "roominfluence": 0,
    "mode": 1,
    "targetflowtemp": 26,
    "heatingtype": 3,
    "summersetmode": 1,
    "summermode": 0,
    "modetype": 1
  }
}

is the api answer, but there are more entities:

will wait now for next b10 - its a pleasure helping you!
Harvey

[harvey637]

hi,
with sorting .....
I see, that after saving customizations the entities (in the ems-esp gui in correct count as customized) are displayed in one order,
after autorefresh this order is changed.
Even when pressing "refresh" button the sorting display order is toggled between two orders ....
just some observation ...

[proddy]

yes, that is a bad side affect from my sorting I added, I'm fixing it...

[harvey637]

Hi,
running latest b10 (after "remove (hidden)" PR) - nothing to complain (but the recreation of the customization :-)
Thank You,
relaxed waiting for next dev-release ...
cu
Harvey

[proddy]

and thanks for the great feedback and reviewing the changes. nice collaboration!