Saving "Customizations" fails with > 63 entities selected

[mvjt]

Bug description
Using EMS-ESP Version v3.4.0b2

Steps to reproduce
Disabling a large set of parameters in the Customization page. Right now, I can easily trigger the 400 error. I've disabled 62 parameters and if I disable one more I get Error 400 on save.

Expected behavior
Save successfully

Screenshots

Device information
{
"System": {
"version": "3.4.0b2",
"uptime": "000+08:01:05.068",
"uptime (seconds)": 28865,
"freemem": 113,
"reset reason": "Software reset CPU / Software reset CPU",
"temperature sensors": 0
},
"Network": {
"connection": "Ethernet",
"hostname": "ems-esp",
"MAC": "8C:CE:4E:95:50:97",
"IPv4 address": "10.10.10.121/255.255.255.0",
"IPv4 gateway": "10.10.10.1",
"IPv4 nameserver": "1.1.1.1"
},
"Status": {
"bus status": "connected",
"bus protocol": "Buderus",
"bus telegrams received (rx)": 136258,
"bus reads (tx)": 7767,
"bus writes (tx)": 0,
"bus incomplete telegrams": 1,
"bus reads failed": 0,
"bus writes failed": 0,
"bus rx line quality": 100,
"bus tx line quality": 100,
"MQTT status": "connected",
"MQTT publishes": 9660,
"MQTT publish fails": 0,
"temperature sensors": 0,
"temperature sensor reads": 0,
"temperature sensor fails": 0,
"analog sensors": 0,
"API calls": 3,
"API fails": 0
},
"Devices": [
{
"type": "Boiler",
"name": "IVT Enviline/Compress 6000AW/Hybrid 7000iAW/SupraEco/Geo 5xx (DeviceID:0x08, ProductID:172, Version:02.03)",
"entities": 59,
"handlers received": "0xC2 0xD1 0xE3 0xE4 0xE5 0xE9 0x0494 0x0495 0x048F",
"handlers fetched": "0x14 0xE6 0xEA 0x048D 0x048A",
"handlers pending": "0x10 0x11 0x15 0x1C 0x18 0x19 0x1A 0x35 0x16 0x33 0x34 0x26 0x2A"
},
{
"type": "Thermostat",
"name": "Rego 2000/3000 (DeviceID:0x10, ProductID:172, Version:02.03)",
"entities": 31,
"handlers received": "0x06 0x02A5 0x02A6 0x031D",
"handlers fetched": "0x02B9 0x02AF 0x029B 0x0471 0x02BA 0x02B0 0x029C 0x0472 0x02F5 0x023A 0x0240",
"handlers pending": "0xA3 0xA2 0x12 0x02A7 0x02BB 0x02B1 0x029D 0x0473 0x02A8 0x02BC 0x02B2 0x029E 0x0474 0x02A9 0x02BD 0x02B3 0x029F 0x0475 0x02AA 0x02BE 0x02B4 0x02A0 0x0476 0x02AB 0x02BF 0x02B5 0x02A1 0x0477 0x02AC 0x"
},
{
"type": "Gateway",
"name": "KM200/MB LAN 2 (DeviceID:0x48, ProductID:189, Version:04.07)",
"entities": 0
},
{
"type": "Controller",
"name": "Rego 3000 (DeviceID:0x09, ProductID:240, Version:38.03)",
"entities": 0
},
{
"type": "Connect",
"name": "Wireless sensor base (DeviceID:0x50, ProductID:236, Version:03.03)",
"entities": 0
},
{
"type": "Unknown",
"name": "rf room temperature sensor (DeviceID:0x40)",
"entities": 0,
"handlers received": "0x0435"
}
]
}

[proddy]

Roughly how many entities?

[mvjt]

128 (error 400 is generated when I've disabled > 62

[proddy]

I've reproduced. Changing _exclude_entities_handler.setMaxContentLength to 1024 in WebCustomizationService,cpp fixes the 413 (too large) error. Need to dive into why the webserver is throwing the 400.

Out of interest, why are you excluding so many entities?

[mvjt]

Hi,Because I want to avoid excessive entities in HA and avoiding unnecessary load on the MQTT broker. Sent from ProtonMail for iOS På sön, jan. 30, 2022 vid 10:25, Proddy ***@***.***> skrev: I've reproduced. Changing _exclude_entities_handler.setMaxContentLength to 1024 in WebCustomizationService,cpp fixes the 413 (too large) error. Need to dive into why the webserver is throwing the 400. Out of interest, why are you excluding so many entities? —Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: ***@***.***>

[MichaelDvP]

The http:400 is the json limit here, Tested this with exclude entities by name, where the error was triggered with 60 exclusions. (Tested with limit 2048 (also for maxContentLength) and 80 exclusions, i don't have more),

[proddy]

is there do you think to set the buffer in WebCustomizationService.cpp ?

I'm doing a lot of travel these days for work so haven't started on the Customization improvements. If you have an implementation using the short name and can prefix a Hex value for the flags, and rename exclude_entities to something generic then I can work on building the Web table where you can toggle flags on/off.

[MichaelDvP]

is there do you think to set the buffer in WebCustomizationService.cpp ?

From WebCustomizationService.cpp the maxContentLengthis set (http:413 if to small), but maxJsonBufferSize is declared const and gives http:400. I have set both to 2048 now.

If you have an implementation using the short name and can prefix a Hex value for the flags, and rename exclude_entities to something generic then I can work on building the Web table where you can toggle flags on/off.

I'll add the prefix. Suggest to make it 2char as hex-byte, giving some more room for flags in future. Giving a entity_id like 03hc2/tempautotemp and add flags to dv.state.
I'll make a feature branch, then you can add the web table, push it here or to my github?

[proddy]

i think push it directly to v3.4 if it works (I'm sure it will) and then we can release 3.4

[MichaelDvP]

With extra bits and renaming i have to touch a lot of files. The readonly is not easy to handle the tags, e.g. excluding hc2/daytemp, but not hc1/daytemp and daytemp with default hc (id =-1). The command list shows without tag, so call thermostat commands shows all.
I have now four flags to high nibble of State:

    enum DeviceValueState : uint8_t {
        DV_DEFAULT           = 0,        // 0 - does not yet have a value
        DV_ACTIVE            = (1 << 0), // 1 - has a validated real value
        DV_HA_CONFIG_CREATED = (1 << 1), // 2 - set if the HA config topic has been created
        DV_HA_CLIMATE_NO_RT  = (1 << 2), // 3 - climate created without roomTemp
                                         // high nibble as mask for exclusions
        DV_WEB_EXCLUDE      = (1 << 4),  // 16 - not shown on web
        DV_API_MQTT_EXCLUDE = (1 << 5),  // 32 - not shown on mqtt, API
        DV_READONLY         = (1 << 6),  // 64 - read only
        DV_FAVORITE         = (1 << 7)   // 128 - sort to front

but reserved a byte for flags in webpage/store, so we can expand another 4 bits without breaking updates.
The only modification to cusutomizations-webpage for now is sending the name with fixed flags instead of number.
const exclude_entities = deviceEntities.filter((de) => de.x).map((new_de) => "07" + new_de.s);
where 07 matches WEB/MQTT/READONLY flags. The id-number is obsolete, but still generated.

I'll make a PR (after a bit more testing/fixing), better you take a look first. The current beta is pretty stable with low complains, i don't want to break something short before release.

BTW: I've changed AsyncWebServer to set maxJsonBufferSize individual, so we can use in WebCustomizationServer:

    _exclude_entities_handler.setMaxContentLength(2048);
    _exclude_entities_handler.setMaxJsonBufferSize(2048);
    server->addHandler(&_exclude_entities_handler);

[proddy]

excellent! That was the way I would have coded it.

[harvey637]

Hi,
after deleting and recreating customisations there are some entities, that are not shown in the customisation page:
my personal boiler (bosch 7400) shows:
dhw 3way valve active
dhw starts
dhw active time

Also the last line of the customisationlooks a bit strange: no (empty) entity name, no (empty) code, but value = 0.

Because of the three entities not shown they cannot be excluded from data. Not a big problem, but ...
thank you!
harvey

[MichaelDvP]

Ah, ok, than it's the json-buffer to show the exclusion page. We need to add a X here (XXXLARGE, 20k instead of 16k).

[proddy]

I think this is fixed in b10, checking in soon