create safebuild app to fit into factory partition to give ESP32 more flash memory

[proddy]

Originally posted by @MichaelDvP in #22 (comment)

Tasmota have introduced a "safeboot" shortly. That's (as far as i understand) a smaller OTA_0 partition, holding only a minimal image with OTA upload and a larger OTA_1 for the firmware without upload option, only a switch to reboot to safeboot partition. Maybe that's a way.

[proddy]

See https://tasmota.github.io/docs/Safeboot/ and https://github.com/arendst/Tasmota/blob/development/partitions/esp32_partition_app2880k_fs320k.csv for examples.

The idea is to create a smaller firmware image that is half the size and fits into 830Kb (it's now 1.9Mb). This could be without the web so only boot up with the telnet console for emergency OTA uploading.

[MichaelDvP]

OTA and uploading emsesp-firmware works only if executed in the small partition and write to the large.
The small should hold a minimal esp8266-react with only enabled OTA and uploading, don't know how large this is.
In emsesp the firmware upload should be replace with a reboot to small partition.
It's not emergency OTA, it's the only OTA from there.
For changing the small firmware USB-flashing should be enough.

[MichaelDvP]

I've made a test by removing all project and mqtt stuff from ems-esp and flash to a factory or a OTA partition.
Using asymetric OTA works, then you can flash from small ota_0 to large ota_1 the ems-esp full software and from ems-esp you can flash the small loader. If using factory the small partition can only flashed via USB. To switch between partions i've added a button.

And this is the other partiton with ems-esp.

[proddy]

that's friggin' awesome! you managed to create a bare-bone EMS-ESP with the WebUI. Other than MQTT and the project stuff what else did you have to exclude to fit this into the safe-boot partition?

Also what happens when you switch the partition? I've been following and supporting platformio/platform-espressif32#785 (comment)

[MichaelDvP]

I've removed as much as possible, compiled, checked the size and set factory partition to this size (0x14000, it fits also in 0x130000).
Check here: https://github.com/MichaelDvP/EMS-ESP-Loader
Changing partition in ems-esp to reboot to factory or ota partition scheme is:

void RestartService::partition(AsyncWebServerRequest * request) {
    const esp_partition_t * partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_APP_FACTORY, NULL);
    if (partition) {
        esp_ota_set_boot_partition(partition);
    } else {
        const esp_partition_t * partition1 = esp_ota_get_next_update_partition(NULL);
        esp_ota_set_boot_partition(partition1);
    }
    request->onDisconnect(RestartService::restartNow);
    request->send(200);
}

Also what happens when you switch the partition?

If target the partition is flashed, it reboots to to this software, if target partition is blank, esp crashes.
With actual symetric partitions i can have v3.5.0 in one and v.3.4.4 in the other and switch between (when adding a switch to v3.4.4).

[MichaelDvP]

Also what happens when you switch the partition?

If target the partition is flashed, it reboots to to this software, if target partition is blank, esp crashes.

Crashes only with two ota partitions and one blank, with factory partition and blank ota the reboot fallback to factory, but i've also added some code in loader to avoid rebooting to erased ota-partition.

I'd like to add a reboot to EMS-ESP-Loader button(factory partition) to dev now. The button is only shown if a factory partition exists, not in standard ota_0/ota_1 scheme. It helps me testing with official dev and other user do not see it. Also in future we can say v3.5.0b9 and up are able to reboot to factory. Is this ok?

[proddy]

yes this is good. It's a nice addition. We would need to give bbqkees instructions on how to load the two images

[MichaelDvP]

For the 16MB it is not needed. For C3 with 4MB in very near future. There are two ways:

• use espressif flasher with loader on 0x10000 and ems-esp on 0x150000 (or generate once a combined image with this flasher)
• look at https://github.com/MichaelDvP/EMS-ESP-Loader/releases/tag/Releases use special flasher with factory partition and flash loader, then boot loader and use web upload for ems-esp.

[proddy]

@MichaelDvP I forgot about this. Shall we make this the standard for v3.5.0? So a smaller OTA partition with the stripped EMS-ESP.

[MichaelDvP]

I think as long as it fit's use the sysmetric OTA1/2. For v3.6 you have to decide how to manage.
Asymetric OTA1/OTA2 could be flashed from the other partition,
Factory/OTA1 can only update the OTA1 from small image in factory partition. he factory image has to be flashed with usb.
Actually the reboot checks for a fatory partition to allow reboot to loader, but we can also add a reboot to other partition button to change between ota1/2.

[proddy]

v3.5.0 just about fits. For v3.6.0 we'll need a version for the 4MB flashes (when Turkish and Czech languages are added). I've set the milestone to 3.6.0

[tp1de]

I can confirm that Michael's solution works great. I can't judge possible risks with the 2 steps flashing, but it is very convenient to handle.

For 3.6.0-dev.7

[proddy]

@MichaelDvP could we make this the defacto standard for our next version? It would mean

• creating a repo for EMS-ESP-Loader and copying over your repo
• changing the theme and some text in the Web so people know it's the minimized version
• providing a new partition file partitions.esp32.4MB.bin
• writing instructions on how to flash and upgrade
• helping out BBQKees so all future versions are configured this way

anything else I missed?

[MichaelDvP]

@bbqkees

Can you elaborate a bit further on this?

About the minimal image:
I used ems-esp v3.5.0 and removed all that is not used for OTA/upload. What is implemented now is: Network, AP, Security, NTP and System. Gateways E32/S32 are detected automatically, board profile is not used. (I think for first flashing it's ok to use wifi/AP but we should read board profile from file once set by ems-esp to allow easy update for custom users.) The implemented settings are used by minimal and ems-esp, No reconfigure needed if you change between images(partitions).
BTW: imagine: all ems-stuff, telnet, syslog, mqtt,dallas, analog is 700k and we have now 700k extra for ems-esp, that is a lot of possible additions,

About partitions:
ESP32 supports two type of partitions:

• APP (code execution)
• Data (no code execution
  Subtypes of APP are:
• Factory (no write from code, only from usb)
• OTA (write from code possible)
  If there is more than one APP partition, a data-partition of subtype "otadata" defines, what partition is booted.
  If otadata points to an empty partition we have bootloop.

First approach was to put loader in factory and add code to v3.5. to reboot to factory partition (if exist). The loader check if the OTA-partiton starts with a reset-vector or is erased to allow reboot to ota-partition (without new writing). Uploading a file writes the partition and reboots to this partition if write was successful. If something goes wrong (wrong chiptype, to large image, etc) the ota is erased and no reset vector is written, so it's not possible to boot to the empty partition. In ems-esp v3.5. reboot to other partition is only possible if this is factory subtype. In this approach we can not update the loader via OTA, only with new USB-flashing.

I've tested with with two ota partitions. Then we can write the ems-esp-image from loader and a new loader from ems-esp. This also allows to flash older(<= 3.4.4) images (no button to boot to other partition), because we can upload the loader image and esp32 changes to this partition after upload. But we have to add code to deny a reboot if the small ota-partiton is empty (after trying to upload a full image). This is already implemented since v3.5.1 restartservice.cpp. I think it's all save now.

@proddy

creating a repo for EMS-ESP-Loader and copying over your repo

If you create the repo, i can push or PR the code

changing the theme and some text in the Web so people know it's the minimized version

You can see it clearly because dashoard/settings/help are missing, maybe change the color of the ems-esp logo to red should be enough.

providing a new partition file partitions.esp32.4MB.bin

Do you want to modify the existing ems-esp-flasher to only flash the loader on 4MB esp?. Then we need to add a warning, that flashing v3.5 direct on 4MB is not possible anymore and has to use the loader. (16MB is still symmetric without loader).

• modify the 4MB-partitons for v3.6 to

# Name,   Type, SubType, Offset,   Size,    Flags
nvs,      data, nvs,     0x9000,   0x5000,
otadata,  data, ota,     ,         0x2000,
app1,     app,  ota_1,   ,         0x140000,
app0,     app,  ota_0,   ,         0x2A0000,
spiffs,   data, spiffs,  ,         64K,

to allow the compiler/linker to check size.

[proddy]

creating a repo for EMS-ESP-Loader and copying over your repo

If you create the repo, i can push or PR the code

https://github.com/emsesp/EMS-ESP-Loader

done. Thanks @MichaelDvP

[MichaelDvP]

https://github.com/emsesp/EMS-ESP-Loader
done. Thanks @MichaelDvP

Tried to push: Permission to emsesp/EMS-ESP-Loader.git denied to MichaelDvP.

Please fetch the code from https://github.com/MichaelDvP/EMS-ESP-Loader then i'll make PRs for future changes.

[proddy]

Ok, strange though since we should both the same permissions.

[MichaelDvP]

Now it works. Actually there is no build action. Question if we should build esp32, esp32s2 and esp32c3, because these are the chips with 4MB flash. But if users are able to compile the c3 code, they also can compile the loader, so a esp32 bnary should be enough?

[bbqkees]

For the loader it's best to keep it as simple as possible. 99.9% will need the ESP32 one. The loader concept may already be hard to grasp for the non-technical people. So only the ESP32 bin would suffice here. For the very few who use S2 or C3 they have to build the firmware themselves anyway, so they can also build the loader. Op vr 24 mrt 2023 om 08:50 schreef MichaelDvP ***@***.***>:

…

Now it works. Actually there is no build action. Question if we should build esp32, esp32s2 and esp32c3, because these are the chips with 4MB flash. But if users are able to compile the c3 code, they also can compile the loader, so a esp32 bnary should be enough? — Reply to this email directly, view it on GitHub <#608 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSGZAUQTUDAW4R67KEGI6DW5VG6BANCNFSM57VRCPUA> . You are receiving this because you were mentioned.Message ID: ***@***.***>