Feature Request! Firmware and Data (SPIFFS) update OTA and NTP Timezone

[te260701]

Hi Khoi,
It would be great if you include OTA update as well.
I think it should be separate from Wifi page.
So, after connected to local network, show different page for update and/or custom config.

I think it will be most complete starter library for ESP. Who doesn't want to be able to update over the air :) when you have wifi.

Maybe include ntp like NTP-TZ-DST.ino too. I think it should be basic features needed for every esp.

Cheers,
Sam

[khoih-prog]

Hi Sam,

1. For OTA, I think that's a good idea and will consider to include it after some research and test. My idea is that :

• It'll be optional so that it won't use more memory if user doesn't need the feature
• The ESP, after connected to WiFi, will act as WebServer besides WebClient.
• User then connects to its local IP to access the OTA page, as well as editing previous config data: WiFi Credentials, etc.
• OTA will download from a local firmware file (to be safe and more control), not from somewhere on the Cloud.

2. Do you mean you can select NTP timezone (maybe NTP servers) by using the config page to avoid hardcoding ?

Regards,
KH

[te260701]

1. sounds good to me
2. In config page can enter timezone (jstz.js can automatic set it too, I think this even can be on the same page as wifi setting) so esp know what local time is it. In the example NTP-TZ-DST only have to config Timezone and it will even automatically fallback/forward Day light saving time. It is good for standalone device for schedule on/off, no need outside command or hub to turn on/off. Definitely, this must be optional.

[khoih-prog]

1. Will start ASAP
2. Sounds great. I'll spend more time to research, possibly need more of your feedback later.

[te260701]

I have jstz script working now. I'm not expert but I can help with web page and js if you need.
Here the setting page i'm working on. I'm making a clock lol.

[khoih-prog]

Yes please.
I'm the worst GUI and Web designer. I'll use your web design as well as js. And possibly need lot of your help to integrate.

About the jstz, is that possible to use the one from:

https://github.com/pellepim/jstimezonedetect
or
https://www.npmjs.com/package/jstz

Your screen design is so impressive.
Please post them along with guide how to integrate.
Regards,

[te260701]

About the jstz, is that possible to use the one from:

https://www.npmjs.com/package/jstz
Yes, it is the same one I use. I just link in html header.
<script src='https://cdnjs.cloudflare.com/ajax/libs/jstimezonedetect/1.0.4/jstz.min.js'></script>

Do you plan to use file system or embed the html in the code (progmem)? I think it is easier serving html from FS.

I don't know if you want to touch wifimanager part which have embedded html. Or we can have html serving from spiffs after wifi connected. Then you dont have to intergrate jstz in the code.

[khoih-prog]

I think we'll embed the html in the code progmem. This way it's easier for user to use and modify.

We'll also provide option to have html serving from spiffs after wifi connected.

We'll include both and configurable by user.

Anyway, these htmls are just String or const char[], and we can have them directly from progmem or read from SPIFFS.

Some issues for SPIFFS

1. How can we read them from SPIFFS at the first boot?
2. Is this more complicated for normal user having to put htmls into SPIFFS before using?
   In ESP_FSWebServer, user has to run curl script to load files to SPIFFS. It might be too much for normal users.
3. If we have to put htmls into SPIFFS, when? Maybe when we don't see the html files.

• To save from external files => using curl script to save files from certain directory, similar approach as FS_WebServer
• To save from memory, they're already somewhere somehow in memory, and we don't save anything and are slower to read them from SPIFFS.

4. Later, it'll more complicated if we need to change library or update htmls

What do you think?

[te260701]

1. I dont know what do you mean. Is it for wifi config page? That why I think Wifi page can left untouch (embed).
2. It is just like FSBrowser I guess. Use Sketch Data Upload Tool.
3. FSBrowser has upload file thru web interface no need curl after sketch data upload.
4. Yes, this is what I also think embed have an edge over spiffs. Update once and done. But spiffs is good when html need update no need to compile sketch and update (especially testing and fine tune interface). Just upload html via file upload page. I think maybe just embed upload page in to progmem since I dont think this will be change ever. I thought it would be easier leaving HTML out of embeded code. User can edit their own interface no need to edit code in library.

I thought leaving HTML out would be better for library. User can edit their own web interface without update code in library.

FSBrowser, in the code there is route for REST api (server.on("/all")). The web get data from send GET request url /all and server response with json data

[khoih-prog]

1. Yes, for wifi config page as well as first boot when we have no TZ info yet, we have to run js script (from somewhere) to read TZ info, then save that data to a var in SPIFFS to use later.
2. and 3. Yes. I almost forget this feature. We'll embed into progmem index.htm and edit.htm.gz instead of using Sketch Data Upload Tool

But the con is that user later has to manually and inconveniently upload files for each board. The TZ script is also very short, progmem is a good choice.

4. I totally agree. This way we'll embed into progmem index.htm and edit.htm.gz to enable upload page. Sophisticated users can modify / upload htmls as necessary.

So, we'll go with embedded option for TZ now.

Later we will add optional FSBrowser feature to let user add / modify config files and/or html / scripts for other purpose, not for TZ.

Cheers,

[mikekgr]

• OTA will download from a local firmware file (to be safe and more control), not from somewhere on the Cloud.

Dear @khoih-prog
thanks for considering to enhance your library with these add-ons!
One thing that I think will be better is:
Please give us the opportunity to have http OTA as well. It is more convenient at least to my use case!

Thanks and Best Regards,
Mike Kranidis

[khoih-prog]

Dear @mikekgr
Can we do something such as :

1. In the OTA page, user can mark either OTA from local file or from Cloud (fixed or configurable domain / IP Address which user can change)
2. If from local file, proceed to update from it
3. If from Cloud, proceed Cloud update
   I actually have experience with local file update only, and haven't done Cloud update.

Can you point to some OTA library code you're using reliably?

Regards,
KH

[mikekgr]

Dear @mikekgr
Can we do something such as :

1. In the OTA page, user can mark either OTA from local file or from Cloud (fixed or configurable domain / IP Address which user can change)
2. If from local file, proceed to update from it
3. If from Cloud, proceed Cloud update
   I actually have experience with local file update only, and haven't done Cloud update.

Can you point to some OTA library code you're using reliably?

Regards,
KH
Dear @khoih-prog
please see the bellow links ( I am not sure that this code is something that I have to be proud, but it is working fine in my cases... )
https://community.blynk.cc/t/self-updating-from-web-server-http-ota-firmware-for-esp8266-and-esp32/18544

https://community.blynk.cc/t/self-updating-from-web-server-http-ota-firmware-for-esp8266-and-esp32/18544/55

Thanks and Best Regards,
Mike Kranidis

[khoih-prog]

Dear @mikekgr

Got the OTA sample codes for ESP32 and ESP8266. Compiled OK. Will test and find a way to include into the library.
Hopefully this will not increase the code size too much to render it unusable.

You did a great work there for the community.

Thanks and Regards,
KH

[khoih-prog]

Hello @te260701 and @mikekgr

Could you have a look at the new library

https://github.com/yknivag/ESP_OTA_GitHub

Automatically update your ESP from exported compiled binaries attached to GitHub releases of your project. Supports upgrade and downgrade of sketch. SPIFFS updating not yet supported.

it permits safe OTA from GitHub and let me know your thought which one we'll like to go

1. Local file
2. Configurable or fixed url
3. GitHub

Regards,

[khoih-prog]

Hello @te260701 and @mikekgr
I just add the NTP support, you can see that TZ name is displayed in the Main and Info_Short pages

Please check and verify if it's OK. I still don't know how you'd like to use the new timezone var yet. Can you help?

Best Regards,
KH

[te260701]

Hi KH,
I see you have jstz and autodetect user timezone. Using this https://github.com/esp8266/Arduino/blob/master/cores/esp8266/TZ.h library, esp will use TZ to calculate local time (including DST).

I would like ESP to be able to turn on/off at specific local time ie turn on at 6am everyday (no need to change even when it is Daylight Saving Time).

Timezone setting page could be at the same as Wifi setting or page after successfully connect to wifi where OTA update should be.

Cheers,
Sam

EDIT: Here is stripped down version of example NTP-TZ_DST.ino

#include <TZ.h>
//#include <PolledTimeout.h>              // timer function
#include <time.h>                       // time() ctime()
#include <sys/time.h>                   // struct timeval
#include <sntp.h>                       // sntp_servermode_dhcp()

static timeval tv;
static timespec tp;
static time_t now;

#define MYTZ TZ_America_Toronto

//set timer (showtime) interval in ms
//static esp8266::polledTimeout::periodicMs showTimeNow(5000); 


void showTime() {
  //gettimeofday(&tv, nullptr);
  now = time(nullptr);
  
  const tm* localTime = localtime(&now);
  Serial.print("Local Date/Time: ");
  Serial.println(ctime(&now));
}

void setup() {
  configTime(MYTZ, "pool.ntp.org");
}

Here js to create dropdown list of timezone:

function loadTimeZone() {
	tz=["","Africa_Abidjan","Africa_Accra","Africa_Addis_Ababa","Africa_Algiers","Africa_Asmara","Africa_Bamako","Africa_Bangui","Africa_Banjul","Africa_Bissau","Africa_Blantyre","Africa_Brazzaville","Africa_Bujumbura","Africa_Cairo","Africa_Casablanca","Africa_Ceuta","Africa_Conakry","Africa_Dakar","Africa_Dar_es_Salaam","Africa_Djibouti","Africa_Douala","Africa_El_Aaiun","Africa_Freetown","Africa_Gaborone","Africa_Harare","Africa_Johannesburg","Africa_Juba","Africa_Kampala","Africa_Khartoum","Africa_Kigali","Africa_Kinshasa","Africa_Lagos","Africa_Libreville","Africa_Lome","Africa_Luanda","Africa_Lubumbashi","Africa_Lusaka","Africa_Malabo","Africa_Maputo","Africa_Maseru","Africa_Mbabane","Africa_Mogadishu","Africa_Monrovia","Africa_Nairobi","Africa_Ndjamena","Africa_Niamey","Africa_Nouakchott","Africa_Ouagadougou","Africa_PortomNovo","Africa_Sao_Tome","Africa_Tripoli","Africa_Tunis","Africa_Windhoek","America_Adak","America_Anchorage","America_Anguilla","America_Antigua","America_Araguaina","America_Argentina_Buenos_Aires","America_Argentina_Catamarca","America_Argentina_Cordoba","America_Argentina_Jujuy","America_Argentina_La_Rioja","America_Argentina_Mendoza","America_Argentina_Rio_Gallegos","America_Argentina_Salta","America_Argentina_San_Juan","America_Argentina_San_Luis","America_Argentina_Tucuman","America_Argentina_Ushuaia","America_Aruba","America_Asuncion","America_Atikokan","America_Bahia","America_Bahia_Banderas","America_Barbados","America_Belem","America_Belize","America_BlancmSablon","America_Boa_Vista","America_Bogota","America_Boise","America_Cambridge_Bay","America_Campo_Grande","America_Cancun","America_Caracas","America_Cayenne","America_Cayman","America_Chicago","America_Chihuahua","America_Costa_Rica","America_Creston","America_Cuiaba","America_Curacao","America_Danmarkshavn","America_Dawson","America_Dawson_Creek","America_Denver","America_Detroit","America_Dominica","America_Edmonton","America_Eirunepe","America_El_Salvador","America_Fortaleza","America_Fort_Nelson","America_Glace_Bay","America_Godthab","America_Goose_Bay","America_Grand_Turk","America_Grenada","America_Guadeloupe","America_Guatemala","America_Guayaquil","America_Guyana","America_Halifax","America_Havana","America_Hermosillo","America_Indiana_Indianapolis","America_Indiana_Knox","America_Indiana_Marengo","America_Indiana_Petersburg","America_Indiana_Tell_City","America_Indiana_Vevay","America_Indiana_Vincennes","America_Indiana_Winamac","America_Inuvik","America_Iqaluit","America_Jamaica","America_Juneau","America_Kentucky_Louisville","America_Kentucky_Monticello","America_Kralendijk","America_La_Paz","America_Lima","America_Los_Angeles","America_Lower_Princes","America_Maceio","America_Managua","America_Manaus","America_Marigot","America_Martinique","America_Matamoros","America_Mazatlan","America_Menominee","America_Merida","America_Metlakatla","America_Mexico_City","America_Miquelon","America_Moncton","America_Monterrey","America_Montevideo","America_Montreal","America_Montserrat","America_Nassau","America_New_York","America_Nipigon","America_Nome","America_Noronha","America_North_Dakota_Beulah","America_North_Dakota_Center","America_North_Dakota_New_Salem","America_Ojinaga","America_Panama","America_Pangnirtung","America_Paramaribo","America_Phoenix","America_PortmaumPrince","America_Port_of_Spain","America_Porto_Velho","America_Puerto_Rico","America_Punta_Arenas","America_Rainy_River","America_Rankin_Inlet","America_Recife","America_Regina","America_Resolute","America_Rio_Branco","America_Santarem","America_Santiago","America_Santo_Domingo","America_Sao_Paulo","America_Scoresbysund","America_Sitka","America_St_Barthelemy","America_St_Johns","America_St_Kitts","America_St_Lucia","America_St_Thomas","America_St_Vincent","America_Swift_Current","America_Tegucigalpa","America_Thule","America_Thunder_Bay","America_Tijuana","America_Toronto","America_Tortola","America_Vancouver","America_Whitehorse","America_Winnipeg","America_Yakutat","America_Yellowknife","Antarctica_Casey","Antarctica_Davis","Antarctica_DumontDUrville","Antarctica_Macquarie","Antarctica_Mawson","Antarctica_McMurdo","Antarctica_Palmer","Antarctica_Rothera","Antarctica_Syowa","Antarctica_Troll","Antarctica_Vostok","Arctic_Longyearbyen","Asia_Aden","Asia_Almaty","Asia_Amman","Asia_Anadyr","Asia_Aqtau","Asia_Aqtobe","Asia_Ashgabat","Asia_Atyrau","Asia_Baghdad","Asia_Bahrain","Asia_Baku","Asia_Bangkok","Asia_Barnaul","Asia_Beirut","Asia_Bishkek","Asia_Brunei","Asia_Chita","Asia_Choibalsan","Asia_Colombo","Asia_Damascus","Asia_Dhaka","Asia_Dili","Asia_Dubai","Asia_Dushanbe","Asia_Famagusta","Asia_Gaza","Asia_Hebron","Asia_Ho_Chi_Minh","Asia_Hong_Kong","Asia_Hovd","Asia_Irkutsk","Asia_Jakarta","Asia_Jayapura","Asia_Jerusalem","Asia_Kabul","Asia_Kamchatka","Asia_Karachi","Asia_Kathmandu","Asia_Khandyga","Asia_Kolkata","Asia_Krasnoyarsk","Asia_Kuala_Lumpur","Asia_Kuching","Asia_Kuwait","Asia_Macau","Asia_Magadan","Asia_Makassar","Asia_Manila","Asia_Muscat","Asia_Nicosia","Asia_Novokuznetsk","Asia_Novosibirsk","Asia_Omsk","Asia_Oral","Asia_Phnom_Penh","Asia_Pontianak","Asia_Pyongyang","Asia_Qatar","Asia_Qyzylorda","Asia_Riyadh","Asia_Sakhalin","Asia_Samarkand","Asia_Seoul","Asia_Shanghai","Asia_Singapore","Asia_Srednekolymsk","Asia_Taipei","Asia_Tashkent","Asia_Tbilisi","Asia_Tehran","Asia_Thimphu","Asia_Tokyo","Asia_Tomsk","Asia_Ulaanbaatar","Asia_Urumqi","Asia_UstmNera","Asia_Vientiane","Asia_Vladivostok","Asia_Yakutsk","Asia_Yangon","Asia_Yekaterinburg","Asia_Yerevan","Atlantic_Azores","Atlantic_Bermuda","Atlantic_Canary","Atlantic_Cape_Verde","Atlantic_Faroe","Atlantic_Madeira","Atlantic_Reykjavik","Atlantic_South_Georgia","Atlantic_Stanley","Atlantic_St_Helena","Australia_Adelaide","Australia_Brisbane","Australia_Broken_Hill","Australia_Currie","Australia_Darwin","Australia_Eucla","Australia_Hobart","Australia_Lindeman","Australia_Lord_Howe","Australia_Melbourne","Australia_Perth","Australia_Sydney","Europe_Amsterdam","Europe_Andorra","Europe_Astrakhan","Europe_Athens","Europe_Belgrade","Europe_Berlin","Europe_Bratislava","Europe_Brussels","Europe_Bucharest","Europe_Budapest","Europe_Busingen","Europe_Chisinau","Europe_Copenhagen","Europe_Dublin","Europe_Gibraltar","Europe_Guernsey","Europe_Helsinki","Europe_Isle_of_Man","Europe_Istanbul","Europe_Jersey","Europe_Kaliningrad","Europe_Kiev","Europe_Kirov","Europe_Lisbon","Europe_Ljubljana","Europe_London","Europe_Luxembourg","Europe_Madrid","Europe_Malta","Europe_Mariehamn","Europe_Minsk","Europe_Monaco","Europe_Moscow","Europe_Oslo","Europe_Paris","Europe_Podgorica","Europe_Prague","Europe_Riga","Europe_Rome","Europe_Samara","Europe_San_Marino","Europe_Sarajevo","Europe_Saratov","Europe_Simferopol","Europe_Skopje","Europe_Sofia","Europe_Stockholm","Europe_Tallinn","Europe_Tirane","Europe_Ulyanovsk","Europe_Uzhgorod","Europe_Vaduz","Europe_Vatican","Europe_Vienna","Europe_Vilnius","Europe_Volgograd","Europe_Warsaw","Europe_Zagreb","Europe_Zaporozhye","Europe_Zurich","Indian_Antananarivo","Indian_Chagos","Indian_Christmas","Indian_Cocos","Indian_Comoro","Indian_Kerguelen","Indian_Mahe","Indian_Maldives","Indian_Mauritius","Indian_Mayotte","Indian_Reunion","Pacific_Apia","Pacific_Auckland","Pacific_Bougainville","Pacific_Chatham","Pacific_Chuuk","Pacific_Easter","Pacific_Efate","Pacific_Enderbury","Pacific_Fakaofo","Pacific_Fiji","Pacific_Funafuti","Pacific_Galapagos","Pacific_Gambier","Pacific_Guadalcanal","Pacific_Guam","Pacific_Honolulu","Pacific_Kiritimati","Pacific_Kosrae","Pacific_Kwajalein","Pacific_Majuro","Pacific_Marquesas","Pacific_Midway","Pacific_Nauru","Pacific_Niue","Pacific_Norfolk","Pacific_Noumea","Pacific_Pago_Pago","Pacific_Palau","Pacific_Pitcairn","Pacific_Pohnpei","Pacific_Port_Moresby","Pacific_Rarotonga","Pacific_Saipan","Pacific_Tahiti","Pacific_Tarawa","Pacific_Tongatapu","Pacific_Wake","Pacific_Wallis","Etc_GMT","Etc_GMTm0","Etc_GMTm1","Etc_GMTm2","Etc_GMTm3","Etc_GMTm4","Etc_GMTm5","Etc_GMTm6","Etc_GMTm7","Etc_GMTm8","Etc_GMTm9","Etc_GMTm10","Etc_GMTm11","Etc_GMTm12","Etc_GMTm13","Etc_GMTm14","Etc_GMT0","Etc_GMTp0","Etc_GMTp1","Etc_GMTp2","Etc_GMTp3","Etc_GMTp4","Etc_GMTp5","Etc_GMTp6","Etc_GMTp7","Etc_GMTp8","Etc_GMTp9","Etc_GMTp10","Etc_GMTp11","Etc_GMTp12","Etc_UCT","Etc_UTC","Etc_Greenwich","Etc_Universal","Etc_Zulu"]
	const selectList = document.getElementById("timezone");
	for (var i = 0; i < tz.length; i++) {
		var option = document.createElement("option");
		if (i==0) {
			option.value = "";
			option.text = "-- Select Time Zone --";
			selectList.appendChild(option);
		} else {
			option.value = "TZ_" + tz[i];
			option.text = tz[i].replace("_","/");
			selectList.appendChild(option);
		}
	}
	if (data.timezone) { // this data.timezone is (arbitrary) data object received from ESP.  Set the list box to current setting 
		selectList.value="TZ_" + data.timezone.replace("/","_");
	}
}

BTW, I don't run js to auto detect timezone in case user already set timezone for it.