RTC, TIME, NTP on ESP32-S2

[askpatrickw]

It does not look like you can manage or use any system time without RTC.

Note: This is the lowest level dependency as I work up the stack to using the Azure IoT Library.

Board: FeatherS2
Build: 08-23-2020 Code built locally

>>> from adafruit_ntp import NTP
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "adafruit_ntp.py", line 40, in <module>
ImportError: no module named 'rtc'

>>> import time
>>> time.time()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: RTC is not supported on this board

>>> help("modules")
__main__          displayio         microcontroller   storage
_os               errno             micropython       struct
_pixelbuf         espidf            neopixel_write    supervisor
_time             fontio            os                sys
array             framebufferio     pulseio           terminalio
bitbangio         gamepad           random            time
board             gc                re                touchio
builtins          io                sdcardio          ulab
busio             ipaddress         sharpdisplay      usb_hid
collections       json              socketpool        vectorio
digitalio         math              ssl               wifi
Plus any modules on the filesystem

Espressif IDF on RTC and System Time

https://docs.espressif.com/projects/esp-idf/en/latest/esp32s2/api-reference/system/system_time.html

[anecdata]

Should this issue be closed, or is it awaiting a separate NTP API (like NINA)?

[askpatrickw]

RTC is done
Time is done
The NTP library is not done.

I'm sure other libraries need roughly the same port to support the ESP32S2's onboard wifi vs the previous model of using the esp32 wifi co-processor.

I'm not sure how the team prefers to track that.

[tannewt]

What would the NTP API include?

[askpatrickw]

The NTP library already exists, it just only works with the esp32 wifi co-processor.
https://github.com/adafruit/Adafruit_CircuitPython_NTP/blob/master/adafruit_ntp.py

It (like requests did) takes an ESP object today.

esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)

ntp = NTP(esp)
# Wait for a valid time to be received
while not ntp.valid_time:
    time.sleep(5)
    ntp.set_time()

I assume it should be modified to work with SocketPool in the same way you updated requests.

Is this a repetitive operation at this point?

Could that logic could itself be turned into a module (or library) and used across requests, ntp, and other libraries which use adafruit_esp32spi explicitly. I know some of those are board specific libraries or examples so they might not actually be impacted but it looks like many are general purpose libraries.

[anecdata]

With NINA (and the NTP library), esp.get_time() uses ESP32 Arduino core WiFi.getTime(), and that's automatcially On with NINA; the chip periodically gets the time behind the scenes, and we can query it.

For the ESP32-S2, looks like there's an IDF sequence to enable the S2 to automatically get the time in the background with SNTP:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32s2/api-reference/system/system_time.html#sntp-time-synchronization
From there, it could presumably be sync'd to time/RTC.

It's not clear to me how this works since it says nothing about the chip being set up as AP or station. Seems it would have to be a station to send out UDP requests. ESP32-S2 starts up as an AP by default in CircuitPython. A bit mysterious.

At the lowest level, maybe this would be a feature of the wifi built-in module?

[askpatrickw]

The ESP32SPI call for get_time which is called by NTP is here
https://github.com/adafruit/Adafruit_CircuitPython_ESP32SPI/blob/6fc889d6eaecd6c2a959d423c7cff60f81b8e3ad/adafruit_esp32spi/adafruit_esp32spi.py#L885

I see what you're saying now @anecdata about adding a get_time() equivalent to the ESP32S2 WIFI library.
Since the NTP library works with ESP32SPI by being passed a fully setup network would the equivalent of that be to add .get_time to Radio?

>>> import wifi
>>> dir(wifi)
['__class__', '__init__', '__name__', 'Network', 'Radio', 'radio']
>>> dir(wifi.radio)
['__class__', 'connect', 'enabled', 'hostname', 'ipv4_address', 'mac_address', 'ping', 'start_scanning_networks', 'stop_scanning_networks']

and then to pass a radio to NTP() which would then decide if its ESP32SPI or ESP32S2 Radio and call the appropriate get_time?

[anecdata]

That sounds reasonable. I'm not clear on what the API should be. CPython has a 3rd-party ntplib and maybe some other ways people do it, not sure how well they map to CircuitPython. I mainly wanted to highlight the auto-NTP feature of the ESP32 & S2, the potential issue of AP vs. station, and the possibility that auto-sync to time/RTC may be possible by default if that's desirable.

[askpatrickw]

It looks like all that functionality might be same for the ESP32 and S2
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/system_time.html

To have the ESP32S2 implementation work the same as ESP32SPI wifi.radio.get_time is for sure needed and it needs to return the time since the epoch (in seconds) . And then adafruit_ntp would to be modified to accept wifi.radio OR ESP32SPI, figure out which it has and then call the right get_time.

Might be as simple as just adding a check for Radio and using the radio.get_time the same as esp is used...

...
if "ESP_SPIcontrol" or "Radio" in str(type(esp)):
    self._esp = esp
else:
    raise TypeError("Provided object is not an ESP_SPIcontrol or WIFI.Radio object.")
...

[tannewt]

I'm a bit torn between making it all work under the hood or supporting this API: https://circuitpython.readthedocs.io/en/5.3.x/shared-bindings/rtc/__init__.html

The benefit of the RTC API would be that it would work with ESP32SPI.

[anecdata]

I've done a little digging in NINA, Arduino, and IDF. NINA does execute the SNTP init code in WiFi init:
https://github.com/adafruit/nina-fw/blob/4b244e7559e126bb5ceec1f7e46980b7a539c950/arduino/libraries/WiFi/src/WiFi.cpp#L568

I can't conclude for sure (not a C dev, and I think the LWIP stuff is opaque to us), but it seems to me that no SNTP packets are going to go out unless the user puts the device in station mode (or AP+STA). Default is AP mode.

I lean toward enabling SNTP by default in CircuitPython startup, but letting the user:

1. put the device in station mode, if/when desired

2. sync ESP32-S2 SNTP to the ESP32-S2 processor RTC using the CircuitPython RTC API, if/when desired
   (This leaves open the possibility of using an external RTC and sync-ing that with the CircuitPython RTC API, and also leaves the ESP32-S2 time/RTC to behave like any other processor until the user takes action.)

3. there could still be a direct API (exposing the IDF function) to get_time without sync-ing to RTC. This gives the common structure to ESP32SPI and the NTP library that @askpatrickw points out.

Note that NINA gives 1-second time granularity, but IDF can provide more precision (accuracy varies depending on timer source(s) chosen, and time drift varies depending on power modes).

P.S. I looked before, but could not find where the ESP32-S2 is initially put in AP mode (it seemed to happen before any CircuitPython core code runs). For low power and other reasons, we may want to think about keeping the radio off until the user directs in code (if that's even possible).

[askpatrickw]

@tannewt I thought we did have RTC... I can use it for example to do a hack NTP using requests...

    radio = wifi.radio
    pool = socketpool.SocketPool(radio)
    requests = adafruit_requests.Session(pool, ssl.create_default_context())
    response = requests.get("http://worldtimeapi.org/api/timezone/Etc/UTC")
    if response.status_code == 200:
        r = rtc.RTC()
        r.datetime = time.localtime(response.json()['unixtime'])
        print(f"System Time: {r.datetime}")
    else:
        print("Setting time failed")

@anecdata The ESP32* have a provisioning app called smartconfig. I wonder if that is the AP Mode you're seeing.

I 100% agree that having the time be correct for new programmers if they connect to wifi is the way to go. It removes an obstacle. A power use may want to turn it off, that's ok too. A good test case will be what happens if the wifi network has no internet.

[anecdata]

@askpatrickw I see AP mode mainly due to my wi-fi sniffers catching the Beacon frames that occur before CircuitPython switches to STA mode with wifi.radio.connect. But AP starting up also used to show up in the debug console before circuitpython messages.

[anecdata]

Yes, could well be the scaffolding for SmartConfig.

LOL, I hack NTP too, but using a server date header, and use that to sync to the RTC module like you do.

[tannewt]

So, I just looked at NTP and think we should implement it from Python. All it requires is UDP support on the socket (which the -S2 doesn't have yet but shouldn't be too hard.) Implementing it in Python means it'll be simple to use for all network sources. The native -S2 NTP adjusts the RTC by default which I'd rather not do. I'd like to adjust the RTC with:

import rtc
rtc.RTC().datetime = ntp.datetime

This way of setting the RTC makes it clear when time is changing and also prevents time from changing while asleep.

Here are some examples and they are very simple:
https://stackoverflow.com/questions/39466780/simple-sntp-python-script

[anecdata]

Watching delayed Deep Dive... on the issue of adjusting RTC, what's the reason not to allow setting time back? Might execute some event twice? Setting forward risks losing an event. Why not leave it to the user?

BTW, don't know about ESP32-S2, but M4 RTC calibration often does not have the range to effectively calibrate, the drifts on some chips are way outside the range (e.g., my PyPortal is off by about one second every minute).

[tannewt]

@anecdata If you have code that assumes monotonic clock math and does an unsigned subtract you'll end up with a very large, incorrect positive difference that could effectively hang everything.

I'm open to more precise timing down the road. I don't know a ton about it.

[askpatrickw]

I tried to build @tannewt's tannewt/circuitpython/tree/esp32s2_udp and test it with tannewt/Adafruit_CircuitPython_NTP/tree/raw_ntp but could not successfully build esp32s2_upd.

Although I can build circuitpython@main.

Anyone else had a go at testing these?

[tannewt]

I've handed UDP support off to @hierophect. He's made a PR here for it: #3708

[askpatrickw]

I sent a PR to Tannewt to finish up his NTP changes.
tannewt/Adafruit_CircuitPython_NTP#1

When that PR makes it to ntp/main, I'll close this. (woo hoo!)

[askpatrickw]

Since RTC and Time are completed and the recommendation is not to use NTP I'm closing this. Thanks everyone !

[geudrik]

Not to revive a dead thread for nothing.. @askpatrickw can you link to an example of using RTC to get time synched? I stumbled across this thread looking for the best way to get NTP going. I have no idea what I'm doing with my -S2 so I'm not following what I have to assume is pretty straight forward dialog here 🤯

[askpatrickw]

Best place to ask for help like that is the Adafruit discord or Forums.