Alarms, light sleep, and deep sleep

[tannewt]

After we have #2795 we'll have the mechanic for waking on a variety of sources.

We should add the ability to set what alarms should cause a reload when the user code is finished. This function should validate that the alarms can be run at the lowest power setting.

All RAM and program state can be lost in this sleep state. So, we also need a way to read the wake up cause from user code. CircuitPython should also start up faster when woken by an alarm by skipping things like the safe mode reset delay.

This should work the same when on USB except that the sleep will be WFI only so that USB still works. Any USB writes will reload and clear alarms. Otherwise the alarms will cause a reload with the appropriate state.

[tannewt]

Below is an example. It is the deep sleep variant of this light sleep example.

import alarm.pin
import alarm.time
import sleepio

alarm = sleepio.wake_alarm

pin0_alarm = alarm.pin.PinLevelAlarm(board.IO0, True, enable_pull=True)
time_alarm = alarm.time.TimeAlarm(time.monotonic() + 10.0)

# do something based on what woke us up
if alarm == pin0_alarm:
    print("pin change")
elif alarm == time_alarm:
    print("time out")

sleepio.set_reload_alarms(pin0_alarm, time_alarm)

# Finish code.py which will start the alarms. When on USB it will lead to a light sleep
# to preserve the USB connection and reload when an alarm goes off. When USB is not
# connected, it will deep sleep and restart CircuitPython on wakeup.

Updated for newer sleepio API.

[dhalbert]

@tannewt and I discussed this a few days ago, and I've been working on a further refinement of the API.

I will use the terminology "fired" to indicate an alarm "triggering" or "going off". "Going off" is a bit confusing, since it could sound like "turning off".

I would like to avoid talking about "deep sleep" and "light sleep" in the API, since there are circumstances when the implementation might not actually want to sleep, such as when connected to USB and enumerated. Instead, there are different circumstances in which an alarm can fire. CPy can take advantage of waiting for an alarm and do a light or deep sleep, but it doesn't have to.

I have added an alarm queue mechanism, instead of having supervisor.runtime hold information about which alarms have triggered. I think this will be helpful for event loops that want to poll for alarms and other events.

I have also made the alarm objects more autonomous. In @tannewt's proposal above, the program enables the alarms by informing supervisor.runtime(), by .append() or by .wait_for_alarm(). Instead, I have the alarms manage their own enabling and disabling.

An alarm is not automatically enabled when it is instantiated. Instead, it is explicitly enabled. Once an alarm fires, it becomes disabled, until it is explicitly re-enabled again.

An alarm can fire during normal program operation, during a program-initiated sleep, or after a program has shut down. When an alarm fires, it is recorded in an event queue. For now is just a list, since we don't have a native queue type. The default event queue is supervisor.runtime.alarms. (There might be issues here about atomicity, such as an alarm firing when another alarm is being enqueued, so we might need to add a more specialized queue type.)

Here are the detailed cases when an alarm can fire:

1. During normal program operation, or during a program-initiated sleep. An alarm can fire during normal running, without any sleeping or waiting. There are several subcases:
   a. While program is running, doing other things. The alarm is enqueued.
   b. During supervisor.runtime.wait_for_alarm(). The alarm is enqueued and .wait_for_alarm() returns. CPy can take advantage of the wait and go into a light sleep.
   c. During time.sleep(), which right now does a light sleep. The alarm is enqueued, but time.sleep() does not finish early. This is consistent with the signal-handling behavior of time.sleep(). Since Python 3.5, an interrupt does not shorten time.sleep(), unless the signal handler throws an exception. We don't have alarm handlers, so there's no exception, though throwing an exception on an alarm could be a future feature.
2. After program shutdown. Program shutdown occurs when we fall off the end of the program or call sys.exit(). The alarm is enqueued when the program restarts on the default queue. An open question is we only do this on sys.exit(0), or whether the integer exit code could be saved in backup RAM and be retrievable by the restarted program. That is, does sys.exit(1) not enable shutdown alarms because it's an error exit?

import supervisor
import alarm.pin
import alarm.time

# Did we restart because there was an alarm?
# supervisor.runtime.alarms is a list of alarms that have fired. It is the default event queue.
alarm = supervisor.runtime.alarms.get(0, None)

# Create some alarms. They are not yet enabled.

# trigger can be HIGH, LOW, RISING, FALLING (or RISE and FALL ?)
pin5_alarm = alarm.pin.PinAlarm(board.IO5, trigger=alarm.pin.HIGH, pull=digitalio.Pull.UP)
pin6_alarm = alarm.pin.PinAlarm(board.IO6, trigger=alarm.pin.RISING, pull=digitalio.Pull.UP)

# Arg to TimeAlarm is the number of seconds after the alarm is enabled (not the time at instantiation),
# or it's a struct time, designating a date/time in the future.
time_alarm = alarm.time.TimeAlarm(time.monotonic() + 60)

# Check to see whether an alarm caused us to wake up from shutdown
if alarm == pin5_alarm:
    print("pin5 change")
elif alarm == time_alarm:
    print("time out")

# Enable a pin alarm after the program finishes. This is the deep sleep case.
# Not enabled during normal program flow. Will enqueue on the default queue.
# Passing False disables the alarm if it is enabled.
pin5_alarm.enable_after_shutdown(True)

# Enable alarm now. It will trigger during normal operation, during time.sleep(), or during
# supervisor.runtime.wait_for_alarm(). Enqueue on the default queue when fired.
time_alarm.enable_now(True)

# Enable alarm now, but enqueue on an alternate queue when fired.
my_queue = []
pin6_alarm.enable_now(True, queue=my_queue)

# Wait for any .enable_now alarm.
supervisor.runtime.wait_for_alarm()
# Fetch the alarm that fired.
alarm = supervisor.runtime.alarms.get(0, None)
my_alarm = my_queue.get(0, None)


# Any .enable_now alarm can fire during this time.
time.sleep(5)

.enable_now() and enable_after_shutdown() could possibly be properties, but then it becomes awkward to specify an alternate queue for .enable_now. So for consistency I have made them methods.

[tannewt]

What is the difference between a pin level alarm and a pin edge alarm? Won't a low level alarm fire at the same time as falling edge would?

My biggest concern is having each alarm enabled independently of each other and independent of use. This prevents validating that 1) the alarms can all be set at the same time and 2) they can all work at the desired sleep level.

1 is important because pin alarming might not actually be independent. On the ESP32-S2 (reference) only one pin can be checked independently with ext0. ext1 can check multiple pins but they are either an NAND ALL_LOW or OR ANY_HIGH. This means you have to validate alarms as a group.

2 is important because some alarms may only work in higher sleep states. For example, we can check pin levels independently on the S2 if the GPIO peripheral is active and can trigger interrupts.

I prefer preserving sleepio as a separate module and indicator of what ports and boards support these new APIs. Note that the APIs in the issues are a bit old and my sleepio branch has a newer API that is more sleepio centric.

My proposal for listening for a set of alarms while running code would be to add a call similar to sleep_until_alarm that takes in alarms but returns an iterator. These iterators would act as the queue. They'd return None if no alarm has fired and StopIteration when all alarms have. You can call it multiple times to get multiple iterators.

However, I'm not certain this is what we'd need for asyncio integration. I'd much rather focus on light and deep sleep only.

[ladyada]

fyi some chips have irq-on-level and some have irq-on-edge - i suppose it only really matters if the irq pin is LOW on entry-to-sleep, if its falling-edge it would not wake immediately. If its level it would exit sleep immediately.

[microdev1]

I have lost track of the changes in sleepio and it seems like the api has become quite complex.

[dhalbert]

@tannewt and I talked again today, and revised the example in @tannewt's comment above.

A few more notes:

Re rising/falling edge detection: Both level and transition can often be detected. I have seen both, and I even see "transition either direction".

Waking from deep sleep on pin change: On a number of chips, this is implemented as "tamper detection". On nRF, it is just called "DETECT") Some chips just do level detection (SAMD51, nRF). SAMD51 only does it on a few pins (5). STM does only rising/falling, it appears (EXTI registers). nRF has an "or" kind of function (LDETECT), vaguely like the ESP32, maybe. On light sleep, most chips can do level or transition.

MicroPython does interrupt handlers for pins like this:

Pin.irq(handler=None, trigger=Pin.IRQ_FALLING | Pin.IRQ_RISING, *, priority=1, wake=None, hard=False), where wake can be (to quote):

wake selects the power mode in which this interrupt can wake up the system. It can be machine.IDLE, machine.SLEEP or machine.DEEPSLEEP. These values can also be OR’ed together to make a pin generate interrupts in more than one power mode.

[dhalbert]

I am now working on this and revising the API toward the revised example above. I am making some changes, and I have some suggested changes that I would welcome comments on:

1. I renamed sleepio to just sleep, because an alarm is not necessarily pin-based. We have a lot of *io modules, but not all are (e.g. rtc).
2. sleep.reset_reason and sleep.last_wake require that the module dictionary be in RAM rather than flash, since their values change. We don't do that much: I think the only other example is _bleio.adapter, which is changeable only for historical reasons (it used to be fixed). On smaller boards we might want sleep, but RAM is short, so I'm wondering if these should become functions instead. For example sleep.get_reset_reason() and sleep.get_last_wake().
3. sleep.reset_reason is more interesting to the user that just to be used via sleep, and I'm thinking it could be in microcontroller or microcontroller.cpu. Even without using sleep, a user might want to know the reason for a reset.
4. If alarms are only used in sleep, then the top-level alarm module could be moved into sleep.alarm. It might be sleep.alarm.pin and sleep.alarm.time, or the classes might simply be at a higher level, e.g. sleep.alarm.PinAlarm or even sleep.PinAlarm. @tannewt I know this might be an issue if some alarms are port-specific, since you like a module not to have optional parts.

[tannewt]

I am now working on this and revising the API toward the revised example above. I am making some changes, and I have some suggested changes that I would welcome comments on:

Thanks for picking this work up! I'm excited to try it.

1. I renamed `sleepio` to just `sleep`, because an alarm is not necessarily pin-based. We have a lot of `*io` modules, but not all are (e.g. `rtc`).

I worry that just sleep will conflict with time.sleep. lowpower is an option too. I don't mind having *io though because it increases the chance of the module name being unique and also brands it as CircuitPython.

2. `sleep.reset_reason` and `sleep.last_wake` require that the module dictionary be in RAM rather than flash, since their values change. We don't do that much: I think the only other example is `_bleio.adapter`, which is changeable only for historical reasons (it used to be fixed). On smaller boards we might want `sleep`, but RAM is short, so I'm wondering if these should become functions instead. For example `sleep.get_reset_reason()` and `sleep.get_last_wake()`.

I don't buy this argument. Each additional entry in the module table is only 8 bytes. There are only a few existing APIs for the module so it'd only be ~40 bytes or so. That isn't worth an inconsistent API.

3. `sleep.reset_reason` is more interesting to the user that just to be used via `sleep`, and I'm thinking it could be in `microcontroller` or `microcontroller.cpu`. Even without using `sleep`, a user might want to know the reason for a reset.

I'd prefer it in the new module so that we don't need to implement it for all platforms and can convey when it is implemented. Importing a native module to get access to it is low cost.

4. If alarms are only used in `sleep`, then the top-level `alarm` module could be moved into `sleep.alarm`. It might be `sleep.alarm.pin` and `sleep.alarm.time`, or the classes might simply be at a higher level, e.g. `sleep.alarm.PinAlarm` or even `sleep.PinAlarm`. @tannewt I know this might be an issue if some alarms are port-specific, since you like a module not to have optional parts.

I have no problem with the nested modules because they can fail on import. I don't like the higher level classes though because it doesn't fail on import and isn't obvious to convey when something is supported. Having multiple top level modules is the simplest and most consistent option.

[timonsku]

Alarms sounds a lot like interrupt handlers. Even if they don't use interrupts I think it would be a bad idea if they are exclusive to the sleep modules API, especially if true interrupt support is added later on, this could become confusing.
I would very much like to see interrupts outside of waking from sleep.

[dhalbert]

Alarms sounds a lot like interrupt handlers. Even if they don't use interrupts I think it would be a bad idea if they are exclusive to the sleep modules API, especially if true interrupt support is added later on, this could become confusing.
I would very much like to see interrupts outside of waking from sleep.

That was my proposal in this comment: #2796 (comment). I agree we could add this later, in some way, and that's a motivation for creating a separate top-level alarm module, instead of putting it inside sleep. So I will leave it that way.

[dhalbert]

@tannewt I had one more idea this morning, which is not to move alarm inside sleepio, but instead, just to dispense with sleepio, and put the sleepio functions as functions on alarm. E.g., alarm.reset_reason, alarm.last_wake, alarm.sleep_until(), etc.

Otherwise what I see is that sleepio is almost completely dependent on the existence of alarm, and really serves no purpose otherwise. The opposite is not true as @PTS93 points out.

These are all cosmetic and do not affect me proceeding on the implementation.

[tannewt]

@dhalbert merging sleepio into top level alarm is ok with me!

[timonsku]

That sounds nice to me too :)

[deshipu]

Looking from the point of view of implementing a main loop for async, would there be a way to wait (sleep until) on several different conditions at once? For example, a timeout, a pin raise, a socket connection, a user input on STDIO and a UART transmission? Because that's what we would ultimately need.

[microdev1]

@deshipu Yes, multiple alarms/conditions can be enabled at once. Also, there is provision to get the alarm that caused wake-up + parameters associated with it.

[deshipu]

What I'm worried about the most is that there won't be alarms available for some of the conditions we want to wait on. But I suppose we can just keep adding new kinds of alarms?

[microdev1]

Alarms are implemented as separate modules and there availability will vary on a port specific basis.

[deshipu]

Another thing is that we want to be able to wait on an explicit list of alarms, not necessarily on all alarms that are enabled. That's because there may be user alarms created and enabled independent from the async main loop, that we would need to ignore somehow.

[dhalbert]

There will be alarm.sleep(alarm1, alarm2, ...). That will do a light sleep until one of those alarms is triggered.

For deep sleep, alarm.wake_after_exit(alarm1, alarm2, ...) will awake and restart code.py when one of the alarms is triggered.

The names above are tentative; I'm still thinking about the best naming.

For an async loop, I proposed above that alarms that trigger will put themselves on a queue, but the API and semantics are not yet specified. You don't have to be asleep for that to happen. We can open a new issue for that. It will not be in the first version of this: our short-term highest priority is deep sleep on the ESP32-S2, for battery reasons.

[dhalbert]

Ligh sleep example, copied from #2795. Exact function names may change, and sleepio has been eliminated in favor of putting everything in alarm.

Here is an example:

import sleepio
import time
import alarm.pin
import alarm.time

pin0_alarm = alarm.pin.PinLevelAlarm(board.IO0, True, enable_pull=True)
time_alarm = alarm.pin.TimeAlarm(time.monotonic() + 10.0)

while True:
    # start with a light sleep until an alarm triggers.
    alarm = sleepio.sleep_until_alarm(pin0_alarm, time_alarm)
    # do something based on what alarmed
    if alarm == pin0_alarm:
        print("pin change")
    elif alarm == time_alarm:
        print("time out")

[Example updated for newer sleep API, see #2796]

[deshipu]

@dhalbert that looks good, I suppose I got confused by the triggered alarms being stored on a single queue. To implement this, we need to be able to remove the alarms we are waiting on from the queue, but at the same time leave all the other alarms in there. That doesn't sound like a queue data structure anymore. Unless you want to just put the events that don't match our list back on the queue? What happens when there are two different alarms for the same thing? Will that create duplicate events?

[dhalbert]

@deshipu In the sleep case, there is no queue. I was making a strawman proposal for non-sleep event queuing. I think it needs a lot more thought, but could be added to alarm at a later date, or it could be a separate module. Let's discuss this in a new issue, thought at the moment I will not have time due to working on sleep. Also I think we can learn from what the sleep use cases end up being.

[kvc0]

@dhalbert I believe that API pattern is suitable for integration with the tasko event loop. Looking forward to playing with alarms, thanks for working on this!

[tannewt]

a socket connection, a user input on STDIO and a UART transmission

This is exactly why I don't want us to think about async now. When we do async I think we'll want to look at all of our existing APIs and provide async friendly versions. This is a huge task and a distraction from adding sleep support that is broadly useful.

[deshipu]

@tannewt Fair enough. I think that leaves two use cases, I suppose: waiting on an interrupt pin of a sensor, and waking up periodically to make measurements? Maybe also a soft power button?

[tannewt]

@tannewt Fair enough. I think that leaves two use cases, I suppose: waiting on an interrupt pin of a sensor, and waking up periodically to make measurements? Maybe also a soft power button?

Yup, I think we get pretty far with just pin level alarm and time based alarm.

[shaiss]

Is there a branch for testing this out? This could be super handy for my magtag

[dhalbert]

Is there a branch for testing this out? This could be super handy for my magtag

This is my first priority right now. Iwill submit a PR as soon as something is working. The ESP32-S2 is the first targeted port.

[dhalbert]

Partly fixed by #3467. Let's close this and open more task-specific issues.