ESP32S2: I2C bus object crashes or failes when used after soft reboot

[ThomasAtBBTF]

This code works fine when the MagTag boots cold.

import board
import busio
print(dir(board))
i2c = busio.I2C(board.SCL, board.SDA)

def printdevices():
    i2c.try_lock()
    # Find the I2C devices available.
    devices = i2c.scan()
    print("I2C devices:", len(devices))
    for device in devices:
        print(hex(device))
    i2c.unlock()
    print("I2C unlocked! ")

printdevices()

After connecting to Putty (here with COM29)
and pressing ctrl-c to get to the CP prompt
and pressing ctrl-d to reload...

There is a core code hard crash:

And the CDC connection to Putty is disconnecting:

PS: I observe similar behaviour using a featherS2.

[ThomasAtBBTF]

Just for your information:
In CP 6.2.0-beta.1 there is no hard fault anymore...
But the code "hangs" for one minute or two in
i2c.scan()
after that the board reboots.

[anecdata]

With latest S3, Adafruit CircuitPython 6.2.0-beta.1-115-ge7438c481 on 2021-02-05; Adafruit MagTag with ESP32S2, I get the same behavior... slight delay, then Volume ejects and (I think) board resets. On a fresh hard reset, it will work once.

[hierophect]

I've been testing this today, it appears to be the same problem as #4079, and probably #4147. Your test sketch catches the problem the most cleanly, I think. I can verify that at least for me, busio.I2C(board.X, board.X) does not cause this issue, only board.I2C.

This makes me think it has something to do with how the internal IDF object is dealt with internally during reset. The current version of I2C on the ESP32S2 doesn't do much on reset, so from what I can tell, board.I2C() tries to re-use the same internal I2C ID during soft reboots, whereas busio.I2C makes a new one. But it's tricky to test, since I'm having trouble getting a backtrace off of the reset point.

[hierophect]

@skieast noted on discord that #3803 is a related issue. Re-enabling the driver delete resolves this problem but will cause the wifi-i2c conflict to reappear

[skieast]

I have been doing some testing as well, including using IDF v4.3-beta1. Mostly to see if the problem changes or goes away with a newer IDF.

[hierophect]

Some notes from testing today:

• The current I2C problems (crashing or failure after soft reboot) do appear to have been caused by Working, tested with two i2c busses #3803, as reverting it (by re-adding the i2c_driver_delete to reset/deinit and having I2C initialize new drivers on reboot) solves those issues. However, as predicted by @skieast, it causes the Wifi/I2C conflict hang in ESP32S2 hangs when interrupting running code that includes I2C object  #3743 to appear again.
• It is not clear what causes this conflict hang. Circuitpython is still looping properly in run_code_py (so it isn't stuck in an infinite loop somewhere) but it no longer maintains USB.
• If we try and solve it in the original way (keeping i2c_driver_delete removed and only ever initializing an I2C peripheral once) an unidentified reset occurs when i2c_master_cmd_begin is called for the second time after a soft reboot. The inputs to this failing call are identical to those passed into successful calls, but it does only seem to happen with an I2C instance that has persisted through a soft reboot like board.I2C() (it won't happen to one you newly create). Resets cannot be replicated when stepping through code manually.

I'm not sure what they key takeaways here are. It's weird that only the persistent I2C has the issues when the driver isn't deleted... maybe the IDF loses something crucial over a soft reboot, even when the Circuitpython object and driver are persistent and haven't been touched? The Wifi hang on the other hand seems like maybe an interrupt priority issue?

[skieast]

Ok, in some brief testing with idf v43-beta1 it seems like the i2c is better. also when i import wifi it doesnt crash. I was having lots of issues probably related to intermittent usb cable connections, a hazard of working on a laptop on my couch.
Onward to actual code.
Also the i2c.c code from CPY is basically the same as what Scott did originally. I added some logging and changed the semaphore around a bit. Nothing much. I also changed the order of some routines inside the constructor.

So i2c works but the wifi will crash when the application is reloaded. This is similar to a previous issue but seems a bit different.
I am looking at the wifi now as i2c looks ok. As OK as I can determine.

[hierophect]

@skieast If it reverts the I2C reset changes, and crashes when it's reloaded, that sounds very much the same as what I was experiencing when I reverted #3803. What do you find to be different, compared to before?

[skieast]

I'm currently testing with the 4.3 branch of the idf. I2c works with changes to i2c.c. Still crashes when wifi is used and the app reloaded. However before just doing an import wifi would crash. Now it gets farther. I'm out now so will look some more later.

[hierophect]

@skieast interesting. I never got a crash on the import of wifi, even on the existing branch of the IDF on Circuitpython main, and your experience is the same as my current Main branch testing. So it doesn't sound like the upgrade to 4.3 has changed much.

[skieast]

I'm trying to remember what wasnt working when I did my PR back in december. At that point I think I was using the REPL,

import i2c
import wifi
i2c = board.I2C()
(then some sort of loop doing nothing)

Interrupt this code, then start entering it again. Would hang at the import wifi statement.
At least thats what I remember. No real difference.

[makermelissa]

I found some success with multiple probes. See this comment for more information:
adafruit/tinyuf2#74 (comment)