Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyPortal Titano - Wippersnapper gets disconnected from WiFi and never reconnects #585

Closed
T-Mosher opened this issue May 9, 2024 · 25 comments
Assignees

Comments

@T-Mosher
Copy link

T-Mosher commented May 9, 2024

Reference this forum thread:
https://forums.adafruit.com/viewtopic.php?t=210269

In short, I have a PyPortal Titano that monitors the state of charge of a LiPo battery via an I2C interface. It reports the battery status every one minute. It is installed in an area that has weak/intermittent WiFi service.

At random intervals (ranging from hourly to weekly), the device's Adafruit IO status goes to Offline.

If I monitor the USB serial port connection, the code is still running and pinging, but I2C publishing is not successful.

The code never recovers from this condition, until I press the Reset button.

Logs are in the forum thread.

Arduino board
PyPortal Titano: SW v1.0.0-alpha.82.
NINA: 1.7.7

To Reproduce
See description above and in the forum thread.

Expected behavior
I expect the Wippersnapper software to automatically recover if an I2C or WiFi connection or publishing failure occurs.

Which components are connected to your device
The issue occurs with either an Adafruit INA219, or with the Adafruit MAX17048 battery gauge via the I2C interface.

The connection failure is not related to the state of charge of the battery. The Titano is running from USB +5V power. The battery, charger, and a 60 mA load are only an experiment (so the feed has something useful to report).

After verifying that the issue occurs with either of those sensors alone, I recently added an HTU31D temperature sensor.

I also have the two discrete outputs from the Titano connnected to an Adafruit latching relay (for controlling a LiPo battery charger module).

Desktop (please complete the following information):

  • OS: Windows 10
  • Browser: Firefox
  • Version: current release
@T-Mosher T-Mosher changed the title PyPortal Titano - Wippersnapper gets disconnected from WiFI and never reconnects PyPortal Titano - Wippersnapper gets disconnected from WiFi and never reconnects May 9, 2024
@tyeth tyeth self-assigned this May 9, 2024
@T-Mosher
Copy link
Author

T-Mosher commented May 14, 2024

Follow-ups:
The NINA firmware has been updated.

I added some other sensors, they update at different intervals. The "WiFi won't reconnect" issue happens regardless of how many sensors are configured.

Here is one of the serial port logs recorded while the Titano SW is running but isn't reconnecting the WiFi:

PING!
PING!
PING!
PING!
Sensor 0x41
Temperature: 22.86 degrees C
Sensor 0x41
Humidity: 32.68%RH
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent!
PING!
PING!
PING!
PING!
PING!
STATUS LED BLINK KAT
Executing periodic event on A16
Publishing pinEvent...Published!
Sensor 0x40
Voltage: 3.66 v
Sensor 0x40
Current: 59.20 mA
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent!
Sensor 0x36
Voltage: 3.65 v
Sensor 0x36
Read: 5.74 %
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent!
PING!
PING!
PING!
PING!
PING!
PING!
PING!

@tyeth
Copy link
Contributor

tyeth commented May 16, 2024

Very strange, and the PING messages in theory mean it's sending and receiving a ping/pong message from the MQTT broker (i.e. wifi still active).

@T-Mosher try this firmware (mostly untested so shout if it's broken) - taken from #584
PyPortal_Titano_beta82_PbErrors_585_g2fa39e88.zip

Also could you just confirm whether you have cut the 5V/3V jumper on the PyPortal Titano to select the i2c bus voltage?

@T-Mosher
Copy link
Author

I did not cut the trace, because the Titano tutorial page has this note in a big blue box:
Unlike the original PyPortal, the I2C connector is level shifted to 3V so its safe to use without changing the power trace

@T-Mosher
Copy link
Author

I'll try that test build shortly. Thanks for your support!

@T-Mosher
Copy link
Author

T-Mosher commented May 19, 2024

I just loaded that test build, I'll post back if there are results.

Note that my wifi has been behaving well for the last few days, so I haven't had any recent connection issues with the previous build (but did going back a few days).

@T-Mosher
Copy link
Author

T-Mosher commented May 20, 2024

The "went offline" issue did happen with the test build.
Here's what the serial port is doing after the issue (I was not logging when it happened).

Sensor 0x40
Voltage: 4.12 v
Sensor 0x40
Current: 69.50 mA
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publ ish I2CDeviceEvent!
PING!
PING!
PING!
PING!
PING!
PING!
PING!
STATUS LED BLINK KAT
Sensor 0x40
Voltage: 4.12 v
Sensor 0x40
Current: 69.40 mA
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent!
PING!
PING!
PING!
PING!
PING!
PING!
Sensor 0x40
Voltage: 4.12 v
Sensor 0x40
Current: 69.40 mA
PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent!
PING!
PING!

@tyeth
Copy link
Contributor

tyeth commented May 21, 2024

Thanks @T-Mosher, a new build ready for you...
wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.82-26-g9dda2fda.zip

You've helped identify some shortcomings in the logging that I was previously unaware of, win win for everybody!

This build is not expected to resolve the problem, but hopefully identify the issue in more detail.

@T-Mosher
Copy link
Author

Will install that this evening.

@tyeth
Copy link
Contributor

tyeth commented May 22, 2024

@T-Mosher made a minor mistake in that one, will be printing i2c failures for each success 🙃

Corrected 📈
wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.82-28-g7c55ad4d.zip

@T-Mosher
Copy link
Author

Ok, I have the corrected one installed now.

@T-Mosher
Copy link
Author

Seems to be working. Just to verify, is this what you expect to see on the serial port?
Sensor 0x40
Voltage: 4.14 v
Sensor 0x40
Current: 69.70 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING:
SUCCESS!
Sending PING:
SUCCESS!
Sending PING:
SUCCESS!
Sending PING:
SUCCESS!
Sending PING:
SUCCESS!
Sending PING:
SUCCESS!

@tyeth
Copy link
Contributor

tyeth commented May 22, 2024 via email

@brentru brentru self-assigned this May 23, 2024
@T-Mosher
Copy link
Author

Update on testing: No lockups have occurred in the last few days, so testing continues.

@T-Mosher
Copy link
Author

Just a question, when the terminal says PING, is that a real IP ping that goes over the WiFi? Or is it some other internal use of the word 'ping'?

@tyeth
Copy link
Contributor

tyeth commented May 24, 2024 via email

@tyeth
Copy link
Contributor

tyeth commented May 30, 2024

Okay @T-Mosher, third times a charm, this one should do what we've been hoping for.

It announces the RSSI in the serial monitor, and successfully restarts the PyPortal Titano's onboard ESP32, in theory we may fail to send a datapoint or two but will then reconnect at the next scheduled ping.

Give it a try and let us know how you get on, also thanks again for being a willing test pilot!

wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.83-31-g9c0c151b.zip

@T-Mosher
Copy link
Author

OK, I've installed it. Will keep you posted on results.

FYI, I've been running the previous release you posted above (beta.82) since May 21 (9 days ago), and have not seen any lockups. I don't know if any of the changes in that build addressed the issue, or I've just been lucky with a stable WiFi connection lately. I have no way to tell.

@tyeth
Copy link
Contributor

tyeth commented May 31, 2024 via email

@tyeth tyeth mentioned this issue May 31, 2024
@T-Mosher
Copy link
Author

T-Mosher commented Jun 1, 2024

To speed up the testing process, I'm trying to artificially degrade my WiFi performance (i.e. wrapping things in aluminum foil, etc), but it's not really effective (and I can't just disconnect the router at will because it's shared by other users).

@T-Mosher
Copy link
Author

T-Mosher commented Jun 2, 2024

Just had this happen (failed ping and recovered - but the terminal didn't reconnect, so I have to restart the log).
Note that it did automatically recover the WiFi connection and continued supplying data to the feeds.

Checking back in that log file, there are a total of 20 ping failures - but only that last one caused the serial port logging to stop.

Sensor 0x40
        Voltage: 4.12 v
Sensor 0x40
        Current: 64.70 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -62
Sending PING: SUCCESS!
WiFi RSSI: -76
Sending PING: SUCCESS!
WiFi RSSI: -64
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -78
STATUS LED BLINK KAT
Sensor 0x40
        Voltage: 4.12 v
Sensor 0x40
        Current: 64.40 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
   Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -68
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet

@T-Mosher
Copy link
Author

T-Mosher commented Jun 6, 2024

Had another one of these last night (a WiFi restart). It was associated with a feed alert that it had not updated for 10 minutes (normally reports every 30 seconds).

Note that when this WiFi restart happens, I also get a "ding" from Windows that the USB drive has re-connected. Then the serial port logging stops.

Sending PING: SUCCESS!
WiFi RSSI: -73
STATUS LED BLINK KAT
Sensor 0x40
        Voltage: 4.11 v
Sensor 0x40
        Current: 66.00 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -58
Sending PING: SUCCESS!
WiFi RSSI: -74
Sending PING: SUCCESS!
WiFi RSSI: -59
Sending PING: SUCCESS!
WiFi RSSI: -75
Sending PING: SUCCESS!
WiFi RSSI: -59
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
   Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -63
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet

@T-Mosher
Copy link
Author

Another WiFi error and restart.

I"m not sure if you want me to continue reporting these. Please advise.

Sending PING: SUCCESS!
WiFi RSSI: -64
Sensor 0x40
        Voltage: 4.09 v
Sensor 0x40
        Current: 62.60 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -66
Sending PING: SUCCESS!
WiFi RSSI: -63
Sensor 0x41
        Temperature: 25.34 degrees C
PUBLISHING -> I2C Device Sensor Event Message...ERROR: MQTT Publish failed!
ERROR: Failed to encode and publish I2CDeviceEvent!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -62
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
   Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -63
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet
Adafruit.io WipperSnapper
-------Device Information-------
Fir
   Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94

@tyeth
Copy link
Contributor

tyeth commented Jun 17, 2024

Thanks @T-Mosher that's enough for now, really appreciate the extra feedback. Ideally it probably wouldn't reconnect serial/reboot the main device (only the esp32 co-processor).
I'm going to be cleaning up the associated PR branch this week so may have a new version to test, but you can equally just ignore it and carry on now that you have a fix. It will probably make it into the next WipperSnapper release (or one after if not).

@tyeth tyeth mentioned this issue Jul 1, 2024
@tyeth
Copy link
Contributor

tyeth commented Aug 1, 2024

Forgot to mention this change has been released last week. Current latest version is v1.0.0.beta-88

Closing this for now. Thanks for the help testing, and feel free to reopen this issue if needed (or create a new one).

@tyeth tyeth closed this as completed Aug 1, 2024
@T-Mosher
Copy link
Author

T-Mosher commented Aug 1, 2024

I installed beta-88 a couple of days ago. Seems to be working fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants