CircuitPython TLS version in m5stack

[wz2b]

I'm using circuitpython 9.0.4 on an m5stack dial and it works great, but I can't connect to my MQTT broker:

1716232267: OpenSSL Error[0]: error:1402542E:SSL routines:ACCEPT_SR_CLNT_HELLO:tlsv1 alert protocol version
1716232267: Client <unknown> disconnected: Protocol error.

I would like to be able to:

ssl_context = ssl.create_default_context()
ssl_context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1  # Disable TLS 1.0 and 1.1

but there is no .options

# dir(ssl_context)
['__class__', 'check_hostname', 'load_cert_chain', 'load_verify_locations', 'set_default_verify_paths', 'wrap_socket']

Is there any way to force TLS v1.2? If not, this is a feature request.

Adafruit CircuitPython 9.0.4 on 2024-04-16; M5Stack Dial with ESP32S3

[dhalbert]

Could you try 9.1.0-beta.2 and see if there is any difference in behavior?

Which TLS versions is your broker supporting? I am surprised, because I thought we supported at least TLSv1.2, if not also TLSv1.3 and we'd try the "best" one first.

There are test hosts here for TLSv1.0, v1.1, and v1.2: https://badssl.com/. You can just try a connect or a request from those.

[wz2b]

Yes, I would be happy to try the latest version! My broker is set for tls 1.3 and later, I looked at mosquitto.conf yesterday - when I get back to my office I will post an updated mosquitto.conf to confirm this, and I can try beta.2.

…

On Tue, May 21, 2024 at 12:14 PM Dan Halbert ***@***.***> wrote: Could you try 9.1.0-beta.2 and see if there is any difference in behavior? Which TLS versions is your broker supporting? I am surprised, because I thought we supported at least TLSv1.2, if not also TLSv1.3 and we'd try the "best" one first. There are test hosts here for TLSv1.0, v1.1, and v1.2: https://badssl.com/. You can just try a connect or a request from those. — Reply to this email directly, view it on GitHub <#9265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALTMOPJS3WYCMQE5S5FAQTZDNXGVAVCNFSM6AAAAABIB4ELY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRSHE4DAMZUGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dhalbert]

Can you try setting it to v1.2? Is it v1.3 only? v1.3 may not be supported.

[wz2b]

Setting the broker to use v1.2 and even v1.1 still didn't work, but now I'm thinking maybe the TLS version isn't the issue:

1716314886: OpenSSL Error[0]: error:1402542E:SSL routines:ACCEPT_SR_CLNT_HELLO:tlsv1 alert protocol version
1716314886: Client <unknown> disconnected: Protocol error.

I don't know. Mosquitto is pretty standard - it's probably the #1 MQTT broker used by people who use CircuitPython (I would imagine) so I'm mystified. Trying beta.2 now.

[dhalbert]

The M5Dial has an M5Stamp inside, which is 8MB flash, but no PSRAM. We've seen problems with running out of memory on similar configurations when setting up HTTPS with your own certificates. Is the logging you're showing here from mosquitto? What is being printed on the REPL serial port in CircuitPython?

If you could come up with a minimal example, that would be great. And show us the (readacted as needed) mosquitto config.

[dhalbert]

Also update the libraries to the latest as of today. There have been changes even today that will not be in the bundle until tonight.

[wz2b]

Here's a really simple example:

import time
import board
import busio
from digitalio import DigitalInOut
import adafruit_minimqtt.adafruit_minimqtt as MQTT
import wifi
import socketpool
import ssl

print("Connecting WiFi")
r = wifi.radio
r.connect("RIT-WiFi")
print(r.connected)


# Define callback methods which are called when events occur
def connected(client, userdata, flags, rc):
    print("Connected to MQTT broker!")

def disconnected(client, userdata, rc):
    print("Disconnected from MQTT broker!")

def publish(client, userdata, topic, pid):
    print("Published to {0} with PID {1}".format(topic, pid))

# Set up a socket pool
pool = socketpool.SocketPool(wifi.radio)

ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
# ssl_context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1  # Disable TLS 1.0 and 1.1

# ssl_context.verify_mode = ssl.CERT_NONE

# Create a MiniMQTT client
mqtt_client = MQTT.MQTT(
    broker="192.168.0.99",
    port=8883,
    username="user",
    password="password",
    socket_pool=pool,
    ssl_context=ssl_context
)

# Connect callback handlers to client
mqtt_client.on_connect = connected
mqtt_client.on_disconnect = disconnected
mqtt_client.on_publish = publish

# Connect to the MQTT broker
mqtt_client.connect()

# Publish a message
mqtt_client.publish("test/topic", "Hello from CircuitPython!")

# Disconnect from the MQTT broker
mqtt_client.disconnect()

# Loop forever doing nothing, as an example
while True:
    pass

# mosquitto.conf
per_listener_settings true

persistence true
persistence_location /mosquitto/data

listener 1883
password_file /mosquitto/passwd

listener 8883

# You may want to leave this out entirely
password_file /mosquitto/passwd
certfile /run/secrets/server_certificate
keyfile /run/secrets/server_certificate_key

# This can be tlsv1.2 or tlsv1.1, too.  This key specifies
# the MINIMUM level, so if you specify 1.2, then 1.3 will work
# as well.
tls_version tlsv1.3

# Only provide support for GCM cipher modes - disabled while we try to figure this out
# ciphers_tls1.3 TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256

Notice I commented out all the requirements on things like cipher suites to make things a little looser, just until we can figure this out. This is mosquitto 2.0.18 which is the most recent eclipse-mosquitto image in dockerhub.

[wz2b]

Just for the fun of it, I converted umqtt.simple to work with the socketpool, and got the same exact results:

import struct
import socketpool
import wifi
import ssl

class MQTTException(Exception):
    pass

class MQTTClient:

    def __init__(self, client_id, server, port=0, user=None, password=None, keepalive=0,
                 ssl=False, ssl_params={}):
        if port == 0:
            port = 8883 if ssl else 1883
        self.client_id = client_id
        self.sock = None
        self.server = server
        self.port = port
        self.ssl = ssl
        self.ssl_params = ssl_params
        self.pid = 0
        self.cb = None
        self.user = user
        self.pswd = password
        self.keepalive = keepalive
        self.lw_topic = None
        self.lw_msg = None
        self.lw_qos = 0
        self.lw_retain = False

    def _send_str(self, string):
        self.sock.send(len(string).to_bytes(2, 'big'))
        self.sock.send(string.encode('utf-8'))

    def _recv_len(self):
        n = 0
        sh = 0
        while True:
            b = self.sock.recv(1)[0]
            n |= (b & 0x7f) << sh
            if not b & 0x80:
                return n
            sh += 7

    def set_callback(self, f):
        self.cb = f

    def set_last_will(self, topic, msg, retain=False, qos=0):
        assert 0 <= qos <= 2
        assert topic
        self.lw_topic = topic
        self.lw_msg = msg
        self.lw_qos = qos
        self.lw_retain = retain

    def connect(self, clean_session=True):
        pool = socketpool.SocketPool(wifi.radio)
        self.sock = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
        
        # Resolve address and connect
        addr_info = pool.getaddrinfo(self.server, self.port)
        addr = addr_info[0][-1]
        self.sock.connect(addr)

        if self.ssl:
            context = ssl.create_default_context()
            self.sock = context.wrap_socket(self.sock, server_hostname=self.server, **self.ssl_params)

        premsg = bytearray(b"\x10\0\0\0\0\0")
        msg = bytearray(b"\x04MQTT\x04\x02\0\0")

        sz = 10 + 2 + len(self.client_id)
        msg[6] = clean_session << 1
        if self.user is not None:
            sz += 2 + len(self.user) + 2 + len(self.pswd)
            msg[6] |= 0xC0
        if self.keepalive:
            assert self.keepalive < 65536
            msg[7] |= self.keepalive >> 8
            msg[8] |= self.keepalive & 0x00FF
        if self.lw_topic:
            sz += 2 + len(self.lw_topic) + 2 + len(self.lw_msg)
            msg[6] |= 0x4 | (self.lw_qos & 0x1) << 3 | (self.lw_qos & 0x2) << 3
            msg[6] |= self.lw_retain << 5

        i = 1
        while sz > 0x7f:
            premsg[i] = (sz & 0x7f) | 0x80
            sz >>= 7
            i += 1
        premsg[i] = sz

        self.sock.send(premsg[:i + 2])
        self.sock.send(msg)
        self._send_str(self.client_id)
        if self.lw_topic:
            self._send_str(self.lw_topic)
            self._send_str(self.lw_msg)
        if self.user is not None:
            self._send_str(self.user)
            self._send_str(self.pswd)
        resp = self.sock.recv(4)
        assert resp[0] == 0x20 and resp[1] == 0x02
        if resp[3] != 0:
            raise MQTTException(resp[3])
        return resp[2] & 1
    
    def disconnect(self):
        self.sock.send(b"\xe0\0")
        self.sock.close()

    def ping(self):
        self.sock.send(b"\xc0\0")

    def publish(self, topic, msg, retain=False, qos=0):
        pkt = bytearray(b"\x30\0\0\0")
        pkt[0] |= qos << 1 | retain
        sz = 2 + len(topic) + len(msg)
        if qos > 0:
            sz += 2
        assert sz < 2097152
        i = 1
        while sz > 0x7f:
            pkt[i] = (sz & 0x7f) | 0x80
            sz >>= 7
            i += 1
        pkt[i] = sz
        #print(hex(len(pkt)), hexlify(pkt, ":"))
        self.sock.send(pkt[:i + 1])
        self._send_str(topic)
        if qos > 0:
            self.pid += 1
            pid = self.pid
            struct.pack_into("!H", pkt, 0, pid)
            self.sock.send(pkt[:2])
        self.sock.send(msg)
        if qos == 1:
            while True:
                op = self.wait_msg()
                if op == 0x40:
                    sz = self.sock.recv(1)
                    assert sz == b"\x02"
                    rcv_pid = self.sock.recv(2)
                    rcv_pid = rcv_pid[0] << 8 | rcv_pid[1]
                    if pid == rcv_pid:
                        return
        elif qos == 2:
            assert 0

    def subscribe(self, topic, qos=0):
        assert self.cb is not None, "Subscribe callback is not set"
        pkt = bytearray(b"\x82\0\0\0")
        self.pid += 1
        struct.pack_into("!BH", pkt, 1, 2 + 2 + len(topic) + 1, self.pid)
        #print(hex(len(pkt)), hexlify(pkt, ":"))
        self.sock.send(pkt)
        self._send_str(topic)
        self.sock.send(qos.to_bytes(1, "little"))
        while True:
            op = self.wait_msg()
            if op == 0x90:
                resp = self.sock.recv(4)
                #print(resp)
                assert resp[1] == pkt[2] and resp[2] == pkt[3]
                if resp[3] == 0x80:
                    raise MQTTException(resp[3])
                return

    # Wait for a single incoming MQTT message and process it.
    # Subscribed messages are delivered to a callback previously
    # set by .set_callback() method. Other (internal) MQTT
    # messages processed internally.
    def wait_msg(self):
        res = self.sock.recv(1)
        self.sock.setblocking(True)
        if res is None:
            return None
        if res == b"":
            raise OSError(-1)
        if res == b"\xd0":  # PINGRESP
            sz = self.sock.recv(1)[0]
            assert sz == 0
            return None
        op = res[0]
        if op & 0xf0 != 0x30:
            return op
        sz = self._recv_len()
        topic_len = self.sock.recv(2)
        topic_len = (topic_len[0] << 8) | topic_len[1]
        topic = self.sock.recv(topic_len)
        sz -= topic_len + 2
        if op & 6:
            pid = self.sock.recv(2)
            pid = pid[0] << 8 | pid[1]
            sz -= 2
        msg = self.sock.recv(sz)
        self.cb(topic, msg)
        if op & 6 == 2:
            pkt = bytearray(b"\x40\x02\0\0")
            struct.pack_into("!H", pkt, 2, pid)
            self.sock.send(pkt)
        elif op & 6 == 4:
            assert 0

    # Checks whether a pending message from server is available.
    # If not, returns immediately with None. Otherwise, does
    # the same processing as wait_msg.
    def check_msg(self):
        self.sock.setblocking(False)
        return self.wait_msg()

[wz2b]

It's odd that the SSL module doesn't support options like verify certificate (to false) and TLS version - is this to save code space?

[dhalbert]

We are using mbedtls under the covers. It may or may not provide some of this functionality. We implemented a subset to cover most use cases. Additions are welcome via PR.

[wz2b]

I would love to but I feel like I'd be over my head when it comes to TLS ....

[wz2b]

Is this possibly related? https://forums.mbed.com/t/tls-version-number-in-a-client-hello-packet/4863

[dhalbert]

Looking at the compilations settings, the Espressif boards are compiled to support TLSv1.2. The link you posted above is not what the source code looks like any more. It enforces a minimum TLS version based on the compilation options.

[wz2b]

This is really stretching what I know about TLS but you're right. I wrote a small shim around circuitpython's socket class so I could inject some logging. The first thing it sends is:

16 03 03 00 d3 01 00 00 cf 03 03 00 01 05 f1 fa ee 98 08 fe 52 73 ee 93 3f 46 62 a0 6b a0 8b a8 49 1e 6d 6a db b0 b4 8a 18 7e 62 00 00 5a c0 2c c0 30 c0 ad c0 24 c0 28 c0 0a c0 14 c0 af c0 5d c0 61 c0 49 c0 4d c0 2b c0 2f c0 ac c0 23 c0 27 c0 09 c0 13 c0 ae c0 5c c0 60 c0 48 c0 4c c0 32 c0 2a c0 0f c0 2e c0 26 c0 05 c0 5f c0 63 c0 4b c0 4f c0 31 c0 29 c0 0e c0 2d c0 25 c0 04 c0 5e c0 62 c0 4a c0 4e 00 ff 01 00 00 4c 00 00 00 18 00 16 00 00 13 74 65 73 74 62 65 64 2e 67 69 73 2e 72 69 74 2e 65 64 75 00 0a 00 08 00 06 00 1d 00 17 00 18 00 0d 00 0e 00 0c 06 03 06 01 05 03 05 01 04 03 04 01 00 0b 00 02 01 00 00 16 00 00 00 17 00 00 00 23 00 00

Breakdown

• 16: Content Type (22 for Handshake)
• 03 03: TLS Version (TLS 1.2)
• 00 d3: Length of the message (211 bytes)
• 01: Handshake Type (ClientHello)
• 00 00 cf: Length of the ClientHello message (207 bytes)
• 03 03: TLS Version (TLS 1.2)
• 00 01 05 f1 fa ee 98 08 fe 52 73 ee 93 3f 46 62 a0 6b a0 8b a8 49 1e 6d 6a db b0 b4 8a 18 7e 62: Random
• 00: Session ID length (0 bytes)
• 00 5a: Cipher Suites length (90 bytes)
• c0 2c c0 30 c0 ad c0 24 c0 28 c0 0a c0 14 c0 af c0 5d c0 61 c0 49 c0 4d c0 2b c0 2f c0 ac c0 23 c0 27 c0 09 c0 13 c0 ae c0 5c c0 60 c0 48 c0 4c c0 32 c0 2a c0 0f c0 2e c0 26 c0 05 c0 5f c0 63 c0 4b c0 4f c0 31 c0 29 c0 0e c0 2d c0 25 c0 04 c0 5e c0 62 c0 4a c0 4e: Cipher Suites
• 00 ff: Compression methods length (1 byte)
• 01: Compression method (null)
• 00 00 4c: Extensions length (76 bytes)
• 00 00: Extension type (server_name)
• 00 18: Extension length (24 bytes)
• 00 16: Server Name Indication (SNI) length (22 bytes)
• 00 00 13 74 65 73 74 62 65 64 2e 67 69 73 2e 72 69 74 2e 65 64 75: SNI ("testbed.gis.rit.edu")
• 00 0a: Extension type (supported_groups)
• 00 08: Extension length (8 bytes)
• 00 06: Supported Groups length (6 bytes)
• 00 1d 00 17 00 18: Supported Groups
• 00 0d: Extension type (signature_algorithms)
• 00 0e: Extension length (14 bytes)
• 00 0c: Signature Algorithms length (12 bytes)
• 06 03 06 01 05 03 05 01 04 03 04 01: Signature Algorithms
• 00 0b: Extension type (ec_point_formats)
• 00 02: Extension length (2 bytes)
• 01 00: EC Point Formats
• 00 16: Extension type (supported_versions)
• 00 00: Extension length (0 bytes)
• 00 17: Extension type (key_share)
• 00 00: Extension length (0 bytes)
• 00 23: Extension type (pre_shared_key)
• 00 00: Extension length (0 bytes)

This problem appears to be not what I think it is. I'm going to close this ticket. Sorry for the noise but thank you for helping me work through this. I'll leave this info here just in case someone else stumbles across a similar question in the future.

[dhalbert]

No problem - I am looking at a number of things about the SSL implementation that I had not tried to understand in detail previously, in order to figure out why we're having other problems, such as memory issues. For instance, I did not know about the TLSv1.2 compilation option.

[wz2b]

If this helps, you can do this:

        underlying_sock = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
        sock = LoggingSocket(underlying)

all it does is spits out the writes. For what I was doing (trying to see the TLS handshake, which is the very first thing it does) it was helpful. Mainly because I discovered that if you paste the hex to ChatGPT and tell it that it's a TLS negotiation, it will decode the whole thing for you.

class LoggingSocket:
    def __init__(self, sock):
        self._sock = sock

    def send(self, data):
        # Log the data being sent as hexadecimal
        hex_data = ' '.join(f'{b:02x}' for b in data)
        print(f'Sending data: {hex_data}')
        return self._sock.send(data)

    def recv(self, bufsize):
        return self._sock.recv(bufsize)

    def connect(self, addr):
        return self._sock.connect(addr)

    def close(self):
        return self._sock.close()

    def setblocking(self, flag):
        return self._sock.setblocking(flag)

    def __getattr__(self, name):
        # Delegate attribute access to the underlying socket
        return getattr(self._sock, name)