CPython socket compatibility

[brentru]

No description provided.

[brentru]

I'm working on this over on https://github.com/brentru/Adafruit_CircuitPython_MiniMQTT/tree/update-for-cpython-compat

[askpatrickw]

Hey @brentru, I was trying to use your library with
https://github.com/tannewt/Adafruit_CircuitPython_Requests/blob/http11/adafruit_requests.py#L358
And I ran into an issue of the socket_module not being set.

I tried to mimic what _get_socket in requests is doing there by adding socket_pool and ssl_context parameters to MQTT.init and commenting out the check for socket_module in _get_socket. But that lead to me getting an error in connect.

Traceback (most recent call last):
  File "code.py", line 80, in <module>
  File "/Users/askpatrickw/Downloads/Adafruit_CircuitPython_MiniMQTT-update-for-cpython-compat/adafruit_minimqtt/adafruit_minimqtt.py", line 306, in connect
TypeError: function missing required positional argument #3

I'm tired but I'll probably have a go at this in a day or so. If you had any obvious feedback let me know. I'm happy to push into your fork if I get it working.

I should mention this is on Scott's ESP32-S2 Native_wifi branch and with his requests module I linked above.

Cheers!

[brentru]

@askpatrickw I'm not sure what you're using the Requests module for, this library doesn't rely on it.

Could you post your full code.py using this branch so I can take a look at it? I have not tested this on ESP32-S2 yet - waiting on scott's requests fork to be PR'd so I can keep this branch in-line with the final implementation.

[askpatrickw]

Hey @brentru thanks for offering to take a look, here's a full walk through first with your code and then with my attempts...

My code.py is essentially the simple example I'll only show the setup and connect.

There is no way to do his part:
MQTT.set_socket(socket, esp) so I commented it out.

client = MQTT.MQTT(
    broker=secrets["broker"], username=secrets["user"], password=secrets["pass"],
)

# Connect callback handlers to client
client.on_connect = connect
client.on_disconnect = disconnect
client.on_subscribe = subscribe
client.on_unsubscribe = unsubscribe
client.on_publish = publish

print("Attempting to connect to %s" % client.broker)
client.connect()

Output

Attempting to connect to io.adafruit.com
Traceback (most recent call last):
  File "code.py", line 80, in <module>
  File "adafruit_minimqtt/adafruit_minimqtt.py", line 302, in connect
  File "adafruit_minimqtt/adafruit_minimqtt.py", line 110, in _get_socket
RuntimeError: socket_module must be set before using adafruit_requests

That lead me to compare your version of MiniMQTT to Scott's adafruit_requests and I noticed the code looked very similar and using the power of assumption I tried to make them more the same. ;-)

I modified init, adding two parameters

        socket_pool=None,
        ssl_context=None,

and added those parameters to self.

        self._socket_pool = socket_pool
        self._ssl_context = ssl_context

I also changed _get_socket passing in self, commenting out the check for socket_module and had several calls use self and the passed in parameters from init

def _get_socket(self, host, port, proto, *, timeout=5):
    key = (host, port, proto)
    if key in _socket_pool:
        return _socket_pool[key]
    # if not socket_module:
    #     raise RuntimeError("socket_module must be set before using adafruit_requests")
    if proto == "https:" and not ssl_context:
        raise RuntimeError(
            "ssl_context must be set before using adafruit_requests for https"
        )
    addr_info = self._socket_pool.getaddrinfo(host, port, 0, socket_module.SOCK_STREAM)[0]
    sock = self._socket_pool.socket(addr_info[0], addr_info[1], addr_info[2])
    if proto == "https:":
        print("https")
        sock = self._ssl_context.wrap_socket(sock, server_hostname=host)
        print(sock)
    sock.settimeout(timeout)  # socket read timeout

    sock.connect((host, port))
    _socket_pool[key] = sock
    return sock

This is how I call it now and you should note that socketpool is from Scott's native_wifi branch

# Set up a MiniMQTT Client
pool = socketpool.SocketPool(wifi.radio)
client = MQTT.MQTT(
    broker=secrets["broker"], username=secrets["user"], password=secrets["pass"],
    socket_pool=pool, ssl_context=ssl.create_default_context()
)

and that gets me to this error:

Attempting to connect to io.adafruit.com
Traceback (most recent call last):
  File "code.py", line 80, in <module>
  File "/Users/askpatrickw/Downloads/Adafruit_CircuitPython_MiniMQTT-update-for-cpython-compat/adafruit_minimqtt/adafruit_minimqtt.py", line 306, in connect
TypeError: function missing required positional argument #3

And where I'm stumped is how could parameter 3 be missing... isn't it hard coded to "https:" ?

def connect(self, clean_session=True):
        """Initiates connection with the MQTT Broker.
        :param bool clean_session: Establishes a persistent session.

        """
        if self.port == 8883:
            if self.logger is not None:
                self.logger.debug("Attempting to establish secure MQTT connection...")
            self._sock = _get_socket(self.broker, self.port, "https:")
        else:
            if self.logger is not None:
                self.logger.debug("Attempting to establish insecure MQTT connection...")
            self._sock = _get_socket(self.broker, self.port, "http:")

[tannewt]

@askpatrickw Please push a draft PR of your changes. Explaining them here is harder to follow than a PR. It's also not really an issue with this repo if your code is modified. :-)

[askpatrickw]

OK I PR'd to Brent since my changes are from his changes.

[ehagerty]

Hi forgive me if this is the wrong way to ask this question. We have a current production stack using esp32 boards, micropython and google's MQTT stack. I want to move us to CP, however in endeavouring to do so, I ran into what I assume is the basis for this thread. At present am I correct that minimqtt assumes that an esp32 board is only/always an SPI based co-processor, ergo when I try to use an esp32s2 board on its own (the Saola in our case) it won't yet work? Please forgive an old Aspie if I'm breaking etiquette asking here.

[askpatrickw]

@ehagerty You're 100% correct.

[ehagerty]

@ehagerty You're 100% correct.
@askpatrickw wow, now that's service with a smile! Should I go ahead and write my own based on the original? Sorry, just don't know the rules of the road in CP land. Thanks so much for your time!

[askpatrickw]

I think a PR is always welcome in CP land. Jump on the Discord and say hi. https://adafru.it/discord

[ladyada]

we'll be working on adding esp32s2 support to this library this month, we only recently got requests working well - if you can use REST for your project you could start there

[brentru]

@ehagerty ^ ESP32S2 and CPython support to this library this month - you don't need to roll your own unless you want to.

I'll list the branch I'm working off of in a subsequent reply on this issue. feel free to jump in as I work on it, I'm on the discord as @brentru. I'll also comment here when we have a stable PR ready for the community.

You mentioned Google's MQTT stack - are you aiming to use this library on with Google Cloud IoT core?

[ehagerty]

Goodness @ladyada, I'm so sorry to cause any trouble! You are very kind to take the time to respond. @brentru thank you so much as well, I will move over to Discord - sorry old habits die hard. Yes, we use google's IoT stack in production and I took an old tutorial of theirs a year or so ago and made it work for us on MicroPython. Just wanting to move over to the CP world now, particularly b/c I'm very keen for hardware reasons to do our next bulk buy of esp32-s2, and not the older stuff. Once in production it becomes a hassle to migrate as you can imagine - our stuff sits under the beds of individuals we care for in our social care mothership business.

[eukrit]

@brentru @askpatrickw thank you for pointing me here. I've read the whole thread and learned that CPython now supports ESP32-S2. but where can I find sample code or tutorial to with with MQTT? another question, what is SP?

[ehagerty]

@brentru @askpatrickw thank you for pointing me here. I've read the whole thread and learned that CPython now supports ESP32-S2. but where can I find sample code or tutorial to with with MQTT? another question, what is SP?

hi @eukrit, 'sp' was my typing error, i meant 'cp' as in CircuitPython. sorry :-)

[askpatrickw]

@eukrit Until this issue is closed, the MQTT library won't work with the ESP32-S2.

If you're interested in what the work is, you can look at the adafruit_circuitpython_requests library where a similar change to support the onboard WIFI of the esp32-s2 (using a CPython like socketpool) was implemented where it previously only supported the esp32 WIFI sub-board accessed over SPI as the minimqtt does now. Now requests supports both and a similar change is needed here. I hope that helps.

[eukrit]

@askpatrickw I thought this release means that work is ready? https://github.com/adafruit/Adafruit_CircuitPython_AdafruitIO/releases/tag/5.0.0
Please correct me if I'm wrong here.

[askpatrickw]

@eukrit That is the Adafruit_IO Library it does not use this general purpose MiniMQTT library.
Why don't you jump over to the Discord Server if you have more questions. https://discord.gg/adafruit that's a more appropriate place for discussions. Or the Forums...

[brentru]

@eukrit For clarification - the Adafruit IO CircuitPython Library's HTTP API object works with ESP32-S2. This class uses the Adafruit_CircuitPython_Requests library and the Adafruit IO HTTP API (IO_HTTP), not MQTT. Examples can be found here: https://github.com/adafruit/Adafruit_CircuitPython_AdafruitIO/tree/5.0.0/examples/adafruit_io_http

Adafruit IO also has a MQTT API. The Adafruit IO CircuitPython Library includes a object to access this API, IO_MQTT, but it is currently not working for the ESP32-S2 as it requires this issue to be closed first.

[brentru]

This has been addressed in #61 and released in v5.0.0 🎉