ssl: work on anything implementing the socket protocol

[jepler]

@anecdata pointed out that moving ssl to shared-module might pave the way to using ssl with wiznet.

I mentioned that for this to be possible, ssl would have to make Python calls for various methods (connect, send, recv, etc) instead of calling the common_hal socket routines.

I went ahead and implemented this. I very lightly tested it: it can do the standard socket test and fetch from https pages like https://www.adafruit.com/api/quotes.php on an esp32s3 device. I didn't test serving.

I also didn't benchmark, test with pico w, let alone test with wiznet.

The following socket methods need to exist and behave like a standard socket:

• connect
• accept
• bind
• listen
• send
• setsockopt (setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) should be implemented, as a no-op if necessary; these constants should match the ones in the core and be provided on the socketpool-compatible object)
• recvfrom

Additionally, the following property needs to exist on socket objects, and for stream sockets it has to use the same value as socketpool.SocketPool.SOCK_STREAM (1):

• type (this property is also added in this PR, because it didn't exist before)

TODO:

• benchmark & short circuit for built in sockets if it is needed to avoid a performance regression
• test with wiznet
• test with wifi
• test https serving

Additionally, if we choose to go this route we might want to refactor the code a bit so that the Python object representations are sent into the ssl shared_module code, because with this change e.g., a connection address is parsed in shared-bindings/ssl, then turned back into an object in shared-module/ssl to call socket connect, and then parsed a second time in shared-bindings/socket connect. This could be avoided by simply re-organizing so that the shared-module/ssl API took the object.

[michalpokusa]

Is this connected to this PR?
#8947

Will it resolve this issue?

[jepler]

I'm excited about this PR but realistically we should probably not rush it in before 9.0.0.

[justmobilize]

@jepler please let me know if I can do any testing or benchmarks

[michalpokusa]

@jepler When ready, I can also help testing HTTPS serving with adafruit_httpserver

[anecdata]

Tested TCP socket-level HTTPS request with these artifacts and the core-compatible-socket-type-numbers branch of [adafruit_wiznet5k](https://github.com/adafruit/Adafruit_CircuitPython_Wiznet5k/tree/core-compatible-socket-type-numbers), on an Adafruit ESP32-S3 TFT + Adafruit Ethernet FeatherWing, just confirming jepler's result. Also fine with Adafruit ESP32-S2 TFT.

PicoW + WIZnet W5100S hat gets MemoryError: in the context.wrap_socket() line, not sure how to debug or proceed from there. gc.mem_free() = 76064 after the exception.

addendum: non-SSL without context.wrap_socket() over ethernet still works fine, as does SSL over wifi (no regressions found)

[michalpokusa]

Tested HTTPS server on ESP32-S2 TFT.
Board crashes and restarts upon receiving the first request.

HTTP version still works the same.

[anecdata]

@michalpokusa I tried HTTPS Server on ESP32-S3 with sockets from WIZnet, but client got connection refused. I'll see if I can replicate espressif-only behavior...

addendum: ESP32-S3 (Adafruit TFT) HTTPS Server seems to be working OK, though I get:

Traceback (most recent call last):
  File "adafruit_httpserver/server.py", line 472, in poll
  File "adafruit_httpserver/server.py", line 604, in _debug_response_sent
TypeError: 'Socket' object isn't subscriptable

library issue, I suspect, that doesn't affect the response being received successfully.

I'm using this PR artifact + your PR88 branch of HTTPServer

[michalpokusa]

@anecdata Thank for testing. HTTPS server works on ESP32-S3 for me too, although I successfully get the response, maybe the problem is due to self-signed cert? When I added -k to curl if works, but this is not really the issue with CircuitPython or library.

The TypeError is also present, which is a result of (I believe) incorrect types being returned by SSLSocket.accept():

@jepler I belive it got changed in this PR, as previously this did not happen on ESP32-S3

[anecdata]

@michalpokusa Yes, ESP32-S3 works and I get the response. The config that didn't work was an S3 with Ethernet FeatherWing, using WIZnet sockets for the HTTPS server. I am using the -k / --insecure option.

[jepler]

TypeError: 'Socket' object isn't subscriptable

I believe I found the cause for this. I've updated (rebased) the branch on top of current main and test built espressif, but didn't re-test anything.ese

[jepler]

PicoW + WIZnet W5100S hat gets MemoryError: in the context.wrap_socket() line, not sure how to debug or proceed from there. gc.mem_free() = 76064 after the exception.

this may simply not be a good combo, with wifi taking up extra RAM. I think SSL needs some large blocks of memory that may simply not be left after parsing in all the wiznet code. anyway, that's a total guess.

[anecdata]

This PR should close #6535. Related: #2202.

[AndreasTheCat]

Based on internal discussions, this is the only board that doesn't already have SSL where we'll enable it

There's also a separate UF2 for W5100S-EVB-Pico (The W6100-EVB-Pico doesn't have its own, but will run the W5100S).

@anecdata you mentioned a separate UF2 for the W5100S-EVB-Pico, which one is it ?

[anecdata]

@AndreasTheCat https://circuitpython.org/board/wiznet_w5100s_evb_pico/

[anecdata]

Is everyone comfortable with this on the client side? It would be nice to get it merged. Any server tweaks could come after.

edit: looks like there are other open To Dos, not sure the best strategy to avoid risk of performance or other regressions

[dhalbert]

This looks OK to me. I did not test: thanks everyone else who did test. Let's get it in the next beta!

[dhalbert]

edit: looks like there are other open To Dos, not sure the best strategy to avoid risk of performance or other regressions

I saw this after I reviewed. @anecdata Which things are you referring to?

[anecdata]

@dhalbert the open checkboxes in the original post. Serving we're aware of. The question I guess is benchmarking / possible refactoring.

[dhalbert]

@jepler how about those unchecked tasks? I will withdraw my approval for the moment.

awaiting feedback on remaining tasks in OP

[dhalbert]

I am not too worried about performance, but testing on pico w and testing serving I think are necessary.

[jepler]

Bypassing the HTTPServer library and running a TCP server directly with this code, modified for WIZnet, results in the same OSError: [Errno 22] Invalid argument upon attempt to connect, followed by a safe mode.

I investigated this.

With an older version of wiznet, I also got an "OSError: [Errno 22] Invalid Argument".

This appears to be due to incorrect handling of timeout=None in recv_into. (the timeout is always None during an SSL handshake) However, this was corrected as a part of adafruit/Adafruit_CircuitPython_Wiznet5k#156

When using the current latest version of wiznet, curl --insecure https://<ip addr>:443/ was able to get the content, but it also reports "curl: (56) Recv failure: Connection reset by peer". Adding a content-length header fixed this.

My code which is for the wiznet 5500 pico-style board: https://gist.github.com/jepler/f479119ce7526d95106b1db924df92ac

Generated a self-signed certificate on linux with openssl req -newkey rsa:4096 -x509 -sha512 -days 365 -nodes -out cert.pem -keyout key.pem

[anecdata]

Enable SSL also on wiznet_w5100s_evb_pico? I could do this in a separate PR if desired.

[jepler]

I also re-tested https serving over wifi on matrixportal s3 with the script at #9003 (comment)

[jepler]

Enable SSL also on wiznet_w5100s_evb_pico? I could do this in a separate PR if desired.

@anecdata yes if you feel like putting in a PR for this I'd be grateful!

[anecdata]

For completeness, tested Pico W with SSL URL, no apparent regressions. Not sure if there was some other aspect of Pico W to test.

[AndreasTheCat]

For completeness, tested Pico W with SSL URL, no apparent regressions. Not sure if there was some other aspect of Pico W to test.

@anecdata pico w and w5100s (using this firmware, this latest adafruit_wiznet5k and this latest adafruit_connection_manager) still runs into MemoryError at 235 in connection_manager- am I missing a part of the discussion here before I try to free any memory (via freezing/ommiting)

[anecdata]

@AndreasTheCat there are comments above about memory issues with Pico W + WIZnet. ESP32-S3 + WIZnet seems OK. Not sure about ESP32-S2. I don't know if Pico + WIZnet was explicitly tested... Pico should have more memory than Pico W. I'll try to set one up to replicate later.

[anecdata]

Looks like jepler did test a WIZnet dev board (Pico + WIZnet):
#8954 (comment)
So I suspect the issue is the extra memory on a Pico W taken up by wifi-related code. But I'll test some variations and add to this comment later. At the moment I'd say ESP32-S3 + WIZnet is the best combination for SSL on Ethernet... it can run native wifi + multiple WIZnet interfaces with SSL + multiple ESP32SPI (Airlift) interfaces with SSL simultaneously.

Update 1: Pico + W5500:

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; W5500-EVB-Pico with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~50-70KB free after each request

HTTPS Server: OK¹
(using server library from PR #88)
¹still some side effects, but it serves with ~40-50KB free

Update 2: Pico W:

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; Raspberry Pi Pico W with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~50-70KB free after each request

HTTPS Server: MemoryError in poll()
(using server library from PR #88)

Update 3: Pico W + W5100S (using WIZnet interface):

CircuitPython: Adafruit CircuitPython 9.1.0-beta.2 on 2024-05-15; Raspberry Pi Pico W with rp2040
Libraries: ...9.x-mpy-20240516

HTTPS Requests: OK
~10-30KB free after each request

HTTPS Server: MemoryError in WIZnet init
(using server library from PR #88)

[AndreasTheCat]

@anecdata

simply amazing to see

`{
"args": {},
"headers": {
"Host": "httpbin.org",
"User-Agent": "Adafruit CircuitPython",
"X-Amzn-Trace-Id": "Root=1-6646e7c5-53d3c3926e38d67d4b0313a8"
},
"origin": "84.59.157.248",
"url": "https://httpbin.org/get"
}

---

41332 //<-gc.memfree()`

for the first time on pico w with w5100s.
Thank you so much 💯