Fetching HTTPS content

[jobovy]

Description

The current pyodide-kernel does not allow fetching of HTTPS content using urllib, because the requisite HTTPSConnection in http.client and the HTTPSHandler in urllib.request are not loaded as the ssl module isn't available when these are first loaded by the pyodide-kernel. This was already discussed here: jupyterlite/jupyterlite#415, but that was when the ssl module was not supported in pyodide, which it now is.

This leads to issues such as pyodide-http/issues/33 (and further downstream astropy/astropy#15744). The reloading fix suggested by @rth in koenvo/pyodide-http#33 (comment) doesn't resolve the issue, as it leads to a weird CORS error because the HTTPS URL gets changed into an insecure websocket.

I believe the issue is that http.client and urllib.request are loaded indirectly during the initialization of the pyodide-kernel, most likely in the IPython initialization. Because the ssl module is not installed at that point, http.client.HTTPSConnection is not defined.

A possible fix is to install the ssl module before installing other things. For example, inserting await piplite.install(['ssl'], keep_going=True); as the first piplite.install here fixes the issue:

pyodide-kernel/packages/pyodide-kernel/src/worker.ts

Lines 77 to 84 in 2863507

	protected async initKernel(options: IPyodideWorkerKernel.IOptions): Promise<void> {
	// from this point forward, only use piplite (but not %pip)
	await this._pyodide.runPythonAsync(`
	await piplite.install(['sqlite3'], keep_going=True);
	await piplite.install(['ipykernel'], keep_going=True);
	await piplite.install(['comm'], keep_going=True);
	await piplite.install(['pyodide_kernel'], keep_going=True);
	await piplite.install(['ipython'], keep_going=True);

I can submit a PR if this is an acceptable fix (I don't think there is any downside to installing ssl as it's a standard pyodide module now). If it needs a test, please advise on where and how to write a test.

One other thing I noticed that I'll briefly mention: trying to build the kernel failed with a strange error:

@jupyterlite/pyodide-kernel: [piplite] trying in-tree build...
@jupyterlite/pyodide-kernel: Environment `default` is incompatible: no compatible Python distribution available

which I believe is because I was trying to build in a Python 3.10 environment and hatch needed a 3.11 Python environment. Switching to a Python 3.11 environment fixed this issue and allowed me to build the kernel. The contributing docs just say Python >= 3.8, so maybe that needs to be updated or maybe there is something wrong with the build config. I can open a separate issue about this if you like.

Reproduce

>>> import micropip
>>> await micropip.install(["ssl", "pyodide-http>=0.2.1", "astropy"])
>>> from astropy.coordinates import SkyCoord
>>> import pyodide_http; pyodide_http.patch_all()
>>> SkyCoord.from_name('NGC3256')

Works in the pyodide REPL, but gives AttributeError: module 'urllib.request' has no attribute 'HTTPSHandler' in jupyterlite.

Expected behavior

For the code above to work

Context

• JupyterLite version: latest main
• Operating System and version: Linux
• Browser and version: Chrome Version 119.0.6045.199 (Official Build) (arm64)