Windows development challenges

[bollwyvl]

At present, I think all of our contributors are working on Linux/MacOS, and while we test fairly exhaustively on Windows in CI, it's not much fun to develop locally.

Some things I've encountered trying to test #245:

• binder/postBuild is written in bash
  • we could write it in python
  • as more things seem error prone (anything involving node), doit (a la [wip] add doit #183) would probably be helpful
• prettier doesn't work
  • we can change the globs to be cmd-friendly
• nblint doesn't work
  • just needs some path/encoding stuff for node, pathlib

will update as i find more...

[bollwyvl]

Opening the python example:

[W 20:04:32.280 LabApp] [lsp] error in listener <function setup_shadow_filesystem.<locals>.shadow_virtual_documents at 0x0000024320AEC318> for message {'jsonrpc': '2.0', 'method': 'textDocument/didChange', 'params': {'textDocument': {'uri': 'file:///c:/git/lsp/jupyterlab-lsp/.virtual_documents/examples/Python.ipynb.py', 'version': 1}, 'contentChanges': [{'text': 'def square(x):\n    """Can you see me?"""\n    return x*x\n\n\nresult = square(2)\n\n\nfrom statistics import mean\n\n\nundefined_variable\n\n\nclass Dog:\n    \n    def bark(self):\n        print(\'🐕 woof woof\')\n\n\n\n\n\nDog().bark()\n\n\nclass Cat:\n\n    def miaow(self):\n        print(\'miaow\')\n\n\nCa\n\n\nCat.__\n\n\nCat\n\n\ntest = 1\n\n\ntest\n\n\ndef abc():\n    test = 2\n    test\n'}]}}
    Traceback (most recent call last):
      File "c:\git\lsp\jupyterlab-lsp\py_src\jupyter_lsp\types.py", line 96, in __call__
        manager=manager,
      File "c:\git\lsp\jupyterlab-lsp\py_src\jupyter_lsp\virtual_documents_shadow.py", line 172, in shadow_virtual_documents
        await editable_file.write()
      File "c:\git\lsp\jupyterlab-lsp\py_src\jupyter_lsp\virtual_documents_shadow.py", line 36, in write
        return await convert_yielded(self.write_lines())
      File "C:\git\lsp\envs\default\lib\concurrent\futures\thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "c:\git\lsp\jupyterlab-lsp\py_src\jupyter_lsp\virtual_documents_shadow.py", line 51, in write_lines
        self.path.write_text("\n".join(self.lines))
      File "C:\git\lsp\envs\default\lib\pathlib.py", line 1236, in write_text
        return f.write(data)
      File "C:\git\lsp\envs\default\lib\encodings\cp1252.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f415' in position 198: character maps to <undefined>

[dhirschfeld]

I think you want write_bytes there:
https://github.com/krassowski/jupyterlab-lsp/blob/6101ccd78bb88b5dffc72cc709c8009aaf4c74c8/py_src/jupyter_lsp/virtual_documents_shadow.py#L51

[dhirschfeld]

binder/postBuild is written in bash

Depending on how the script is written you can run bash on Windows.
Wiith conda you can install the posix package:

conda install posix

This is quite often used on conda-forge CI to use a single build.sh script which is called from the Windows bld.bat - e.g. https://github.com/conda-forge/winpty-feedstock/blob/master/recipe/bld.bat#L6

[dhirschfeld]

My recommendation on Windows is to always use conda / conda-forge provided tools insofar as is possible.

Edit: It's won't magically fix everything but I think that fewer things will be broken.

[flutefreak7]

I've had challenges with the conda ecosystem several times over the years and have been happily using a pip-based setup for the last two years specifically because there are so many more people using pip and pypi than are using conda and I've found it to be much better supported for a broad range of tasks. My issues were with conda and conda-forge not having libraries I needed, pip not mixing well with conda installations causing completely broken installations, and with issues like PyCharm and pyinstaller not knowing where conda was putting things half the time.

In my experience a library that wants to support Windows should support pip-based installations first, unless you have a ton of non-python dependencies for which conda packages make more sense than wheels and/or you exist as part of a tight web of tools that are already only on conda-forge (I experienced this with FreeCAD for example).

All that said, if you need MSYS or git-bash or whatever (the same as what conda install posix is installing) as a dependency of jupyterlab-lsp, then make it a dependency and document it. It's not unreasonable for a Windows developer to have git-bash installed as part of Git for Windows.

[bollwyvl]

Thanks for the info, folks! We're aware of most of the options presented, I was really just documenting how badly I've set up the content of this repo for windows devs while trying to run down some issues... anybody is welcome to PR fixes!

As to bash-like-things on windows:

yeah, my cross-platform inclination is to reach for python over trying to introduce another bash to windows. I've been bitten by trying to support cygwin-wsl-posh-cmdr-powershell-gitbash. luckily the bash language server we test doesn't need bash!

As to pip-vs-conda:

We maintain, and test, a pip-based setup approach for unit tests... which will get a developer up to being able to test one language server and no UI. We end up using whatever nodejs happens to be lying around on CI. This approach carries the dangling caveat of, "go figure node, and r out yourself." In the readthedocs pipeline, we have to bring our own node (theirs is EOL, last i checked), but then use pip for everything else.

Due to the nature of supporting LSP, we're pulling in at least three languages already, on three platforms, and would love to test more (Julia, elixir, latex, etc.) so yeah, "just pip" is kind of a non-starter. Heck, we do use pip to install stuff that I haven't had the time to PR to conda-forge. But anyhow: what I'm not going to do with my free time is build, document, and maintain rube-goldbergian contraptions of brew-choco-apt-pacman-yum-dnf-snap-flatpak-emerge on two proprietary operating systems and a multiverse of free distros, that at the end of the day can't be distributed as a functioning component short of containers or heavyweight VM. Theoretically PRs welcome, I guess...

[dhirschfeld]

build, document, and maintain rube-goldbergian contraptions of brew-choco-apt-pacman-yum-dnf-snap-flatpak-emerge on two proprietary operating systems and a multiverse of free distros

...and this is a very important benefit you get from using conda - it's not just python packages, you can install all most of the dev tooling you need too, in a cross-platform way.

I'm also surprised when anyone involved in analytics advocates pip as it doesn't have a dependency resolver so can silently break your environment! Since it popped up on gitter this morning the below is a classic example:

pip happily installed numba=0.49.1 which should have been disallowed because of scipy=1.4.1.

The best-case scenario is a loud exception but the likelihood of a silent error (wrong values) scares me - I don't trust results from analysts who use pip unless they've been created from a carefully curated production environment (and not their local venv/pipenv)

TBF: This is a known issue and they're looking to improve it pypa/pip#988

...and in case you're interested, some background on the issues around building wheels:
https://www.xhochy.com/2019/09/15/how-we-build-apache-arrows-manylinux-wheels.html

[bollwyvl]

@stonebig please hoist this to a new issue, with the full version rigamarole... unless you're trying to build from source and make a contribution! The one-line fix is almost certainly for us to add encoding="utf-8", but it's still good to have something dedicated to your concerns.

pip has a dependency resolver)

welp, not to start flamewar, but given the number of packages I've seen that have had to change things to avoid the pip solver from hanging indefinitely... I'm not quite sure it's fully-cooked...

so what "cp1252" is doing there ?

probably what your language server is spitting out on stdout... you can also try running the whole shooting match under a bat script with set PYTHONENCODING=utf-8... but please set up that issue!