Add language server (LSP) support in notebook server and jupyterlab-monaco #26
Conversation
1ffa3dc to
afeb3f4
Compare
|
Very exciting!
It's also worth noting that LSP integration with codemirror is not
impossible:
https://github.com/brijeshb42/codemirror-lsp
So binding the monaco implementation too tightly should be avoided if
possible, even if the reference implementation lives next to the editor
integration.
I'd also like to point out that sourcegraph was recently open sourced
APL-2.0:
https://about.sourcegraph.com/blog/sourcegraph-is-now-open-source
In a Hub setting, this could act as a service, and provide LSP across all
of an organization's repositories, multiplying the impact of this work.
Presumably, at the Lab level, this would be a separate extension, but it
may be worth considering in the architecture, e.g. multiple, simultaneous
LSPs, or somehow having the server extension unify them.
Finally, if we embrace LSP, we'd want to make the Notebook format
discoverable to language servers. No thoughts on how that might be
accomplished generally, though there might be some worthy experiments to be
done with python, as its machinery is fairly well exposed.
|
|
Agreed. I think that the @jupyterlab/services classes will act as an interface between the server extension and the lab extension:
or
Open source Sourcegraph is very interesting! This part looks relevant: https://github.com/sourcegraph/sourcegraph/blob/master/src/backend/lsp.tsx#L3 I guess that VS Code is another valuable reference implementation although at first glance I can't find a LSP proxy equivalent.
@bollwyvl Can you clarify? Would this provide intellisense when editing a notebook file in the editor (JSON)? If so, this may be a valuable resource: https://github.com/sourcegraph/sourcegraph-extension-api |
|
Great, thanks for writing this up @gnestor
|
I agree. However, I checked out https://github.com/brijeshb42/codemirror-lsp and unforauntely, it actually invisibly mounts a monaco-editor instance to the DOM and uses it to interface codemirror with language servers. So, even the codemirror integration is using monaco-editor under the covers. I think that we should depend on monaco-editor to get this working in jupyterlab and once a solid codemirror integration is available, we can step back and design a generic interface that both monaco-editor and codemirror can use.
I say jupyter_language_server. |
@ellisonbg Since we haven't had a larger team meeting in a while, what's the new jupyter_server work? |
|
@willingc This is something that @Zsailer has been working on. I believe the intention is to decouple it from the notebook repo and standardize the way that clients interface with it (e.g. by inheriting it or by providing server extensions). @ellisonbg and @Zsailer can provide more context. |
|
Thank you so much for raising this up as something generic to use across all frontends, this makes me pleased to review and contribute. |
|
Jupyter Server Design and Roadmap Workshop is one of the community workshops accepted and planned for 2019: https://blog.jupyter.org/jupyter-community-workshops-a7f1dca1735e. I believe @lresende submitted the proposal, but I hope @Zsailer and others interested in jupyter_server will be able to participate. |
Yes! I've been working hard on revamping
👍 Yup! @lresende, @kevin-bates and I worked together to make the proposal happen. To peak your interest, here are some highlights from my current draft:
Obviously there is a lot of conversation that needs to happen once this new draft is posted, as this affects many projects around the jupyter ecosystem. I look forward to discussing this will you all soon. 😃 For some semi-recent discussion on this topic, see this issue. |
|
All very exciting stuff!
Indeed, over in the metadata-services work a number of these things are
important.
Back on THIS topic: that's a nasty, inefficient trick, mounting an extra
editor and therefore webworker per cm instance!
Here's a lighter-weight implementation, that uses the vscode libraries
directly:
https://github.com/wylieconlon/lsp-editor-adapter
I love me some Monaco, but it just isn't going to be the answer to 💯
editors on the page (think big notebooks, spread sheets,
blockly/simulink-style hybrids).
|
I pursued a simple and dirty solution:
|
|
Hi there! I work on the PowerShell language server. We're unique in that since PowerShell is also a shell (in addition to a language), the interactive/REPL experience is also very important. To support this experience, we add on a message to the Language Server Protocol that we call The reason I bring this is up is because between us and Jypiter, we are more than one scenario for the language server protocol to support a native There might be some conversation that could be had between Jupyter & LSP to better support each other. This plays nicely with both Jypyter adoption skyrocketing in the LSP space (python ext for vscode Jupyter support, Azure Data Studio top notch Notebooks experience) and the want to bring LSP goodness (all the symbols support and whatnot) to Jupyter scenarios. |
|
@TylerLeonhardt Interesting stuff! Specific to your case, you may wish to reach out to the folks on the powershell kernel! More broadly, there's a bit of impedance between the JSON RPC LSP model and the mixed state model of the Jupyter Kernel Message spec. Can you interrupt an We're working some of these issues on jupyterlab-lsp (mentioned above), considering multiple clients talking to multiple servers, extension into the multi-user space, "middleware" to support language (server) quirks, etc. The implementation is getting more robust, and we hope at some future time to be able to propose it for inclusion into Jupyter(Lab|Server) core. But, last i checked, LSP defines about 120 message types, and seems to be growing at a fair clip. I think we handle maybe 10 at this point. So there's a lot to do!
Indeed, I've been tinkering with the opposite direction, making a Language Server out of a kernel! I think, today, one could definitely write a one-size-fits-most wrapper kernel that, with a little bit of JSON(-E) skullduggery, could wrap up an existing Language Server JSON RPC in enough metadata to make it play nice with the rest of the jupyter tooling without spreading too much headache in either direction. But as I understand it today, a second language server process would be required, as most Language Servers are not written to accept multiple clients with different capabilities, and no doubt a kernel would be a good deal different than, say, an editor.
In the short term, there's some funding available for just such outreach, and we might propose something as part of jupyterlab-lsp. Hinted at in that issue, I think JKM, LSP would benefit better human- and machine-interoperable specifications, without reference implementation hangovers. The best model to follow here is probably DAP, but I would prefer something a bit more PR-able and composable than a monolithic JSON file (ahem notebooks).
I've got a heavily WIP branch which adds (hierarchical) symbols, and indeed, it's really quite nice. We've got some plumbing to do be able to land that, but should be able to start going after those 100-odd missing messaged! |
|
I just wanted to highlight that a new issue on the jupyterlab-lsp repository future in jupyterlab/team-compass, which may be of interest of the wider Jupyter community (especially of those who have engaged in this conversation on Language Server Protocol adoption). Please feel invited to chime in and give your thoughts over here, even if your use case/interest is not specific to JupyterLab, but relevant to the the lsp extensions that we have developed. |
|
Thanks for putting so much into the PoC and documentation here and elsewhere. Seems like there's momentum on this JEP atm to discuss more seriously. I think we need to assess if Jupyter is willing to support a second protocol for the JEP side of things here. A lot of good work was put into prototyping this out and making a working example and I don't want to discount that work but as I understand it, it would be a serious shift for a lot of libraries to adopt a new protocol and I would want to have an idea of remaining work and migration plans (not in full but at least a high level) if the community were to move forward with lsp based language interfaces. In an attempt to better understand, here's my take on the proposal so far? LSP brings:
Cost of Core LSP adoption:
Unknowns (for me at least):
|
|
As an observer, I'm also curious about overlap with the debugger- it sounds like both of these have backend components to replicate the notebook code as files on disk? Can these be unified? Another potential cost of the LSP is supporting full notebook Python syntax- cell magics are called out in #26 (comment) but line magics are a bit more complicated. I've thought that a kernel API which provided the result of inputtransformer along with a source map of the transformations would be needed to avoid otherwise effectively deprecating line magics, but there may be alternate shortcuts. |
|
@blois both magics (cell and line) are supported in jupyterlab-lsp, but the implementation (as all transformations) is on the frontend. Have a look here: https://github.com/krassowski/jupyterlab-lsp/tree/master/packages/jupyterlab-lsp/src/magics. The idea was that one could install additional extensions providing magics support for specific kernels/packages (such as rpy2), as many magics are defined in third part extensions and are not exclusive to a single kernel. Having JupyterLab in mind I designed it to be frontend (typescript) oriented, but now I see that for the wider community it might have been useful to have all of this logic on backend (not saying that this would be a perfect solution either)... Kernels could provide simple transformers as JSON-packages (regular expression + name), but the problem is that one cannot transform more complex things with regular expressions alone... |
|
@blois @MSeal just want to highlight that in the jupyterlab-lsp (PoC) implementation of LSP we went for the kernel agnostic approach, which has a benefit of working even if your kernel dies or resets. In fact in the very first PoC version the LSP servers were running from the local machine and we were hypothesising allowing users to connect to to third-party LSP server provider (like sourcegraph). This is very different from the debugger team approach. Edit: of course, even if we were designing the final version to work without the kernel, the kernel could still provide extra features (such as the magics transformations etc). Edit 2: I probably should highlight that what I and @bollwyvl referred to in the other thread as reusing kernel comms is distinct from the changing kernels comms (and actually does not require any changes in the jupyter comms specs at the moment) |
|
magics:
yeah, nothing but the kernels (or kernel extensions) that offer them really
know what they mean! We have done some thinking about that:
https://github.com/krassowski/jupyterlab-lsp/issues/191
multiple message sources:
oh yes, please!
https://github.com/krassowski/jupyterlab-lsp/issues/184
multi-language-files-on-disk:
unifying them would be swell! But then, dap/lsp don't _exactly_ agree on
what a file even _is_ (uri vs uri|path|???).
multiple jupyter clients:
- nteract: does it want to be an IDE?
- nbclient: what would it _do_ with knowledge of diagnostics which might
take longer to generate than running a notebook? With the offer to run a
formatter?
|
|
Just a note from Jupyter Book's perspective, we support an enhanced flavor of markdown called MyST markdown that has all of the syntax needed for Sphinx features. We've already got an LSP implementation of the MyST markdown syntax (and a corresponding vscode plugin for it. It would be great to have LSP within Jupyter tools so that users could benefit from this there as well. (not to mention that there are LSPs for rST etc as well). All of this is to say that I think in some cases, LSP is not just for developers, but also for people writing content with markup languages. |
Looking at that code, i was getting excited, as I've been looking for a robust, easy-to-install markdown language server. But alas, it appears to be An extension to An LSP Client, not An LSP Server that would work with Any LSP Client. Anyhow, it's also not published on npm, so it's hard for anybody outside the vscode walled garden to reuse it... and somebody could namesquat it right now, and you'd have no recourse. To move any of that closer to An Jupyter Client, I'd probably start with a codemirror mode, which all of the Jupyter clients could then use. I charted a really dark path to using your tmbundle directly (spoiler: it involves As for the multi-language embedding (neat!): that's what the magic/tranclusion stuff above is on about. The challenge is whether we get to a place where those kinds of rules can be declared and shared, rather than having to be reimplemented everywhere.
Again, on the lookout for an easy to install/extend one! |
With regard to LSP-support, this is something that I would expect to have as part of the nteract desktop app (and the core SDK). I believe that the features that an LSP provides are useful, even in a non-IDE important. On the other hand, things like debugging support don't make sense to include in the desktop app but we do have an issue to track adding support in the SDK so that other apps using our JS packages can integrate debugging.
The LS protocol is evolving so both clients and kernels have to be mindful about staying on top of the protocol. IMO, its a fair price to pay for the benefits of consistency with other tools in the editor ecosystem. |
|
Thanks @captainsafia.
Would the existence of a reference "LSP Proxy Kernel" to any number of Language Servers that spoke LSP on comm-per-server help move nteract closer to offering that to users? Or is it more likely to reuse all the vscode stuff on the node side and do a custom bridge to the chromium side? |
I suspect that the costs for both would be about the same. All of the experimentation I have done has been around using VS Code's Node APIs to connect to a language server that I manually launch. An LSP proxy kernel might introduce some other challenges that would typically be handled by the VS Code's LS APIs. The implementation cost for nteract aside, I'm in favor with any implementation that is relatively client-agnostic. Having access to Node-specific APIs shouldn't be a deterrent to clients that might not be implemented in node. An LS-over-comms approach might make more sense for that. |
|
Would this work with theia as well as vscode? Does nteract also implement LSP?
https://github.com/eclipse-theia/theia
https://en.wikipedia.org/wiki/Language_Server_Protocol
Til about https://langserver.org :
A community-driven source of knowledge for Language Server Protocol implementations
There's not yet a Markdown (or MyST) implementation listed.
…On Fri, Jun 12, 2020, 10:16 PM Safia Abdalla ***@***.***> wrote:
Would the existence of a reference "LSP Proxy Kernel" to any number of
Language Servers that spoke LSP on comm-per-server help move nteract closer
to offering that to users? Or is it more likely to reuse all the vscode
stuff on the node side and do a custom bridge to the chromium side?
I suspect that the costs for both would be about the same.
All of the experimentation I have done has been around using VS Code's
Node APIs to connect to a language server that I manually launch.
An LSP proxy kernel might introduce some other challenges that would
typically be handled by the VS Code's LS APIs.
The implementation cost for nteract aside, I'm in favor with any
implementation that is relatively client-agnostic. Having access to
Node-specific APIs shouldn't be a deterrent to clients that might not be
implemented in node. An LS-over-comms approach might make more sense for
that.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#26 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAMNS7QSC3A3DT4V5J7YE3RWLOQJANCNFSM4F7CCSVA>
.
|
Considering the proposal will also start exposing portions of the JupyterLab logic as part of language servers, this would actually also help VS Code to adopt/expose these functionalities on their environment. Have we thought about the consequences to the Lab ecosystem, particularly when we consider the discussion we had in the JupyterLab vision for the next few years. |
Thus far, we've tried to avoid co-mingling anything client-related in the server end of things: it depends on However, as the websocket pipe is not standardized to the extent that Of course, it's not strictly possible to entirely divorce the language server discovery from client configuration, of course, and for any given language server, one will see see their long page of how to configure things for, usually, VSCode, neovim, then one or two other things, depending on the community of interest.
If you can expound on this some, that would help. But basically, we're trying to reuse existing, externally maintained capabilities, rather than hoping folks with re-implement them, per-syntax, just for Jupyter clients. An example is code formatting: while we the client we've implemented doesn't support it yet, once it can create and accept the proper LSP messages, then any syntax with a language server will be able to use it... and it will likely work exactly the same way as as any other system. The more we can work at the protocol level and get better language features without worrying about "which client get it (first)," the better we will be able to offer users robust, open source tools. |
Exactly, in Elyra's case, we were prototyping getting pipeline validation, syntax-check, etc all done via LSP, which then just also worked in VSCode, making the |
Add language server (LSP) support to notebook server/jupyter-server and jupyterlab-monaco
Problem
The Language Server protocol is a protocol for programming languages to provide code intelligence such as auto complete, go to definition, and find all references to client applications such as code editors (like VS Code) and code managers (like Sourcegraph and Sourcegraph for Github).
Once a language server is created for a programming language (e.g. Python language server), it becomes an
m + nproblem because each client application must only build an integration with the server (e.g. Python for VS Code) vs. building the code intelligence parts and the integration.The overall complaint is that the JupyterLab/Jupyter Notebooks code editing experience is missing many of the luxuries of modern code editors and IDEs, like static code analysis for features like autocomplete for unexecuted code, type checking, linting, formatting, go to definition, and find all references. For a popular review of these complaints, see slides 60-71 from I Don't Like Notebooks.
Proposed Enhancement
The monaco-editor is an
open-source text editor written in Javascript and used in VS
Code.
jupyterlab-monaco is a JupyterLab
extension that allows users to edit documents using monaco-editor.
monaco-editor ships with language features for
TypeScript, JavaScript, CSS, LESS, SCSS, JSON, HTML. In order to support other
languages (e.g. Python), it must connect to a language server (e.g. Python language
server).
TODO
/lsp/python/or/lsp/r/.@jupyterlab/services
that provide a client-side interface to those endpoints?
Notes
Libraries
NPM module to connect Monaco editor with language servers
Reference implementations
References
@ellisonbg @ian-r-rose @jasongrout