R: completion not available for tidyverse packages

[syu-id]

tidyverse is a collection of some of the most popular packages in R which all play within the "tidy data" framework. library(tidyverse) reduces the effort to load various tidyverse packages by loading all the core packages at once: ggplot2, tidyr, dplyr, etc. It should be noted that tidyverse uses a rather sophisticated method to load the dependencies.

The problem: Currently it seems that completion for tidyverse packages is not available on JupyterLab if they are loaded with library(tidyverse) rather than loaded individually. For example, no completion is suggested for ggplot2's plotting functions such as geom_*.

However, developers of R's languageserver package have already addressed tidyverse's problem, and I can confirm that completion for tidyverse packages works nicely on Vim with the LanguageClient-neovim plugin. So the current issue might be specific to jupyterlab-lsp.

randy3k, languageserver's developer, suggests that the problem might have something to do with the following issues:

• language server should be able to fufill requests even without textdocument/didOpen Notifications REditorSupport/languageserver#15
• Completion on user-defined functions doesn't work. REditorSupport/languageserver#27

My environment:

• jupyter-lsp: v0.6.0b0
• jupyterlab-lsp: v0.6.1
• languageserver (R): v0.3.2

[krassowski]

Thanks for checking our extension and opening this detailed issue. Does it happen in notebook only or in the file editor as well? If it's notebook only, is probable because they do not support some functionalities for synced documents (a pity, many servers assume that they have access to a file on the filesystem when they should not). Either way, I have an interest in fixing it and will get back to you tonight (in ~10 hours time).

…

On Tue, 5 Nov 2019, 07:47 Shaoyun Yu, ***@***.***> wrote: tidyverse is a collection of some of the most popular packages in R which all play within the "tidy data" framework. library(tidyverse) reduces the effort to load various tidyverse packages by loading all the core packages at once: ggplot2, tidyr, dplyr, etc. It should be noted that tidyverse uses a rather sophisticated method <REditorSupport/languageserver#20 (comment)> to load the dependencies. *The problem*: Currently it seems that completion for tidyverse packages is not available on JupyterLab if they are loaded with library(tidyverse) rather than loaded individually. For example, no completion is suggested for ggplot2's plotting functions such as geom_*. However, developers of R's languageserver package have already addressed <REditorSupport/languageserver#20 (comment)> tidyverse's problem, and I can confirm that completion for tidyverse packages works nicely on Vim with the LanguageClient-neovim plugin. So the current issue might be specific to jupyterlab-lsp. randy3k, languageserver's developer, suggests <REditorSupport/languageserver#20 (comment)> that the problem might have something to do with the following issues: - REditorSupport/languageserver#15 <REditorSupport/languageserver#15> - REditorSupport/languageserver#27 <REditorSupport/languageserver#27> My environment: - jupyter-lsp: v0.6.0b0 - jupyterlab-lsp: v0.6.1 - languageserver (R): v0.3.2 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/krassowski/jupyterlab-lsp/issues/95?email_source=notifications&email_token=ABMQBRQTBO5EIFEGIBLIMWTQSEQH5A5CNFSM4JI63V22YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWZ64QQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMQBRVPMQUMTYFQVP2TX3DQSEQH5ANCNFSM4JI63V2Q> .

[syu-id]

Thank you for the reply!
Unfortunately, completion for tidyverse packages didn't work in either notebook or file editor.
Support for tidyverse would be great because these packages are virtually indispensable for many R users.

[krassowski]

@RongMu I have just tried with the latest (not yet released) version of R languageserver and with the latest (not yet released) version of jupyter-lsp and jupyterlab-lsp and it worked for me:

I installed it with:

source("https://install-github.me/REditorSupport/languageserver")

Not sure if there was a fix between 0.3.2 and master or if the change in paths handling on our side did the job, but please try giving it a go with R's languageserver master.

[syu-id]

Thank you for testing out the issue.
Yes, completion works fine if packages are loaded individually eg. library(ggplot2).
The trouble is that if you use a single shot of library(tidyverse) to load the tidyverse packages, then completion for these packages is not available on JupyterLab (but it is available on other editors such as Vim).

[randy3k]

tidyverse uses a non stadnard way to attach packages to the current workspace. Due to computation cost, we only resolve the package dependencies of library(tidyverse) when the file is saved or opened.

[bollwyvl]

Ah, I feared that. There are some features of pyls that must have the same requirements, and no doubt, this will happen all over.

I've been dreading when we really had to start doing something different than other implementations, but it is kind of inevitable for the notebook case.

Crazy Option 1:
Over on https://github.com/krassowski/jupyterlab-lsp/pull/81#issuecomment-547914298, i proposed a potential way to react to incoming languages. We could (ab)use that to actually write files out to disk (buh).

@mgr.on_message(languages=".*", methods=r"textDocument/didSave$")
async def do_thing_with_message(message, sender, session):
    uri = message["textDocument"]["uri"]

    if ".lsp.ipynb" not in uri:
        return message

    if message["method"] == "textDocument/didChange":
        await patch_on_disk(uri, message["params"]["contentChanges"])

Crazy Option 2:
If URIs are actually URIs, and not always file:///s, we could run an https://localhost:8888/lsp/virtual and offer the files up that way, and never actually write to disk, but still use the listener interface. But i have a feeling that won't work for most servers.

Crazy Option 3:
Or... we start submitting PRs to language servers to start supporting .ipynb. We've got a leg up on this for python with https://github.com/deathbeds/importnb, and i've tried some approaches, this sounds like a ton of work.

Crazy Option 4:
We start patching Kernels to provide language server support.

[krassowski]

@bollwyvl I suspect that we do not need to go so far for this case. LSP offers textDocument/didSave and I think that what Randy meant is that they listen for this signal rather than watch files. Anyways the issue is on us - as far as I remember we do not emit this signal.

But you are right that we need something like this for pyls - they use rope for renaming and it works exclusively on the filesystem (and this is why we do not have rename action in notebooks yet - experimented with it a few days ago)

Crazy option (1) sounds like the best short-term solution. The question is where and how to write notebooks. Probably we would want plain hidden files in the same directory as notebooks so that advanced refactoring tools do not have a problem with unresolved relative paths.

[randy3k]

@bollwyvl I suspect that we do not need to go so far for this case. LSP offers textDocument/didSave and I think that what Randy meant is that they listen for this signal rather than watch files. Anyways the issue is on us - as far as I remember we do not emit this signal. ...

Exactly, we listen for didSave and didOpen notifications to perform some of the more intensive jobs.

[randy3k]

I just realized that implementing textDocument/didSave is not sufficient to fix this. It is because languageserver loads the document from the disk upon textDocument/didOpen and textDocument/didSave. It will require some work in languageserver to support virtual documents.