Intellisense stop working the moment it connects to remote jupyter server

[Dgelemi]

Environment data

• VS Code version: 1.62.2
• Jupyter Extension version (available under the Extensions sidebar): v2021.10.1101450599
• Python Extension version (available under the Extensions sidebar): v2021.11.1422169775
• OS (Windows | Mac | Linux distro) and version: windows 10
• Python and/or Anaconda version: Python 3.8.10 (locally); 3.8.8 in wsl2
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Remote - WSL2 ubuntu 20

Expected behaviour

Open a notebook and see IntelliSense underlying missing variable, highlighting import not used and other functionalities.

Actual behaviour

Nothing happens and features are not appearing, text is all same colours

Steps to reproduce:

Launch jupyter server on wsl2 terminal.
Open VScode and open a notebook; at that stage the notebook has IntelliSense working (I can see different colours for example)
click on yes to connect to remote server
no more colours or info in the notebook

1. What happens
   Before
   
   Open dialogue
   
   Click yes, then this is what is displayed:
   

Logs

Not sure if it's all of it:

[2021-11-17 15:28:30.565] [exthost] [info] Extension host with pid 47272 started
[2021-11-17 15:28:30.566] [exthost] [info] Skipping acquiring lock for c:\Users\user\AppData\Roaming\Code\User\workspaceStorage\3377dadeb7f1f6022e096e6aa1b5a108.
[2021-11-17 15:28:31.000] [exthost] [info] ExtensionService#_doActivateExtension vscode.microsoft-authentication, startup: false, activationEvent: 'onAuthenticationRequest:microsoft'
[2021-11-17 15:28:31.015] [exthost] [info] ExtensionService#_doActivateExtension amazonwebservices.aws-toolkit-vscode, startup: false, activationEvent: 'onLanguage:python'
[2021-11-17 15:28:31.295] [exthost] [info] ExtensionService#_doActivateExtension ms-python.python, startup: false, activationEvent: 'onLanguage:python', root cause: ms-python.vscode-pylance
[2021-11-17 15:28:31.368] [exthost] [info] ExtensionService#_doActivateExtension ms-toolsai.jupyter, startup: false, activationEvent: 'onLanguage:python'
[2021-11-17 15:28:31.470] [exthost] [info] ExtensionService#_doActivateExtension VisualStudioExptTeam.vscodeintellicode, startup: false, activationEvent: 'onLanguage:python'
[2021-11-17 15:28:31.515] [exthost] [info] ExtensionService#_doActivateExtension vscode.ipynb, startup: false, activationEvent: 'onNotebook:jupyter-notebook}'
[2021-11-17 15:28:31.523] [exthost] [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: true, activationEvent: ''
[2021-11-17 15:28:31.531] [exthost] [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '', root cause: atlassian.atlascode
[2021-11-17 15:28:31.555] [exthost] [info] ExtensionService#_doActivateExtension redhat.vscode-yaml, startup: true, activationEvent: '', root cause: atlassian.atlascode
[2021-11-17 15:28:31.750] [exthost] [info] ExtensionService#_doActivateExtension Oracle.oracledevtools, startup: true, activationEvent: ''
[2021-11-17 15:28:32.930] [exthost] [info] ExtensionService#_doActivateExtension zhuangtongfa.material-theme, startup: true, activationEvent: ''
[2021-11-17 15:28:33.578] [exthost] [info] ExtensionService#_doActivateExtension ms-toolsai.jupyter-renderers, startup: false, activationEvent: 'api', root cause: ms-toolsai.jupyter
[2021-11-17 15:28:33.676] [exthost] [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: ''
[2021-11-17 15:28:33.687] [exthost] [info] ExtensionService#_doActivateExtension atlassian.atlascode, startup: true, activationEvent: '*'
[2021-11-17 15:28:33.908] [exthost] [warning] [redhat.vscode-yaml] Accessing a resource scoped configuration without providing a resource is not expected. To get the effective value for '[yaml]', provide the URI of a resource or 'null' for any resource.
[2021-11-17 15:28:33.965] [exthost] [info] ExtensionService#_doActivateExtension vscode.configuration-editing, startup: false, activationEvent: 'onLanguage:jsonc'
[2021-11-17 15:28:33.976] [exthost] [info] ExtensionService#_doActivateExtension vscode.json-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
[2021-11-17 15:28:34.003] [exthost] [info] ExtensionService#_doActivateExtension vscode.typescript-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
[2021-11-17 15:28:34.057] [exthost] [info] Eager extensions activated
[2021-11-17 15:28:34.184] [exthost] [info] ExtensionService#_doActivateExtension vscode.github-authentication, startup: false, activationEvent: 'onAuthenticationRequest:github'
[2021-11-17 15:28:34.236] [exthost] [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
[2021-11-17 15:28:34.246] [exthost] [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-wsl-recommender, startup: false, activationEvent: 'onStartupFinished'
[2021-11-17 15:28:34.285] [exthost] [info] ExtensionService#_doActivateExtension eamodio.gitlens, startup: false, activationEvent: 'onStartupFinished'
[2021-11-17 15:28:34.328] [exthost] [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-containers, startup: false, activationEvent: 'onStartupFinished'
[2021-11-17 15:28:34.375] [exthost] [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-wsl, startup: false, activationEvent: 'onStartupFinished'
[2021-11-17 15:28:36.248] [exthost] [info] ExtensionService#_doActivateExtension ms-python.vscode-pylance, startup: false, activationEvent: 'api', root cause: ms-python.python
[2021-11-17 15:28:38.642] [exthost] [info] ExtensionService#_doActivateExtension mechatroner.rainbow-csv, startup: false, activationEvent: 'onLanguage:plaintext'

Infos

This was working fine few weeks ago, but I don't know what changed to have the behaviour now.

Thank you

[rchiodo]

Thanks for the bug. This is because the remote server can't be used to provide pylance with a python interpreter. We should likely change it to default to the active interpreter if remote.

[Dgelemi]

Thank you, is it an option I can change in settings?

[rchiodo]

Sorry but no. It's a function of how pylance is started. The only way to change it is to not use a remote server.

[zpincus]

I see that this is on the Nov 2021 milestone, but from the above discussion and that in #8348, it also sounds like it's not completely fixable -- e.g. pylance can't currently communicate with the environment on remote servers, if I understand, and that isn't likely to be fixed in short order?

Might it be possible in the future for users to be able to configure their remote jupyter environments in a way that can support pylance? Or alternately, I and many others run "remote" jupyter servers on localhost in the same environment that vscode can see, so that the notebooks' lifecycles aren't linked quite so intimately to that of vscode's UI. Could there be a way to support this use-case? (Or of course, just make it easier to persist the "local" jupyter server sessions... but that's a separate issue / kettle of fish.)

[rchiodo]

Yeah there's no way to get pylance to run on a remote server. The best way to do that is to use one of VS code's remote solutions and let us run jupyter on the remote server. (Then the VS code extension host will run on the remote machine and pylance can run there)

That doesn't meet your likely use case though. You want to control the remote kernels/server (probably because we don't have any kernel management or for other reasons).

So intellisense in remote scenarios will probably be nerfed for some time.

I think it's more likely we'll have a kernel management story before we trick pylance into working with a 'local' remote server.

[rchiodo]

We might be able to have a barebones intellisense story for remote (pylance will only read the active notebook and none of your local source files, well because they're not local). That will likely be doable in the short term.

[zpincus]

Thanks @rchiodo! I'd honestly prefer to just use the local jupyter server since it seems to be the preferred and best-supported use-case; unfortunately right now it's particularly tough to work with local jupyter servers in combination with the remote-ssh extension. This is because the remote-ssh extension still requires a UI reload at least once a day when it can't reconnect to the server within some time window, thus trashing all existing local jupyter sessions. (FWIW my internet connection is solid and the remote server is a cloud instance so it's solid too -- but reconnections still fail a lot during sleep/wake and similar, and require the UI to reload.)

Even if I weren't using remote-ssh, it would be nice to decouple the lifecycle of local jupyter sessions from that of the vscode UI (to allow long-running sessions to persist through UI reloads for extension updates or vscode restarts), but while that's a "nice-to-have" when developing locally, it's a lot closer to a "need-to-have" for effective development via remote-ssh.

There are plenty of github issues for remote-ssh about difficulties reconnecting, but generally forcing a UI reload seems to be considered a consequence-free operation... This is a problem that sort of seems to fall within the cracks between vscode-jupyter and vscode-remote.

[Syndorik]

@rchiodo, do you know where we can follow the progress of kernel management in Vscode?

[rchiodo]

Honestly there isn't just one issue. Anything with a label that contains the word 'kernel' would be appropriate.

Our iteration plan (current one) outlines the work we plan to do in the next iteration. We also use milestones to track work that is slightly farther out.

For example, this issue is scheduled for the January milestone (shipping early February).

[zpincus]

Here are a couple of the relevant kernel management issues, with various comments I found useful in figuring out workarounds: #3998, #1378, maybe #1654. To me #3998 is the biggie in terms of being able to fix most of these issues if the proposed changes were implemented. Also, #7976 has some relevance.

[zpincus]

I just noticed that intellisense/pylance will still happily do its static analysis on notebook contents when the overall connection is to a remote server, but there is no specific kernel selected for that notebook. (E.g. I connect to a remote server, and open up an old notebook which vscode doesn't have any kernel associated -- so the kernel picker button still says "select kernel".)

In that scenario, pylance seems to be happy to run in whatever its default environment is, and provide hovers and completions and code coloration and whatnot, just as if the notebook were any other .py file being edited in that vscode session. Only once I connect to a new kernel on the remote server does all of that go away, to be replaced by basic syntax highlighting and whatever completions jupyter provides.

Is it really not conceptually / architecturally possible to provide a setting whereby for remote jupyter servers the default pylance would still be used for notebook editing instead of (or merged with) the completions provided by jupyter?

Obviously this would not work / make sense in the case that the default/local pylance is running in a totally different environment than the remote jupyter. But even if I had to click "OK" every time to say "yes yes I pinkie-swear that my remote jupyter is running in the same environment that vscode has access to", this would be a pretty big usability win in such scenarios.

It just seems odd for the notebook code editing experience to be categorically better with no kernel connected than with a remote kernel.

[rchiodo]

It just seems odd for the notebook code editing experience to be categorically better with no kernel connected than with a remote kernel.

Yes it's possible to fix this. It should use the local default interpreter in that case. It just happens to be a bug right now that when you select the remote kernel, the filtering logic internally then goes - no match for any local interpreter - and skips all intellisense.

[zpincus]

Oh great! I had misunderstood that all of this was unfixable. But the only part that's difficult/impossible is to somehow get pylance running on the remote jupyter session, whereas just using the local pylance will be back in place once the filtering bug is fixed. Sorry for the noise :(

[Dgelemi]

Thank you @rchiodo

[Martin-Molinero]

Hey all!
Before I open a new issue wanted to ask here: I'm seeing that docstrings are not showed for remote jupyter server, wanted to double check this is a known issue or not? Is there some specific setup/configs/versions for this to work in the remote jupyter server?

What's interesting is that autocomplete does work, suggesting methods/properties/etc, which makes me think that the docstrings should or are very close to be available

Using:

• VSCode insider 1.64.0
• Jupyter extension v2022.1.1001614873

Thanks!

[rchiodo]

@Martin-Molinero can you open a new issue? Docstrings should be working in remote scenarios (well for insiders).

Can you also include a gif or image of what it looks like?

This is what it shows for me:

CTRL+k, CTRL+i should bring it up if hover doesn't work.

[Martin-Molinero]

Hey!
I can confirm it's working with the insiders version of vscode and extension, I also had to update to the latest jupyterlab & hub which was what I was missing, sorry I didn't update my comment.
Could you clarify a couple of thoughts: in this remote context, is autocomplete driven/created by the extensions host side or the remote notebook side is sending it through? Would it work without having pylance extension?
Is there an ETA for this insiders release? 😁

Thanks!

[rchiodo]

Autocomplete in remote is driven by two things (we have two sources for autocomplete):

1. Pylance running on the local machine. It can read your local files and uses the active python environment for python files to determine what packages it thinks are installed.
2. Jupyter provides autocomplete itself. We talk to the jupyter kernel to get autocomplete suggestions as well. We combine these with the data from pylance (to remove duplicates).

The second one will work with or without pylance installed.

Remote support should ship sometime in the beginning of February.

[Martin-Molinero]

Thanks again!