-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remote Python Development using Spyder #10946
Comments
Not at the moment but we do plan to add this capability for a future version. |
I'm not sure I understand correctly what you want @abhijeetbadhe, but if you have SSH access to your server, you could mount the folder on your local machine using SSHFS: https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh I haven't done this with Spyder, as I mostly develop locally and sometimes connect to a remote kernel when I need some more computational power. But I've done it successfully with other editors in the past, and usually works like a charm! |
@fohria, we'd like to provide is a more integrated experience. So users would only need to provide their SSH credentials and then we'd create a remote kernel for them, show Files placed on the remote project dir and allow files to be edited in our editor. |
Using something like this https://github.com/althonos/fs.sshfs @ccordoba12 ? |
Yep, seems like the right project to use. |
@ccordoba12 Back when we discussed this at considerable length in the context of me writing a grant for Spyder 5 to "Army Labs" (by which I assume you meant either DARPA or ARL), with this as the headline feature, you found spurplus and recommended we use that, since it was a higher level solution than just SSHFS or paramiko alone. Might be worth digging that out if we want to implement something like this since we spent a lot of time discussing the high-level details of the implementation and how we would propose implementing it in stages. By the way, after all the work we both put in on that grant, what happened to it? Did they have any feedback/suggestions for us about this remote development project? Did it even get submitted? Although I remember bugging you about it a lot, I don't recall if I ever got a clear answer; I know you were very busy at the time. |
Both projects seem similar. We'll have to evaluate which one is better maintained and more useful for us.
Sorry but support from TDK Micronas and Kite came in and it was much better than what we proposed to Army Labs. Besides, I couldn't make the contract with them through Quansight, which was a major issue for me. So I decided not to continue talking with them. |
Yeah, and I'm not sure
Considering I put several straight days of my volunteer time into it at your request and you emphasized its urgency and importance to me at a time when I was very busy with other high-priority things, I would really have appreciated that at the time as a straight, honest answer to my many questions about it. But at the end of the day, that's almost a year in the past and we've both learned and grown a lot since then, so hopefully our communication has improved to a point where it won't be an issue in the future. I hope that grant can still be useful, and be submitted somewhere else at some point to get this funded. |
Yeah, sorry about that. I was just in the middle of our initial talks with Kite and TDK people, so I forgot about it.
Well, basically all the ideas on how to improve our remote workflow capabilities came from it, so it was very useful just for that. |
Related, it may nice for Spyder to work with the Jupyter Enterprise Gateway to launch and connect to kernels remotely? |
@fkloosterman, very nice suggestion! We'll see what we can do about it. |
Just curious: RStudio Cloud is a browser-based RStudio -- the front end is elements (such as buttons, scroll bars, etc) that are rendered in the browser, and the back end is the R kernel on the server side. Is it possible for Spyder to do something similar? I don't know enough about Spyder's front end to know the feasibility. I know the scope of this issue is only limited to connecting to a remote back end, but I'm just wondering about this possibility for the future. :) |
No, it's not possible, sorry. |
To expand a bit on @ccordoba12 's response, it is possible, but only in the sense that it would be a completely new frontend, sharing only the Spyder name (and perhaps some backend elements, but those are mostly lower in the stack anyway). It was possible for Rstudio because it was built on a web framework (just like VSCode is based on Electron), so it was relatively easy to port it to the web (but conversely, the Rstudio desktop interface does inherit some limitations as a consequence compared to a "true" desktop application. While some of the panes (Help, Online Help, and the future Viewer pane) do leverage web-based rendering, Spyder as a whole is written entirely in pure Python, and the UI uses PyQt (a Qt binding for Python), powerful, full-featured desktop GUI application development library. This offers considerable advantages of performance, OS integration, and a native experience users expect, alongside being written in the same language it is primarily used to develop to make it easy for users to contribute features and write plugins, but isn't as readily portable to the web. However, if funding or community support is available, there have been plans to considerably expand Spyder's remote development features, including potential integration with Jupyter. Perhaps @ccordoba12 can comment on the current status of that; we've been talking about it for years but money and developer time has always been a limiting factor compared to far more lavishly funded projects like Jupyter. |
Thanks for both your answers @ccordoba12 @CAM-Gerlach . I didn't know that Jupyter is better funded than Spyder. If I had known, I probably would have gone there and ask them: "can Jupyter implement a good variable explorer and a proper debugger, just like what they do in Spyder?" I'm surprised that they still don't (have a variable explorer or a debugger), after this many years. I would say the scope of this issue (i.e., kernel on the server side + front end on the client side) covers 95% of our use cases, and that's quite enough. (The only remaining 5% use cases where we desire a full browser-based IDE is that we somehow want to code on tablets.) |
They get $100 000s or millions from large companies and foundations, I'm not entirely certain where all that money goes, vs. Spyder is developed and in large part crowdfunded by the international community of its users, although it has attracted a rapidly increasing amount of grant money and corporate donations in the past couple years, mostly to implement the most-requested features by users (the two top ones which were better code completion/introspection/analysis and a better debugger, which were the two biggest major enhancements delivered in Spyder 4).
Yeah, I'm not sure, considering even VSCode is getting at least basic support for that sort of thing now. It does help that Spyder is more mature than e.g. JupyterLab and written in Python, attracting a ready pool of talent and avoiding some of the hurdles of web-based development. |
I'd like to clarify that this is possible in theory, but we haven't considered to follow that direction at all. So, in practical terms, you're not going to see a purely web-based Spyder in the future. However, Spyder can run on Binder. That's the closest you'll get to "have Spyder inside my browser" from us. |
Personally, I routinely connect over SSH to a data centre. The files that I sometimes have to test with are in the 5-400 GB range (telescope data). Most of the new software makes use of the NVIDIA GPU resources for several different functions but spyder need not be concerned with that. I can easily live with debugging via print and logging.debug statements as I have for a long time. But, it would be highly productive to be able to use spyder edit & code analysis on a set of Python source files as opposed to individually vi/nano and manual execution of pylint. I could even edit my configuration files, bash scripts, and C source. You might ask, why not use an existing facility?
|
I know at one point @ccordoba12 and I discussed this at length, and following that I actually wrote a grant proposal to get a set of major enhancements related to this funded for Spyder 5, as one of if not the headline new feature. @ccordoba12 had apparently found an agency interested in funding it and I managed to turn a pretty solid proposal around on a pretty short deadline, but unfortunately, for reasons that are still unclear, someone never actually sent along the proposal, so it never happened, and the funding we eventually got for Spyder 5 went in a very different direction instead (the plugin system). However, its possible that the plugin API could be used to develop a plugin that brings better support for this to Spyder. Of the ways I remember doing it, IIRC it came down to either something like SSFS or interfacing with a Jupyter server running on the host to provide the filesystem services we needed. @texadactyl Have you tried the existing SSHFS + connect to remote kernel approach? If you're already connecting over SSH, it should definitely be possible. I'm not 100% sure that |
@CAM-Gerlach
This was already difficult to set up & maintain for 40+ scientists who are highly intelligent but want to remain focused on Physics. Cannot blame them. |
@ccordoba12 Looks like there are at least a couple SSHFS/like backends (e.g. sshfs, fs.sshfs, etc) package on Conda-Forge that could be used for this. How feasible would it be to create a plugin using the new API that would allow, e.g., connecting to, browsing and opening files from them via the Files/Projects pane? We already use paramiko for ssh connections to remote kernels and it sounds like one of them is based directly on it, so such a plugin could reuse a lot of the existing infra. The other component would be the plugin ensuring that the correct (remote) paths are sent to the console; its simple in theory but not sure how hard it would be with the new API. I'm not sure how available someone on the core team might be to work on this, but one of the volunteer core devs might be interested, we could at least help you along, and there's also the community work order process with Quansight; I don't think it would take that much funding to do. I'd also be interested in this myself (though my time is stretched very thing as it is), and I encourage committing some of my OC donations over the years toward this. |
@texadactyl, thanks for the feedback. I'd like to say that this is really top in our priority list, we asked for funding this year and didn't get, but we are expecting to get it from another source next year. Hopefully we'll have an initial solution in a year or so. |
@ccordoba12 I read somewhere in Spyder's website (cannot find it again) that you all got funding to implement this functionality. Any EDA? Edit: I found where I read it |
@raphaelchinchilla, the first part of that project (i.e. a much simplified and friendlier way to start consoles in remote servers) will be released in Spyder 6, to be released at the beginning of next year. |
@ccordoba12 |
Yep, we are.
Ok, thanks for letting us know about it. Do you know how we could set up something like that so we can test it? |
I adapted this from a real-world set up that is in heavy use. Please try it out or the 3-node shortcut (no middleA node).
|
Hi all, I understand very little of the protocols itself. However, there is something called mosh, the mobile shell https://mosh.org/, that allows the connection to the server to drop and reconnect without losing progress. If that could be integrated into Spyder... No other IDE has that |
Hi @ccordoba12 , does your team have access to AWS SageMaker? (Personal/non-profit/education accounts may be cheap-ish.) More and more people are using SageMaker to develop/train machine learning models, so it may be good to test Spyder's remote development on SageMaker. (They currently offer "SageMaker Studio", which is just glorified Jupyter Notebook. And people are asking for "full-fledged IDEs" in SageMaker.) |
@jsh9, we'll try to cover that use case too, i.e. connecting to a remote JupyterHub instance from a local Spyder installation to run computations on it. But that's not part of the plan of this grant. |
Hello,
I was wondering if it is possible to use the Spyder locally (on my mac) for development on remote Linux server with all the features (such as debugging, linting, variable-explorer etc.) available? All my project files would be on the server and the code would also run on the server...
From Spyder documentation and Carlos' posts on several forums, it's clear that it's possible to run a program developed locally on a remote machine (by launching spyder kernels, etc. ), but I have been unable to find whether or not remote development and execution is possible locally?
I would appreciate it if you could let me know...
Thanks & regards,
Abhijeet
The text was updated successfully, but these errors were encountered: