SSH: ValueError: Can't create any SFTP connections! #16
Comments
|
@gcoter Maybe you could try to reproduce it with a docker image with the same config options? So far we are not able to reproduce ourselves. |
|
@efiop Yes it is a good idea, actually I deployed the SFTP server as a docker container on my Raspberry Pi. However, the image I used has been built for ARM. I will try to reproduce the error locally on my computer with the original docker image. |
|
I created a quick SSH server in a Docker container, mounted my DVC cache into it, and set it as the remote for my local project. Running |
|
Also potentially relevant: when I get this error, it is always during the step of querying the remote cache. If I retry enough times and get past that step, then the actual up/downloading always succeeds. |
Do you set any |
No, I didn't use the |
|
Looks like we need to add some limitations on the |
|
Unfortunately still an issue on 2.9.3 |
|
@sjawhar Could you try Also, how many dvc files do you have in |
|
I had the same error. Unfortunately, it is hard to replicate:
I also have directories with tens of thousands of files in the repository. But they where pushed in the past and there was no error. @gcoter @sjawhar Maybe you can try it with an much older version, e.g. 1.2.x. In the past, I did exactly the same steps that resulted in this error with the newest version. With 1.2.x it worked in the past. |
|
Hi @mistermult, thanks for your feedbacks 🙂 Indeed, using an older version worked for me as well and I encountered this issue when using a more recent version. But since I have upgraded DVC (#16), I don't have this issue anymore. Which version of DVC are you using? In my case, I think the issue disappeared after version 2.9.2. |
|
But it is weird because upgrading did not work for @sjawhar 🙁 |
I've tried with There are quite a few outputs in this repo, which might be why I have the issue. There are 18 or so .dvc files, which each track a directory that contains several files. Then each of those 18 directories gets processed through 10 or so stages ( |
|
I'm getting this error still with 2.9.5 |
|
Hi, Would appreciate any help! |
|
@ilankor stack trace: discussion: |
|
@dtrifiro You might want to keep an eye on this one. |
|
Thanks! The strange thing is, other users can run "dvc push/pull" from their profiles and the same server |
|
I am having the same issue with dvc 2.41.1 (dvc-ssh 2.20.0) when I try to push on the ssh server two csv files (each about 50 MB). The Server has |
|
Same here on dvc 2.43.1. Setting the log level to A workaround is to increase the value of |
|
@Cnly Thanks for the research and detailed report! Looks like we might need to tweak the Line 116 in a023383 |
|
@efiop Thanks for the quick response! Unfortunately ...
File "xxx/venv/lib/python3.8/site-packages/dvc_objects/executors.py", line 134, in batch_coros
result = fut.result()
File "xxx/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 568, in _exists
await self._info(path)
File "xxx/venv/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
return await func(*args, **kwargs)
File "xxx/venv/lib/python3.8/site-packages/sshfs/spec.py", line 125, in _info
async with self._pool.get() as channel:
File "xxx/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 171, in __aenter__
return await self.gen.__anext__()
File "xxx/venv/lib/python3.8/site-packages/sshfs/pools/hard.py", line 28, in get
raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!I also tried modifying |
|
Thanks, @Cnly ! 🙏 Looks like we'll need a bit more research here. |
This is correct, and we do have client level maximum sessions value which is set to have a limit of 10, regardless of your
These questions are related, and no, it was not changed due to performance complaints. The issue is that the way Previously, The way it works now is that So in theory, it should still be safe to have a relatively high number of jobs. In practice, it may be that there is a problem with session pool implementations in sshfs where it doesn't properly handle cases where the # of active coroutines is larger than the pool size, which is why I initially suggested that we just use |
Thanks for the explanation 🙏
The code appears to be 2 years untouched and we heavily changed the usage upstream, may be worth dedicating some time to review the implementation on top of that change |
|
I think we should also consider exposing |
|
As per my earlier comment, the only semi-viable solution is to use the old version of dvc 2.41.1. However, the old version of dvc 2.41.1 now breaks the VSCode dvc plugin. |
|
@drozzy Does the suggestion above to use |
|
@dberenbaum Yes, |
|
There was a bug in the sshfs soft channel pool handling that caused this issue for cases where After the fix, it should no longer be necessary for most users to set I think it is still worth exposing |
|
@pmrowla Thank you for looking into it! 🔥 |
|
This fix will be available in the next DVC release, in the meantime users using pip installations can also get the fix with |
|
Great work, thank you! Can confirm this fixes my case. |
|
Thanks again for the considered response. I can verify that DVC 2.53 is working fine in our case, even with MaxSessions at the default of 10 from a machine with 64 cores. |
|
Just a side note for pip users: You need to also update |
|
Actually, I may have spoken too soon. My environment includes both dvc 2.53 and dvc-ssh 2.22.1. Sometimes I can simply Some of this erratic behaviour might be due to the pecularities of our network setup, but it is also possible that this issue is still not quite fixed. I'll collect a verbose log next time it happens, but let me know if there is any other information that might help troubleshoot. |
|
Didn't have to wait long. Here is the (slightly truncated) log: See also ... |
|
@johnyaku, are there other users accessing the SSH server at the same time as you? If someone else is eating up the available server-side session count, that would account for why it appears to only work intermittently for you. You can also now use If you have a scenario where multiple users are regularly accessing your server (either via |
|
Thanks @pmrowla. That is indeed quite possible. We have five "data mover" nodes with fast connections to the remote, and so I think most of the time I'll be the only person pushing from a particular node, but there might sometimes be two or more of us working on it at the same time, depening on the who gets allocated where. I'll double check this next time. |
Bug Report
I open this issue as a follow up to iterative/dvc#6138
Description
dvc pushraises an error when trying to push to an SFTP remote. It used to work with older versions. The SFTP remote I use is a personal Raspberry Pi server. I did not change anything on the server.Reproduce
Unfortunately, since I use a private server, I don't know whether it would be easy to reproduce.
After updating dvc, I tried to run
dvc pushand I got these logs:Expected
Since I did not change the configuration of my server and it used to work, I would expect
dvc pushto work.Environment information
The text was updated successfully, but these errors were encountered: