5.0.8 - External storage SFTP incredible slow

[osos]

I have the OwnCloud 5.0.8 running at my hosting provider and trying to mount my home NAS directory containing pictures. SSH running on standard port.

This is incredible slow, it seems that OC is trying to index and cache everything before even showing the directory tree.

[PVince81]

Yes it is.
Still happening in OC 6 beta 4.
Note that the extreme slowness only seems to happen for pictures.
I've uploaded a 2 MB pic and it takes several minutes (!!!) to scan it.

[PVince81]

Compared to other external storages, SFTP is the slowest.
I haven't looked into the code yet but I suppose that for each access it's opening a new connection which can already be quite slow (same as with ssh).
Ideally it should try to reuse the same connection.

[PVince81]

I debugged a bit and found out that each login call takes about 1 second.
Then, getFile() for a 3.7 MB file from a computer in the same LAN takes... oh it's still loading. 😦

Tried to debug into phpseclib/Net/SFTP.php and there seem to be a few loops that handle packets.
I'm not sure whether the bottleneck is there or on the network. Might be worth trying to upgrade that library and see whether the new version is faster...

[PVince81]

I tried to add some debug logging.
The get() call is the one from the SFTP library and seems to take one minute for a 3 MB file.
And fopen is called four times for some reason... this is the call when clicking on "download"...

{"app":"sftp","message":"construct","level":0,"time":"2013-11-19T16:12:51+00:00"}
{"app":"sftp","message":"login","level":0,"time":"2013-11-19T16:12:52+00:00"}
{"app":"sftp","message":"fopen: test\/P1060367.JPG","level":0,"time":"2013-11-19T16:12:53+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/\/test\/P1060367.JPG => \/tmp\/406fd65555c5ad0802fa262cdd2414af.JPG ) ","level":0,"time":"2013-11-19T16:12:53+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/406fd65555c5ad0802fa262cdd2414af.JPG ) close","level":0,"time":"2013-11-19T16:13:35+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/406fd65555c5ad0802fa262cdd2414af.JPG ) end","level":0,"time":"2013-11-19T16:13:35+00:00"}
{"app":"sftp","message":"fopen: test\/P1060367.JPG","level":0,"time":"2013-11-19T16:13:35+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/\/test\/P1060367.JPG => \/tmp\/eefb171907a206aa4e184772b5a84ab1.JPG ) ","level":0,"time":"2013-11-19T16:13:35+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/eefb171907a206aa4e184772b5a84ab1.JPG ) close","level":0,"time":"2013-11-19T16:14:17+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/eefb171907a206aa4e184772b5a84ab1.JPG ) end","level":0,"time":"2013-11-19T16:14:17+00:00"}
{"app":"sftp","message":"fopen: test\/P1060367.JPG","level":0,"time":"2013-11-19T16:14:17+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/\/test\/P1060367.JPG => \/tmp\/de6b2c1a923d81d13c51cd677d275689.JPG ) ","level":0,"time":"2013-11-19T16:14:17+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/de6b2c1a923d81d13c51cd677d275689.JPG ) close","level":0,"time":"2013-11-19T16:14:59+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/de6b2c1a923d81d13c51cd677d275689.JPG ) end","level":0,"time":"2013-11-19T16:14:59+00:00"}
{"app":"sftp","message":"fopen: test\/P1060367.JPG","level":0,"time":"2013-11-19T16:14:59+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/\/test\/P1060367.JPG => \/tmp\/de206acb7bbc694293430d4900a639ae.JPG ) ","level":0,"time":"2013-11-19T16:14:59+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/de206acb7bbc694293430d4900a639ae.JPG ) close","level":0,"time":"2013-11-19T16:15:42+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/de206acb7bbc694293430d4900a639ae.JPG ) end","level":0,"time":"2013-11-19T16:15:42+00:00"}
{"app":"sftp","message":"fopen: test\/P1060367.JPG","level":0,"time":"2013-11-19T16:15:42+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/\/test\/P1060367.JPG => \/tmp\/955947829792dc6709340dbc852cd52d.JPG ) ","level":0,"time":"2013-11-19T16:15:42+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/955947829792dc6709340dbc852cd52d.JPG ) close","level":0,"time":"2013-11-19T16:16:23+00:00"}
{"app":"sftp","message":"get(\/home\/vincent\/sftpmount2\/test\/P1060367.JPG => \/tmp\/955947829792dc6709340dbc852cd52d.JPG ) end","level":0,"time":"2013-11-19T16:16:23+00:00"}
{"app":"sftp","message":"construct","level":0,"time":"2013-11-19T16:16:23+00:00"}
{"app":"sftp","message":"login","level":0,"time":"2013-11-19T16:16:24+00:00"}
{"app":"sftp","message":"construct","level":0,"time":"2013-11-19T16:16:25+00:00"}
{"app":"sftp","message":"login","level":0,"time":"2013-11-19T16:16:26+00:00"}
{"app":"sftp","message":"construct","level":0,"time":"2013-11-19T16:16:27+00:00"}
{"app":"sftp","message":"login","level":0,"time":"2013-11-19T16:16:27+00:00"}
{"app":"sftp","message":"construct","level":0,"time":"2013-11-19T16:16:29+00:00"}
{"app":"sftp","message":"login","level":0,"time":"2013-11-19T16:16:29+00:00"}

[PVince81]

Looks like the latest phpseclib provides a stream wrapper: https://github.com/phpseclib/phpseclib/blob/master/phpseclib/Net/SFTP/Stream.php

Might be worth a try to use that one to directly stream the file instead of downloading it first.

[PVince81]

Stream wrapper is not usable yet: phpseclib/phpseclib#169

[PVince81]

Fixed by #5965