-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OC7.0b1: all SFTP uploads fail, though able to create folders #9320
Comments
as you can see from the SFTP server's log when OC is connecting, the seclib version as follows: debug1: Client protocol version 2.0; client software version phpseclib_0.3 (bcmath) not sure if it's related... |
@PVince81 yes |
if i mount the same sftp server as a local resource by sshfs user@servername.ru/ocloud/ /mnt/sftp -o allow_other,reconnect,idmap=user,uid=33,gid=33,big_writes and then add the mount as an external storage in OC, everything working lighting fast and as expected, including encryption and public sharing encrypted files. The only issue here, that i cannon add it as a user level external storage, only able to add on a server level, thus all users can see and have full control on everything on this resource, including foreign files, and the $user template is NOT working (like a a /mnt/sftp/$user) in a configuration field, when u add this local storage on a server level. |
Okay that's strange.
The connection parameters shouldn't be that different for the personal mount. Needs further research. |
|
Same issue with FTP and FTPS for 7.0.0RC1. Almost looks like the connector is looping/failing. Several thousand FTP connections (login/logoff/repeat) each minute. |
@bantu can you please help investigate this ? |
confirm - nothing changed for sftp or webdav upload issuer with 7.0rc1 - still not working! |
Webdav is broken for you as well? I admit I have not yet tested that one since going to the stable7 branch, however SMB worked fine for me. FTP and FTPS are the important ones, SMB doesn't help me out here at all. I recall seeing that webdav was incredibly slow elsewhere but I'm not sure if it's related or not (webdav is incredibly slow all the time, anyways). |
Actually it's not just slow - it's unusable slow! stated about the issue here #9351 |
I feel your pain. I've spent 72 hours in the last week -- installing OC of multiple versions (along with various versions of PHP, BSD 9/9.1R/9.2R/10R, three versions of MySQL, etc etc). I've gone so far as to rule out ZFS being the issue and switched back to UFS. Oddly enough I was doing all of this to fix other bugs (or at the very least rule out my setup as the culprit). I did not have external maps failing until 7.x. If I recall, webdav did work for me in the past (on 6.something) but it was pretty slow. At that time, everything else was tested and worked (remote maps anyways). I'm now in the process of disabling everything that does not work, so that I can concentrate on getting the main functionality of OC working first. There are simply too many broken items to deal with (albeit, not all are within core, but many are). |
@MarvinFS I do find it interesting how you and I are both running, for the most part, the same software versions. I'd be interested to know if others with earlier packages have these issues. On my 6.x box that was working pretty well, I had some pretty old software packages and dependencies installed. |
@deklar well i'm not sure if we can compare BSD and linux in terms of dependencies - the kernel is still different, as for me, i've tried to figure out those issue with external storages in OC very hard - it's critical for me - without fast secure cloud ext. storage OC is useless in my case, and, since i can't use webdav (dead slow) and sftp (not working directly from OC) i was searching for solution, tried both old ubuntu12 and newer ubuntu14 LTS servers, tried to change apache and php versions... still no luck. |
Well, beta is beta, I guess. I wouldn't expect Alpha/Beta software to work flawlessly. That said, I upgraded to 7 (then re-installed 7 on 6 different platforms with different configurations each time) to get away from the bugs in 6-Stable. The good news is, some of the bugs are gone. The bad news is, you and I both lost the ability to mount externally. In regards to the dependencies, I'm convinced there is something - a library, the version of PHP with the compiled options, some version of pecl, maybe even Apache -- something, that is causing at-least some of the issues. I had a snapshot of my working 6.x box, so I pooled up the /usr/local and overwrote one of my .... now six (yea, six) OC installations. It replaced all versions of PHP, Apache, dependencies, libs, etc. My pkg(1) was/is a mess - the box is hosed - but, it actually worked -- on 6. It fixed most of the issues. Of course the box is destroyed now so time to re-do it again ... at any rate, I then updated the packages (all but OC, configs, etc) and kept everything as much as I could original to the working snapshot. Doing this, broke nearly everything again. The issue here is, OC needs around 200 packages in total to run with most of the options. Ok, ok, it may only have a "few dependencies" but those dependencies more often than not require 2, 3, 4, 10, 16 or more packages installed just to get the ones required to work. Finding out where the issue is (assuming I'm even right on this) would mean going over every single installed package, upgrading one at a time (which is not possible in many cases) and then re-testing every function after each upgrade - it would take weeks. On a fresh 10R machine using the latest stable packages, OC6 has plenty of issues. Using outdated (and in some cases, exploitable) packages, OC6 runs fine. I'm betting here that OC7 would work better (perhaps not with external maps however, although it could be related) with fewer bugs if I ran it on older packages. We need better debugs... |
@MarvinFS phpseclib's "sftp" stream wrapper is not there, but it is also not expected to be there because it is not a built-in stream wrapper. The stream wrappers you are seeing are probably those of libssh2 or so, which owncloud does not use. "phpseclib_0.3 (bcmath)" indicates that you "only" have the bcmath extension which will speedup phpseclib slightly, but not massively. Please try to install the GMP extension to further speed up BigInteger operations and the mcrypt extensions to speedup some other cryptographic operations. |
@MarvinFS Please amend your report with documentation about how ownCloud was installed. Tarball? Git? The debian packages you will want to install are probably called php5-gmp and php5-mcrypt. |
@bantu I know this isn't my PR, but MarvinFS and I have the same issue. I have GMP 5.1.3 and mcrypt 2.5.8 compiled in (I just added GMP now). Here are my test results with FTP: I'm able to map the drive fine (as I was able to before). Creating a single directory, and then creating a single text file, modifying it and saving it, returning to root path then re-entering the remote path connected to the FTP server 291 times. A little excessive, I'd say. I had to configure the FTP service to allow more than 60 connections per minute by any given IP address (which is bad to do for security reasons) just to make this happen. Uploading a single XLS file connected 45 times (in 3 seconds) Something is really wrong here -- if this is by design, the design is flawed.I was able to upload three files, pretty quickly however - so this is an improvement (137 connections in 8 seconds). Syncing the OC Windows client (I know it's not directly related, but probably still part of Core) connected a minimum number of times, for what it's worth. Summary: It's "better" just not great - unless you have an unlimited FTP (connections per second, etc) setup FTP will timeout/have issues. I was also able to delete the mount without any problem here. SFTP works the same method as above - however is failing and is much slower most likely due to the overhead of the SFTP process itself (as it will require much more memory/resources, and will be slower to initiate than FTP). Uploading a 200KB file failed here as well (I suspect because there was an issue creating the 50 or so connections to do this transaction - that should use just one). With SFTP, I do have the OC log: {"app":"PHP","message":"fopen(): Unable to find the wrapper "sftp" - did you forget to enable it when you configured PHP? at /data/owncloud/apps/files_external/lib/sftp.php#220","level":3,"time":"2014-07-08T15:28:59-05:00"} I was able to create a text file, it just took some time to do (again, probably due to the large number of connections taking place). Webdav - |
Please make this ticket about SFTP only. The missing sftp wrapper error |
@bantu You bet. It's an important one too. Anyone with RC1 should be able to simulate this issue. My question at this point is, has anyone been able to get SFTP to work (and quickly for that matter) using 7RC1? Despite the wrapper error, the pure number of connections (I know we are only talking SFTP but you can't deny how this seems to be the case with Webdav and FTP as well) is bloated to an almost fictional number that would never work in the real world. Tell me what I can do to help you out and I'll do whatever I can. Thanks. |
@bantu Forgive me being lame and i totally understand this is not the place for user support. am i missing something? What do you mean "phpseclib_0.3 (bcmath) indicates that you "only" have the bcmath extension" - only means i need some other lib?
the php5-gmp and php5-mcrypt were not installed, after installation and server reboot (just in case) nothing changed - SFTP uploads fails with "the target folder has been moved or deleted" message.
|
I think those errors may be due to an overloaded SFTP server - I've seen them before, but not always - they pop up when OC is doing some dramatic (see #7910) things it shouldn't be doing. Essentially the SFTP server can't keep up with the connections. It may or may not be related to 7910 but I just did testing on the FTP end, and everything I have tested thus far, tells me that the SFTP side does about the same thing. |
@deklar there are no load on a SFTP server at all here - it's a test server. Also, as i've turned on SFTP debugging - i see only a few connections when it's trying to upload files basically about 3 or 4 connections only! Also i may say when i've tried FTP servers (vsftpd) as an external storage with OC7 - it was working OK! no issues, but the FTP is not a solution for me, due to a lot of issues with firewalls. |
I can confirm this on stable7 (9ccadcf) Also getting this error on upload:
The only change since OC 6 is that now phpseclib is bundled in core's 3rdparty instead of files_external's 3rdparty. Maybe the version has been increased as well. Will try installing the extra modules. |
Same problem after installing php5-gmp and php5-mcrypt. I think the basic file operations work, I was able to delete files. |
Okay, I set a breakpoint in if (function_exists('stream_wrapper_register')) {
stream_wrapper_register('sftp', 'Net_SFTP_Stream');
} When uploading a file it doesn't go there, which means there is a class loading issue. Maybe that file needs to be explicitly included when using the SFTP storage classes ? |
Possibly related: https://github.com/owncloud/core/pull/9198/files#r14651400 |
Yes. The sftp wrapper file of phpseclib 0.3.6 needs to be included
|
tried to update phpseclib install ok: channel://phpseclib.sourceforge.net/Net_SFTP-0.3.7 issue still there! unable to upload! |
@bantu pull request to update phpseclib? Thx |
I have properly analysed this issue now, the conclusion is as follows:
|
Dear Andreas, don't be sorry! |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Steps to reproduce
4 Try to download existing files on the server (yes i know that you better not directly modify FS and exclusive OC resource access)
Expected behaviour
Actual behaviour
Server configuration
UBUNTU12LTS
Web server:
APACHE 2.4
Database:
MYSQL 5.5.37
PHP version:
5.5.14
ownCloud version:
7.0b1
Updated from an older ownCloud or fresh install:
fresh
List of activated apps:
only defaults and encryption enabled
The content of config/config.php:
'oc00bb20a276', 'passwordsalt' => '*** Removed for security reasons', 'trusted_domains' => array ( 0 => 'cloud.soulfly.us:8443', ), 'datadirectory' => '/var/www/owncloud/data', 'dbtype' => 'mysql', 'version' => '7.0.0.2', 'dbname' => 'name', 'dbhost' => '127.0.0.1', 'dbtableprefix' => 'prefix_', 'dbuser' => 'user', 'dbpassword' => '', 'installed' => true, 'ldapIgnoreNamingRules' => false, 'loglevel' => '0', 'forcessl' => true, ); **Are you using external storage, if yes which one:** local/smb/sftp/... SFTP **Are you using encryption:** yes/no YES ### Client configuration **Browser:** FIREFOX 30, IE 11 **Operating system:** Windows 7,8.1 ### Logs #### ownCloud log (data/owncloud.log) {"reqId":"53b19dbfcb01b","app":"PHP","message":"fopen(): Unable to find the wrapper "sftp" - did you forget to enable it when you configured PHP? at \/var\/www\/owncloud\/apps\/files_external\/lib\/sftp.php#220","level":3,"time":"2014-06-30T17:26:25+00:00","method":"POST","url":"\/index.php\/apps\/files\/ajax\/upload.php"} {"reqId":"53b19dbfcb01b","app":"PHP","message":"fopen(sftp:\/\/...@server.name\/ocloud\/Artist.srt): failed to open stream: No such file or directory at \/var\/www\/owncloud\/apps\/files_external\/lib\/sftp.php#220","level":3,"time":"2014-06-30T17:26:25+00:00","method":"POST","url":"\/index.php\/apps\/files\/ajax\/upload.php"} _**Any other sftp external clients can upload files on server successfully except owncloud! *_* The PHP's SFTP wrapper is there: ``` php -r 'var_dump(stream_get_wrappers());' array(17) { [0]=> string(5) "https" [1]=> string(4) "ftps" [2]=> string(13) "compress.zlib" [3]=> string(14) "compress.bzip2" [4]=> string(3) "php" [5]=> string(4) "file" [6]=> string(4) "glob" [7]=> string(4) "data" [8]=> string(4) "http" [9]=> string(3) "ftp" [10]=> string(4) "phar" [11]=> string(3) "zip" [12]=> string(10) "ssh2.shell" [13]=> string(9) "ssh2.exec" [14]=> string(11) "ssh2.tunnel" [15]=> string(8) "ssh2.scp" [16]=> string(9) "ssh2.sftp" } ``` target sftp server is debian 7,no selinux, chrooted user ocloud with no shell on /opt/ocloud folder tried all permission variants up to 777 root@server:/opt# ls -la total 12 drwxr-xr-x 3 root root 4096 Jun 30 22:06 . drwxr-xr-x 22 root root 4096 Apr 24 03:57 .. drwxrwxrwx 3 root root 4096 Jul 1 00:39 ocloud Target SFTP server's logs: ``` Jul 1 00:20:22 marvinfs sshd[6021]: debug1: SSH2_MSG_NEWKEYS received [preauth] Jul 1 00:20:22 marvinfs sshd[6021]: debug1: KEX done [preauth] Jul 1 00:20:22 marvinfs sshd[6021]: debug1: userauth-request for user ocloud service ssh-connection method password [preauth] Jul 1 00:20:22 marvinfs sshd[6021]: debug1: attempt 0 failures 0 [preauth] Jul 1 00:20:22 marvinfs sshd[6021]: debug1: user ocloud matched 'User ocloud' at line 79 Jul 1 00:20:22 marvinfs sshd[6021]: Accepted password for ocloud from IP port 58388 ssh2 Jul 1 00:20:22 marvinfs sshd[6021]: debug1: monitor_read_log: child log fd closed Jul 1 00:20:22 marvinfs sshd[6021]: debug1: monitor_child_preauth: ocloud has been authenticated by privileged process Jul 1 00:20:22 marvinfs sshd[6021]: User child is on pid 6023 Jul 1 00:20:22 marvinfs sshd[6023]: debug1: SELinux support disabled Jul 1 00:20:22 marvinfs sshd[5839]: debug1: Forked child 6025. Jul 1 00:20:22 marvinfs sshd[6025]: Set /proc/self/oom_score_adj to 0 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: inetd sockets after dupping: 3, 3 Jul 1 00:20:22 marvinfs sshd[6025]: Connection from IP port 41381 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: Client protocol version 2.0; client software version phpseclib_0.3 (bcmath) Jul 1 00:20:22 marvinfs sshd[6025]: debug1: no match: phpseclib_0.3 (bcmath) Jul 1 00:20:22 marvinfs sshd[6025]: debug1: Enabling compatibility mode for protocol 2.0 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: permanently_set_uid: 102/65534 [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: SSH2_MSG_KEXINIT sent [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: SSH2_MSG_KEXINIT received [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: kex: client->server arcfour256 hmac-sha1-96 none [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: kex: server->client arcfour256 hmac-sha1-96 none [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: expecting SSH2_MSG_KEXDH_INIT [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: SSH2_MSG_NEWKEYS sent [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: expecting SSH2_MSG_NEWKEYS [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: SSH2_MSG_NEWKEYS received [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: KEX done [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: userauth-request for user ocloud service ssh-connection method password [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: attempt 0 failures 0 [preauth] Jul 1 00:20:22 marvinfs sshd[6025]: debug1: user ocloud matched 'User ocloud' at line 79 Jul 1 00:20:22 marvinfs sshd[6025]: Accepted password for ocloud from IP port 41381 ssh2 Jul 1 00:20:22 marvinfs sshd[6025]: debug1: monitor_read_log: child log fd closed Jul 1 00:20:22 marvinfs sshd[6025]: debug1: monitor_child_preauth: ocloud has been authenticated by privileged process Jul 1 00:20:22 marvinfs sshd[6025]: User child is on pid 6027 Jul 1 00:20:22 marvinfs sshd[6027]: debug1: SELinux support disabled Jul 1 00:20:22 marvinfs sshd[6025]: debug1: do_cleanup Jul 1 00:20:22 marvinfs sshd[6013]: debug1: do_cleanup Jul 1 00:20:22 marvinfs sshd[6009]: debug1: do_cleanup Jul 1 00:20:22 marvinfs sshd[6021]: debug1: do_cleanup ``` SFTP external storage users's json ``` /var/www/owncloud/data/user1# cat ./mount.json { "user": { "user1": { "\/user1\/files\/home": { "class": "\\OC\\Files\\Storage\\SFTP", "options": { "host": "servername.ru", "user": "user2", "password": "", "root": "\/ocloud", "password_encrypted": "deleted=" }, "priority": 100 } } } } ```The text was updated successfully, but these errors were encountered: