File corruption #2425
Comments
|
@PVince81 I think that might be of interest to you? |
|
@chandon did you enable encryption on the server ? |
|
No, as you can see in my description, the examples files are directly downloaded from the server (so they are not encrypted). A binary compare show that the first 0x4000 bytes are corrupted... like a bad "part" included at the beginning of the corrupted files. You can find the same part later in the file |
|
Verrry weird. This guy also had the exact same issue but with encryption enabled: see owncloud/core#10975 (comment) CC @VincentvgNn I'm suspecting a concurrency / race condition issue... You could try installing and enabling the "files_locking" app. |
|
Do you have the corrupted file open in any application? |
|
no, it's just a copy/paste from my mac osX desktop to the owncloud folder, that's all. neither the desktop files or destination files are open in any application (i don't have virus tool or apps like that, only owncloud accessing files in background) |
|
I think this could possibly be the client's fault. In 1.7 we avoid uploading files that were changed too recently, but in 1.6 the following could happen:
On the other hand I think it's odd that the file ends up having the right size and that we consistently see the first 16kb being replaced by a later 16kb chunk of the same file. I don't see a reason why the first read() would go wrong in this particular way, but it might happen. It could also be a server bug. I'll try to reproduce this locally with the 1.6 branch. @chandon maybe you could try whether you still see the issue with the files_locking ownCloud server plugin installed? And if you can, attempting to reproduce the issue with a 1.7 client would also be appreciated - if it's reproducible there it's much more likely to be a server issue. |
|
I've installed the "files_locking" app for 2 days now. I've done a test, i didn't had any corrupted file (which doesn't mean that the problem is solved...). If we still have any corrupted file with the plugin, i'll let you know in a later comment |
|
@chandon Thank you! I've tried it locally with 1.6 (on ext4, with SSD) using 50 copies of your test_ORIGINAL.pptx, but couldn't get a corrupt file in 10 attempts. I tried again with various levels of io and hdd load, but could not produce corrupt data. I also tried with 500 files; and with sleeping 100ms between each copy. |
|
@chandon Did you observe any more corruption? |
|
@ckamm Yes, i did observe a corruption again, always with the MAC OSX Client sync, and with the files_locking enabled. Always the first 0x4000 bytes corrupted (corresponding to another block later...) |
|
@chandon Thanks for your efforts! Have you tried with a 1.7 client? Do you have any up- or download rate limits configured? |
|
@ckamm I've got the 1.6.4 client. We will install 1.7 client. We don't have up/download limit configured nor proxy |
|
@chandon Thanks. If this issue is due to the client reading files while they are written to the sync folder, it should be solved with 1.7. If you can still reproduce it, it's more likely to be a communication or server issue |
|
@ckamm thanks i will give a try with 1.7. If i still got any error, i'll report it |
|
I have seen recent;y a very similar problem using owncloudcmd in our testing framework (https://github.com/cernbox/smashbox/blob/master/lib/test_nplusone.py). The symptoms were very similar: first 16KB block of the file corrupted (corresponding to the 10th block of that file). However this is NOT related to the file update while syncing because all the files are updated BEFORE owncloudcmd is run. And now I am positively sure that I saw this on 1.6.4. |
|
The question is: was there some subtle change from 1.6.3 to 1.6.4 that could trigger that? Nobody has seen that before. Is the 1.7 exposed to this too? Could someone do some comparison? Especially between 1.6.3 and 1.6.4 sources the changes should not be so many... |
|
I already had the problem with 1.6.3... i updated to 1.6.4 to check if the problem was solved (because of the changelog : "avoid data corruption due to wrong error handling, bug #2280") but the problem was still there. At this time, i don't have any corrupted file with 1.7 (using it for 6 days now) |
|
That #2280 was another corruption (that I discovered on server errors). On 1.6.3 did you also see the 16KB block issue? I have been running many tests on previous versions and never seen that. @danimo, @dragotin: Could that be that Qt HTTP stuff is not multithreaded fully or some buffers are reused? I guess you added Qt-based propagator in 1.6 On Nov 13, 2014, at 9:36 PM, chandon notifications@github.com wrote:
|
|
@chandon thanks for letting us know your 1.7.0 experience so far. Even though that is not yet a proof, it's already good news :-) |
|
@chandon: on 1.6.3 did you also see the same 16KB block issue? I have been running many tests on previous versions and never seen that. And last Friday I saw it quite many times. I thought it may have been related to 1.6.4 update. Here is the point: I do continous testing and put/get lterally thousands of files. And this issue is not easily reproducible. So I would really like to understand a root cause. @danimo, @dragotin: Could that be that Qt HTTP stuff is not multithreaded fully or some buffers are reused? I guess you added Qt-based propagator in 1.6 |
|
@moscicki You've seen the corruption for the uploads? |
|
Always uploads. From our setup I would rather exclude the owncloud server problem. Maybe proxy. Or timing issue on the client. kuba On Nov 13, 2014, at 10:20 PM, Markus Goetz notifications@github.com wrote:
|
|
I cannot reproduce the error right now consistently - fortunately, as I am
running OC in production environment and if each 3rd file got corrupted
alongthe way it would be disaster and we would have to stop using owncloud
immediately.
We had more common errors on OCserver 6.x, and clients 1.6.x, it somewhat
improved on OC7.x and clients 1.7.1 and now it more or less works with most
clients 1.8.0 - yet there are exceptions. Using Ubuntu server 14.04, Apache
with SSL, in OpenVZ container and linux kernel 2.6 stable from host OS
which is ScientificLinux6. Clients are on WinXP (1), WinVista32bit (1),
Win7 64bit (cca 5), Win8.1Pro32bit (1), OSX10.10 (1), LinuxMint17.1 64bit
Xfce (2). The last corruption occured from the less often used Linux
client, but earlier WXP, W7 and Mac clients randomly did the same.
I am afraid I cannot help developing some sophisticated automatic tests,
unless you supply me with some code, and in any case I cannot run it on the
production vps if it would make the corruption happen more often for the
real data:(
|
|
@moscicki The two test branches would get me that crash once the server served 3-10 requests. I don't know all the different ways of triggering the issue (guruz knows more) but that one case can't be reproduced without modifying the client unless we would patch and rebuild nginx to close the connection even though it responds with a "Connection: Keep-Alive" header. |
|
The other ways to reproduce the corruptions are in https://codereview.qt-project.org/#/c/110150/7/tests/auto/network/access/qnetworkreply/tst_qnetworkreply.cpp,unified |
|
I just experienced 3 corrupted PDFs out of 3 uploaded PDFs on Mac OS X 10.10 I immediately upgraded OC to 1.8.0 Qt 5.4.0 hopefully it will disappear immediately. I was shocked that just saving 3 PDFs from Thunderbird email to ownCloud folder causes all three to become corrupted during upload to server. Now there is 5 GB of data on that Mac that is perhaps OK but most of this data may be randomly or systematically corrupted on the server. How can I force ALL files in Mac to be reuploaded to the server correctly? |
|
It is even more bizzare that it is only PDFs that are corrupted. Next email saved to the same folder, with 6 DOC and 3 DOCX files, no problems with these files, all uploaded correctly. |
|
As far as I understand the fix comes in 1.8.1. To be confirmed with developers What you can do is the following:
https://github.com/cernbox/smashbox/blob/master/corruption_test/md5blocks To trigger the re-upload do: touch FILE |
|
I was too quick to reinstall the client so I am not sure if there is any logfile of the already uninstalled version of the client. I have original and corrupted files so: first one: total --- cacf4da14a76f62654368f76d39becdb 3ed3e9846b56c09d31934a1d27cb0a8d *** |
|
Yes, first 16K block lost. Others untouched (which is different from what we saw with our test data). But close enough. |
|
second file seems the same pattern ./md5blocks ./correct/BL2015_Clean\ Touch\ (Myci\ prostredek\ na\ lahve).pdf ./corrupted/BL2015_Clean\ Touch.pdf total --- 8f89939eca0acb976e50295b55ffe32e 671960d1c109566846b0a2e9c8b2226a *** |
|
and the third one the very same thing total --- ce7717511ddd205c62c7d4d15f3f96b9 0a2136e4270432388e85440b8604f0ce *** |
|
I do not want to speak for developers but the issue has been apparently fixed (and the bug was not in the owncloud client but in the Qt toolkit the client uses). It is simply amazing that Qt which is used worldwide so much has such bugs. |
|
Yes, @moscicki says it: We consider that fixed. The next upcoming version 1.8.1 will have the fix. Thanks for verifying that! |
|
Does it mean that the client 1.8.0 freshly downloaded from the website, using Qt 5.4.0 is still affected by the same bug? If it is, is there some workaround until 1.8.1 is released? Regarding triggering of reupload with touch, could that be safely performed on a Mac OS X with a command such as find . -exec touch {} + ? The user has several GBs of data there (own) plus GBs of shared data, some directories are read-only on owncloud. Won't changing the timestamps force OC client to reupload files the user does not rights to upload (therefore the owncloud client will show error for each of such file indefinitely as it happens now when user writes to owncloud-RO folder) Is there some script that could detect which files on the server, no matter what user owns them, are possibly corrupted, so we could find them, and try to find if possible the original file. not corrupted, on the owner's PC? Is there some specific data always on some position or is the corrupted part just random garbage? And if the Client 1.8.1 finally solves this critical issue on all plarforms, is there finally some setting on the server side to force all users to upgrade their clients by certain date to the fixed version? We have several external collaborators with their own devices, and I cannot manage their laptops remotely or by any other way. |
This will be possible with ownCloud 8.1 - ref owncloud/core#15683 |
|
You may try to use the md5blocks script with --check option with a single file argument. This does not cover cases of chunked file upload if corruption happened on any other chunk but the first one. I intend to actually extend this option to cover the chunked upload cases. You may estimate how this currently affects you by calculating the ratio of files below 10MB in your entire population of files. This is not 100% waterproof but should give you some idea (you will also probably need to exclude some false positives, such as a file full of zeros as a simplest example). |
|
This bug is tested with QNetworkReply autotest. |
Since QNonContiguousByteDeviceThreadForwardImpl::reset will call UploadDevice::reset with a BlockingQueuedConnection, this allows us to reset the HTTP channel along with its buffers before they get the chance to be reused with a subsequent request.
Expected behaviour
I create a file in a folder shared with Owncloud and it always upload a correct file (not corrupted)
Actual behaviour
When I create a file in a folder shared with Owncloud it sometimes uploads a corrupted file. The file has the same size as the original one, but is corrupted. If it can help, i've uploaded 2 corruption examples here:
https://www.wetransfer.com/downloads/499390103af226851e2ea96237d5487620141029092303/efdb625623a454be881d013a02c10e3e20141029092303/b2bcb7
The original files have been created in the owncloud folder on purpose to test the bug. The corrupted files have been downloaded directly from the server folder /var/www/owncloud/data/test/files
Possibly the same bug as #1969 ?
Steps to reproduce
It's a ramdom bug...
I've past massivly 50 files in my owncloud folder, then i did a md5sum on the server to find the corrupted files among all of them
Server configuration
Operating system: Ubuntu 14.04 LTS server
Web server: Apache 2.4.7-1
Database: MySQL 5.5.37-0ubuntu0.14.04.1
PHP version: 5.5.9-1ubuntu4.3
ownCloud version: 7.0.2(stable)
List of activated apps:
Default apps (i don't use them)
Client configuration
Client version: 1.6.4 build 1195
Operating system: Mac Osx Maverick (we are 2 having the same problem)
The text was updated successfully, but these errors were encountered: