Instant upload sometimes creates corrupt files with 1.4.0 client and 5.0.6 server

[a-schild]

I did take 5 photos, the first 3 are uploaded incomplete on the server, the last 2 pictures are ok.

Here the correct files:
21.05.2013 13:18 3'185'208 DSC_0438.jpg
21.05.2013 13:18 3'359'122 DSC_0439.jpg
21.05.2013 13:18 2'039'503 DSC_0440.jpg
21.05.2013 13:19 2'142'841 DSC_0441.jpg
21.05.2013 13:19 2'090'055 DSC_0442.jpg

Here the files uploaded via instant upload from android:
21.05.2013 13:23 113'208 DSC_0438.jpg
21.05.2013 13:24 1'311'122 DSC_0439.jpg
21.05.2013 13:25 1'015'503 DSC_0440.jpg
21.05.2013 13:25 2'142'841 DSC_0441.jpg
21.05.2013 13:26 2'090'055 DSC_0442.jpg

What's strange is the fact that this are not incomplete uploads, but realy corrupted (+incomplete?) uploads
Attached the correct (left side) header of DSC_0439.jpg and the corrupt (right side) upload of it

When I comprae the file byte per byte, then I see that the corrupt upload of DSC_0438 starts with the bytes which are shown at offset 0x2EE000 in the correct file.
In file DSC_0439 they start at 0x1F4000
In file DSC_0440 they start at 0xFA000

Not sure if it's related to issues #42 or/and #91

[davivel]

Thanks for your so complete report. We'll have to check it.

Probably it's related to those other issues, so let's keep it tracked here.

[davivel]

Hi, @a-schild . Are you still suffering this problem in your environment? Did you try to update server and app? Thanks

[pgschk]

Same issue, but it does not only affect instant upload, but all uploads (or at least images, that's were I immediatley notice it).

owncloud 5.0.9 and Android app 1.4.4

Almost 50% of all uploads are corrupt. Please let me know if I can provide any debug info.

This is the Apache2 log of a corrupted upload:

82.176.xx.xx - user [11/Aug/2013:22:59:26 +0200] "HEAD /remote.php/webdav/SofortUpload/IMG_20130810_205957.jpg HTTP/1.1" 200 745 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:26 +0200] "HEAD /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg HTTP/1.1" 404 678 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:34 +0200] "GET /status.php HTTP/1.1" 200 729 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:35 +0200] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 4634 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:37 +0200] "PROPFIND /remote.php/webdav/Audio/ HTTP/1.1" 207 3109 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:39 +0200] "PROPFIND /remote.php/webdav/SofortUpload/ HTTP/1.1" 207 44161 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:27 +0200] "PUT /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg-chunking-1128-4-0 HTTP/1.1" 201 579 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:22:59:44 +0200] "PUT /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg-chunking-1128-4-1 HTTP/1.1" 201 712 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:23:00:02 +0200] "PUT /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg-chunking-1128-4-2 HTTP/1.1" 201 712 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:23:00:19 +0200] "PUT /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg-chunking-1128-4-3 HTTP/1.1" 201 735 "-" "Android-ownCloud"
82.176.xx.xx - user [11/Aug/2013:23:00:21 +0200] "PROPFIND /remote.php/webdav/SofortUpload/IMG_20130810_205957%20(2).jpg HTTP/1.1" 207 1162 "-" "Android-ownCloud"

Apache responds with 201 for all chunks, so it is understandable that the Android app reports a success.
I sometimes notice other pictures fragmented into these corrupt files. Could this possible be a temp file mixup at the server?

[davivel]

@pgschk , if that log is caught for a corrupted image, then the answer is yes, this probably is a problem in the server side with the composition of the full file.

Files with size over 1MB are broken in pieces ant PUT into the server piece by piece. Then the server side puts all the chunks together. The access log you show there reveals that all the pieces where uploaded without problem, and that the information about the final file was then sucessfully got too (207 in the next PROPFIND).

[pgschk]

Yes, it's from a corrupted upload. Funny enough, when trying to capture this log I had to try 6-7 times to get a corrupted upload ;-)

[enoch85]

Should somebody at the owncloud/core take a look at this maybe? schiesbn maybe?

[pgschk]

Since I am pretty sure this does not affect all environments, here is the server spec:

Ubuntu 12.04 AMD64 in a KVM virtual machine. It does not have any other job than owncloud. As far as I remember all software is installed from repositories and owncloud as .deb.

I'll gladly provide test accounts and help debugging, just let me know.

[enoch85]

I´ve just confirmed that the issue is in Webdav, or at least I think so.

Today when I copied a picture from my desktop in windows to my Webdav mount the picture was broken like the examples in #237 Therefor I think it has something to do with the server, and also maybe in a combination with the server.

[davivel]

And if you upgraded the server from previous versions, instead of doing clean installations, probably it has something to do with the problem too.

[SilvioSedi]

Same problem here. I have only image files that are affected by the problem.

greeting from Switzerland

[pgschk]

@davivel I am 99% sure the server was installed with 5.0.8 and had the problem, so I upgraded to 5.0.9 to check if that helped. So I am not sure that upgrading is part of the problem.

[enoch85]

It has been the same thing on oC 5.0.6 - 5.0.7 - 5.0.8 - 5.0.9 - 5.0.10.

I always upgrade, and never do clean installations.

I run my server on Ubuntu Server 12.04 and WebDAV through Windows 7.

[davivel]

@enoch85 , oC 5.0.6 was not a clean installation?

Sorry, but I do not understand what you mean with "my server on Ubunto Server... ** and WebDAV through Windows 7 **"

[enoch85]

@davivel No, I upgraded from oC 5.0.5 that was a clean installation.

[davivel]

And the problem was also in oC 5.0.5?

[enoch85]

I don't know as I didn't use the Android app in oC 5.0.5.

[pgschk]

Has there been any progress to this issue? :) Thanks!

[davivel]

Not yet, sorry :(

[rperezb]

Hi,

We have detected that this issues is solved with the owncloud server 6, can you update your server version and re-test it?

THX!

[pgschk]

Hey,

I just updated to 6.0Aplha1 and snapped a couple of pictures. All reached the server just fine (using Android App 1.4.4). I'll keep taking a couple throughout the day and will report back. Thank you!

[pgschk]

Now with Andoid App 1.4.5, so far no broken pictures, I think you might be right about this being solved

[masensio]

@a-schild.
Could you please update your server version to ownCloud Serve 6 and let us know if this solves your problem?

Thanks in advance.

[davivel]

Hi everybody here.

Could somebody else check if the problem is fixed with a recent OC server? (>= 5.0.13).

Thanks in advance.

[pgschk]

I did not see the issue since upgrading to version 6.0Aplha1, so I consider it fixed. :)

[davivel]

Hi, @pgschk. Thanks a lot for your response.

@a-schild , @enoch85 , would you like to check also?

[enoch85]

I have not seen it. I have 6.0a.

MVH Daniel

Skickat från Google Nexus 5

[a-schild]

Since the 1.4 and 1.5.0/1.5.1 clients had problems with large file collections I had disabled it in the last months.
With 1.5.2 I did now just reenable it on my phone, I will look what it is doing. (Server at 5.0.13)

[enoch85]

I haven´t experienced this with client 1.5.5 and 6.0.2 server.

I think the problem is solved. Please correct me if i´m wrong.

[a-schild]

Looks also OK for me with 1.5.5 client and 6.0 server

[enoch85]

Close this one then?

[davivel]

Yeeeees. Thanks a lot!