Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 multipart upload #1173

Merged
merged 13 commits into from
Aug 28, 2023
Merged

S3 multipart upload #1173

merged 13 commits into from
Aug 28, 2023

Conversation

ZetaTom
Copy link
Contributor

@ZetaTom ZetaTom commented Aug 3, 2023

This pull request adds support for chunked uploads to s3 via s3 multipart upload, as requested in nextcloud/files-clients#28. The server-side implementation has been merged with nextcloud/server#27034, and the corresponding documentation can be found here: nextcloud/documentation#10740.

TODO

  • discuss with Julius how to properly test with CI

closes nextcloud/android#11745

@codecov
Copy link

codecov bot commented Aug 3, 2023

Codecov Report

Merging #1173 (ee3d96d) into master (3c81a28) will increase coverage by 0.19%.
Report is 12 commits behind head on master.
The diff coverage is 86.04%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1173      +/-   ##
============================================
+ Coverage     51.90%   52.09%   +0.19%     
+ Complexity      968      966       -2     
============================================
  Files           187      187              
  Lines          6922     6906      -16     
  Branches        914      911       -3     
============================================
+ Hits           3593     3598       +5     
+ Misses         2819     2798      -21     
  Partials        510      510              
Files Changed Coverage Δ
...ources/files/ChunkedFileUploadRemoteOperation.java 69.84% <85.36%> (+4.16%) ⬆️
...ncloud/android/lib/common/network/WebdavUtils.java 87.70% <100.00%> (+0.10%) ⬆️
.../com/owncloud/android/lib/resources/files/Chunk.kt 100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

@ZetaTom ZetaTom force-pushed the s3-multipart-upload branch from 850b309 to 284c03f Compare August 9, 2023 15:27
ZetaTom and others added 11 commits August 9, 2023 17:34
Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
- refactor to use new Chunk class
- remove incompatible test cases

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
- test on wifi and mobile

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
- fix calculation of chunk size
- increase CHUNK_SIZE_MOBILE to 10M
  - server enforced minimum of > 5M
- increase CHUNK_SIZE_WIFI to 40M
- reformat file

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
- ported java class to kotlin
- added test cases for calculation of chunks
  - no chunks available on server
  - some chunks available
  - switch chunkSize during upload

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
- increase size of BIG_FILE_ITERATION
  - previously about 14 MB, now about 28 MB

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
Signed-off-by: nextcloud-android-bot <android@nextcloud.com>
Signed-off-by: tobiasKaminsky <tobias@kaminsky.me>
Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
Signed-off-by: nextcloud-android-bot <android@nextcloud.com>
- chunks in S3 may loose padding
- move Chunk.java to data class

Signed-off-by: ZetaTom <70907959+ZetaTom@users.noreply.github.com>
Signed-off-by: tobiasKaminsky <tobias@kaminsky.me>
@nextcloud nextcloud deleted a comment from github-actions bot Aug 28, 2023
@nextcloud nextcloud deleted a comment from github-actions bot Aug 28, 2023
@nextcloud nextcloud deleted a comment from github-actions bot Aug 28, 2023
@nextcloud nextcloud deleted a comment from github-actions bot Aug 28, 2023
@nextcloud nextcloud deleted a comment from github-actions bot Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use MultipartUpload for uploading chunks to s3
4 participants