Use MultipartUpload for uploading chunks to s3

[tobiasKaminsky]

• Android: Use MultipartUpload for uploading chunks to s3 android#11745
• Desktop: Use MultipartUpload for uploading chunks to s3 desktop#5819
• iOS: Use MultipartUpload for uploading chunks to s3 ios#2471
• Server: Use MultipartUpload for uploading chunks to s3 server#27034

I will clarify client's side changes and then post more info.

[tobiasKaminsky]

• based on chunking api
• @juliushaertl checks
  • if/how existing chunks can be listed
  • if chunk size is allowed to be variable
• API for clients
  • Header X-Chunking-Destination: files/user1/path/to/file.bin on each request of the chunked upload process (MKCOL/PUT/MOVE) (entire dav path)
  • same endpoint as chunkedUpload is used
  • name of chunks must be numerical
  • chunk count is <10000
  • old chunking is fallback
    --> streamline old and new chunking system, always send header

To test:

• @juliushaertl checks if he can provide a docker

Clarify if this is a benefit for iOS at all

[juliusknorr]

@tobiasKaminsky Sorry totally forgot to update you on the open points:

• if/how existing chunks can be listed
  • Works now with the latest changes
• if chunk size is allowed to be variable
  • This is possible as long as they are all larger than 5 MB (except the last chunk)

Regarding testing this if it is actually about a dev only setup you may use the minio play instance and just need to add this section to your config.php before installing Nextcloud, but files would then be uploaded to their public test instance:

Ref https://docs.min.io/minio/baremetal/console/minio-console.html

'objectstore' =>
        array (
            'class' => 'OC\\Files\\ObjectStore\\S3',
            'arguments' =>
            array (
                'bucket' => 'your-custom-bucket-name-maybe-random',
                'key' => 'Q3AM3UQ867SPQQA43P2F',
                'secret' => 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
                'hostname' => 'play.min.io',
                'port' => '9443',
                'use_ssl' => true,
                'use_path_style' => true,
                'autocreate' => true,
                'verify_bucket_exists' => true,
            ),
        )

[juliusknorr]

Further updates:

• Header name changed to X-S3-Multipart-Destination
• Capability is now called s3-multipart in the dav section

Full reference at nextcloud/server#27034

[tobiasKaminsky]

5mb as minimum for every chunks, except last one.

[schiessle]

@tobiasKaminsky @juliushaertl sorry for digging out this old ticket. Does this also work if S3 is used as external storage or only with s3 primary storage?

[juliusknorr]

Only primary storage.