Minio Issue with S3 external storage module

[dragon2611]

Not sure if this quite the right repo so apologies if it needs to be in one of the others although I couldn't see one that looked like it would be more appropriate.

It used to the case that It was possible to use a Minio server with the external storage module but it looks like since an update to Minio nextcloud can now only see files in the root of the bucket.

I already raised it with the minio project, but I thought I'd also raise it here incase there's something that can be done from the nextcloud end,

I know there's not really folders in S3 but most tools (including nextcloud) still visualise the structure as files/folders

minio/minio#5916

Steps to reproduce

1. Install a current release of Minio
2. Create a directory and upload a file into it
3. Upload a file into the root directory
4. Setup the Minio server as external storage in nextcloud

Expected behaviour

I would expect to see everything within the bucket that nextcloud has permissions to see

Actual behaviour

Files in the root of the bucket are visble, everything else isn't

Server configuration

Ubuntu 18.04,

Apache 2.4.29

MariaDB 10.1.29-MariaDB-6

PHP 7.2.5-0ubuntu0.18.04.1

13.02

Fresh install (I think, but it might have been upgraded from 13.01)

Nextcloud site

Signing status:
No errors have been found.

List of activated apps:

App list

Enabled:

• activity: 2.6.1
• bruteforcesettings: 1.0.3
• comments: 1.3.0
• dav: 1.4.6
• federatedfilesharing: 1.3.1
• federation: 1.3.0
• files: 1.8.0
• files_external: 1.4.1
• files_pdfviewer: 1.2.1
• files_sharing: 1.5.0
• files_texteditor: 2.5.1
• files_trashbin: 1.3.0
• files_versions: 1.6.0
• files_videoplayer: 1.2.0
• firstrunwizard: 2.2.1
• logreader: 2.0.0
• lookup_server_connector: 1.1.0
• nextcloud_announcements: 1.2.0
• notifications: 2.1.2
• oauth2: 1.1.0
• password_policy: 1.3.0
• provisioning_api: 1.3.0
• serverinfo: 1.3.0
• sharebymail: 1.3.0
• spreed: 3.2.2
• survey_client: 1.1.0
• systemtags: 1.3.0
• theming: 1.4.5
• twofactor_backupcodes: 1.2.3
• updatenotification: 1.3.0
• workflowengine: 1.3.0
  Disabled:
• admin_audit
• encryption
• gallery
• user_external
• user_ldap

Are you using external storage, if yes which one:
as default no
primary use has a folder that is s3 via minio configured as "external storage"

Are you using encryption:
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

LDAP configuration (delete this part if not used)

Client configuration

Browser:
Multiple
Operating system:
Multiple

[amarburg]

I can corroborate this problem (or a version at least)
Nextcloud: Docker nextcloud:latest image == 13.0.4-apache
Minio: Docker minio/minio:2018-06-09T03:43:35Z image

Minio is added to Nextcloud as an S3 external source -- not as primary storage -- connecting to an existing bucket with an existing directory tree of data.

Minio is serving a disk from a local computer, so I can access the underlying file structure multiple ways (at console, through Minio, and through Nextcloud).

As we start, within the bucket:

Minio sees:

2011/10/data.csv

Nextcloud sees:

(no files in bucket)

If I make a directory "test" using Nextcloud:

Minio sees:

2011/10/data.csv
test/

Nextcloud sees:

test/

If I make a directory "test" within "test" using Nextcloud:

Minio sees:

2011/10/data.csv
test/test/

Nextcloud sees:

(no files in bucket)

[ then destroy test/ and test/test/ directory ]

If I then make a directory "test/", Nextcloud can see the folder, as before.

If I then upload a file "test/foo.txt" , Nextcloud can no longer see the folder "test/" -- but the file does exist on the filesystem and in minio

If I then remove test/foo.txt (using Minio), Nextcloud can now see the folder "test/" again.

At a high level, it appears the Nextcloud "no longer sees" top-level folders once they have content (a subdir or a file). We can't test if this is true for subdirs because we cannot create subdirs w/o losing the top-level files.

[ThisIsQasim]

Similar issue here. I am unable to create or move folders to external storage (Minio S3). I can, however, create, upload and move files. That means I can put files to the root of external storage and not make any folders. Even the folders that do already exist are not visible in Nextcloud.

[huihuimoe]

Similar issue here.

There are lots of logs like Debug | OC\Files\Cache\Scanner | !!! Path 'test' is not accessible or present !!! .

By looking for the minio's access log, I found nextcloud server try to list dir on server(?). But minio returns 404.

logs here.

54.64.***.* - - [28/Oct/2018:04:17:14 +0100] "HEAD /test HTTP/1.1" 404 0 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.2.1 curl/7.61.1 PHP/7.2.10"
54.64.***.* - - [28/Oct/2018:04:17:14 +0100] "HEAD /test/ HTTP/1.1" 404 0 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.2.1 curl/7.61.1 PHP/7.2.10"

Probably that this is caused by minio/minio#4434

but won't fix 😢

[kesselb]

Ref #6954

[arno01]

@danielkesselberg I've tried the patched file https://github.com/nextcloud/server/blob/759714eadec507dfa7893a8c05632c5262117025/apps/files_external/lib/Lib/Storage/AmazonS3.php but it didn't make the top-level directories appear when using Minio :/

[kesselb]

I were also running into smaller issues with directories in directories.

Could you try this one? https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (patch file) #14913 (patch file)

[machtech]

I can confirm that we have manually applied kesselb scan-external-s3 patch and it has solved this issue. We are running Nextcloud version 15.0.4

Please consider accepting patch into upstream so arrives in next Stable version, thank you.

Paul

[nagdevAmruthnath]

The following #9808 (comment) gives error
error: unrecognized input

Reproducable

>root@nextcloud:/var/www/html# curl -L  https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3.patch -o /tmp/scan-external-s3.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  273k    0  273k    0     0   507k      0 --:--:-- --:--:-- --:--:--  507k

>root@nextcloud:/var/www/html# git apply /tmp/scan-external-s3.patch 
error: unrecognized input