Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External Storage Rescan / Reindex - S3 #17068

Closed
cloudguy23 opened this issue Sep 9, 2019 · 3 comments
Closed

External Storage Rescan / Reindex - S3 #17068

cloudguy23 opened this issue Sep 9, 2019 · 3 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug

Comments

@cloudguy23
Copy link

Steps to reproduce

  1. Install Nextcloud 16.0.4
  2. Add External storage (Ceph via S3 in this case)
  3. Add files via Nextcloud web portal
  4. Bulk data seed via rclone / s3cmd directly to S3 bucket

Expected behaviour

Files added via Nextcloud web portal (#3) and files added via rclone or s3cmd should be visible / accessible via Nextcloud.

Actual behaviour

Only files added via Nextcloud portal are visible / accessible via web. All files (imported as per #3 and #4 above) are visible when querying bucket directly via s3md ls or Cyberduck.

Attempting to run occ files:scan does not scan / re-generate Nextcloud indexes based on S3 bucket contents.

Server configuration

Operating system: CentOS 7.6

Web server: Apache 2.4

Database: MariaDB 5.6

PHP version: 7.2

Nextcloud version: (see Nextcloud admin page) 16.0.4

Updated from an older Nextcloud/ownCloud or fresh install: no

Where did you install Nextcloud from: tar.gz

Signing status:

Signing status
Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:

App list Enabled: - accessibility: 1.2.0 - activity: 2.9.1 - admin_audit: 1.6.0 - camerarawpreviews: 0.7.0 - cloud_federation_api: 0.2.0 - comments: 1.6.0 - dav: 1.9.2 - federatedfilesharing: 1.6.0 - federation: 1.6.0 - files: 1.11.0 - files_automatedtagging: 1.6.0 - files_external: 1.7.0 - files_pdfviewer: 1.5.0 - files_rightclick: 0.15.1 - files_sharing: 1.8.0 - files_texteditor: 2.8.0 - files_trashbin: 1.6.0 - files_versions: 1.9.0 - files_videoplayer: 1.5.0 - gallery: 18.3.0 - logreader: 2.1.0 - lookup_server_connector: 1.4.0 - nextcloud_announcements: 1.5.0 - notifications: 2.4.1 - oauth2: 1.4.2 - password_policy: 1.6.0 - privacy: 1.0.0 - provisioning_api: 1.6.0 - recommendations: 0.4.0 - serverinfo: 1.6.0 - sharebymail: 1.6.0 - support: 1.0.0 - systemtags: 1.6.0 - theming: 1.7.0 - twofactor_backupcodes: 1.5.0 - updatenotification: 1.6.0 - viewer: 1.1.0 - workflowengine: 1.6.0 Disabled: - encryption - firstrunwizard - survey_client - user_ldap

Nextcloud configuration:

Config report { "system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "16.0.4.1", "overwrite.cli.url": "http:\/\/localhost", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "activity_expire_days": 14, "auth.bruteforce.protection.enabled": false, "blacklisted_files": [ ".htaccess", "Thumbs.db", "thumbs.db" ], "cron_log": true, "enable_previews": true, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\Movie", "OC\\Preview\\PDF", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown" ], "filesystem_check_changes": 0, "filelocking.enabled": "true", "htaccess.RewriteBase": "\/", "integrity.check.disabled": false, "knowledgebaseenabled": false, "logtimezone": "Europe\/UTC", "logfile": "\/var\/log\/nextcloud.log", "log_rotate_size": 104857600, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\APCu", "preview_max_x": 1024, "preview_max_y": 768, "preview_max_scale_factor": 1, "quota_include_external_storage": false, "share_folder": "\/Shares", "skeletondirectory": "", "trashbin_retention_obligation": "auto, 7", "mail_smtpmode": "", "maintenance": false } }

Are you using external storage, if yes which one: local/smb/sftp/...

S3

Are you using encryption: yes/no
No. (nothing beyond TLS and encryption on object store backend)

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No.

LDAP configuration (delete this part if not used)

LDAP config
With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:

Operating system:

Logs

Web server error log

Web server error log
Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log
Insert your Nextcloud log here

Browser log

Browser log
Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...
@cloudguy23 cloudguy23 added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Sep 9, 2019
@cloudguy23
Copy link
Author

Issue resolved with #14913 .. This needs to be part of standard release.

FEATURE Request: Incorporate patch -- administrative enable / disable of behaviour.

@kesselb
Copy link
Contributor

kesselb commented Sep 10, 2019

If this issue is resolved by #14913 its a duplicate of #6954

@kesselb kesselb closed this as completed Sep 10, 2019
@sbw
Copy link

sbw commented Dec 22, 2019

After I updated my instance to 17.0.2, I browsed some of my S3 storage volumes. I must say, performance is MUCH improved. Thanks to the developers who worked on this and pushed it into the releases.

But at the same time, I note that there is still a very significant delay when opening an S3 volume, even if it was just open a moment before in the same session. I remain baffled why the "cached" information in the Nextcloud database is not displayed immediately, especially when raw performance is known to be relatively slow, as for object storage. I don't even expect the Nextcloud display to be updated dynamically, since object storage is often used for slowly-changing archives. I would be fine with refreshing the page to see changes recorded by Nextcloud after the directory is initially displayed.

Again, I recognize it's as much my responsibility as anyone else's to knuckle down and dig into the source code to understand what changes are needed to accomplish that. I hope I can get to that one day.

In any case, thanks for the BIG improvement!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug
Projects
None yet
Development

No branches or pull requests

3 participants