"hasUpdated" function is not being called from the OC root folder for mounted FSs

[jvillafanez]

Expected behaviour

When you click in the OC root folder (/index.php/apps/files) the hasUpdated function is called for each mounted storage inside the root folder so that we can know that there are changes inside the mounted storage.

Actual behaviour

That function is not being called for the mounted storage (just for local storage) so other mounted storages don't report a change until we enter in them.

Steps to reproduce

1. Mount an external FS under the OC root. You'll access to the OC root in the url "/index.php/apps/files" and to the external FS in "/index.php/apps/files?dir=//out" (change the directory name if needed)
2. Add a new file to that external FS outside OC (don't use OC to upload the file)
3. Go to the OC root dir "/index.php/apps/files"
4. Check that the mtime of the "out" folder hasn't changed.
5. Go inside the "out" folder and back and check that the mtime is now the correct one

Server configuration

Web server: apache
Database: sqlite3
PHP version: 5.3.10
ownCloud version: 5.0.11RC2

Client configuration

Browser: chromium and firefox

[DeepDiver1975]

@icewind1991 any idea? Can you please have a look? THX

[icewind1991]

This is intended behavior, hasUpdated is only called for a folder if you enter it, this is done for performance reasons, no need to make expensive calls to external file systems if we aren't browsing the filesystem

[DeepDiver1975]

Okay - but this will result in not up to date information like mtime

[jvillafanez]

I had some issues with the desktop client regarding this, because I think they only check the mtime of the OC root folder to trigger the synchronization. Other clients might present the same issue.

Using only the desktop client, you can upload a file as described but you'll never get it in the desktop. You'll need to use the web interface and/or other clients.

[DeepDiver1975]

this is most probably related to etag propagation - this requires some more testing

[karlitschek]

@PVince81 What do you think?

[karlitschek]

@icewind1991 Can you look into this?

[karlitschek]

@icewind1991 What do you think?

[icewind1991]

Calling hasUpdated on every storage in a folder would give a to large performance hit.

Having a background job call hashUpdated or scan on external storages regulary would be a much better solution.
Having the mtime in the webui be off for a little while is not a big enough issue to warrant for the performance hit fixing it would give.

[DeepDiver1975]

The performance issue are understandable and valid if we talk about real physical external storage (Like amazon etc.) but does this apply to external storages which live in the same network/server?

On bigger installations even the user's root can already be mounted to e.g. a lokal SWIFT/OpenStack storage.
In such a setup desktop sync and web ui is expected to work as if it would be local disk space on the server.
(An yes I know it technically cannot have the same performance 😉 )

[icewind1991]

On bigger installations even the user's root can already be mounted to e.g. a lokal SWIFT/OpenStack storage.

That isn't as much of a problem because we only need to check one storage, if we were to check all storages in a folder there is no real limit to the amount of storages we need to check.

Also there is a big difference in the folder you're currently browsing being out of date, or some folders in the webui having an out-dated mtime.
I just don't think it's worth having any performance hit to solve the latter

[PVince81]

Not sure if related to this, but I found out that whenever I edit a file on a SMB storage, the change isn't picked up by the sync client. Every call from the sync client asks for changes in "files" which aren't there since the etag isn't affected by changed inside mountpoints.
This also means that conflict detection doesn't work: whichever file is edited locally is uploaded by the sync client and will overwrite any changes in the SMB file, regardless whether that one was changed remotely or not.

We need to find a good solution for this, as this makes syncing incompatible with external storages, which partly defeats the purpose of having external storages in the first place.

[PVince81]

(unless one doesn't use the sync client)

[PVince81]

The problem also exists if you modify files directly in the "data" dir, but inside a subdir.
The changes made there aren't detected until the next time the user navigates into that dir in the UI, which forces a rescan and also magically make the sync client find them afterwards.

It does seem that detecting external changes is difficult, especially when the change isn't directly in the root of the storage.

[PVince81]

One possible idea: for every ext storage mountpoint, return a special flag to the sync client. For example: "X-OC-FolderType: Mountpoint"

This way the client knows that it's an ext storage mountpoint and it could then do a PROPFIND on that one if necessary, which should trigger the "hasUpdated" call.

[PVince81]

Or alternatively the mount points could be checked either:

1. at a specific time interval (ex: every hour)
   or
2. on-demand but no more than once per time interval (ex: max once per 15min)
   or
3. check it every X accesses (suggested by @schiesbn)

What do you think @icewind1991 ?

[icewind1991]

Just checking the mount point wont fix the issue of changes not being detected

[PVince81]

So it seems a full rescan is needed in most cases, which is expensive. 😦

[PVince81]

This issue was reported by many different users here: #7955
We need to find a solution for update detection.

[PVince81]

@icewind1991 would a cron job that regularily runs the scan CLI command fix this ?

[timreeves]

Thanks to @PVince81, who pointed me here from another thread. I've been struggling at setting up ownCloud (read) access to my nightly backups, to save logging in via SSH to check/get them. In this struggle I noticed that external local storage can easily be mounted successfully, but often does NOT notice changes to files in such directories. I use a rotational backup principle which overwrites files in the same target directories. So the directory itself seems unchanged, the files are the same in name and number, only their size and mtime have changed. ownCloud Server UI definitely does NOT notice these changes - it tells me that files are say 1 month old, when in SSH I can see that they are from yesterday.

I have found a workable solution by having these mounts visible to a "Backups" user and a cron job to do a files:scan for the "Backups" user daily at 5am. So it now works for me, but was difficult to understand and not the easiest thing to design a working strategy with the current features. Would it not be easy enough to simply (re)scan the local external directory when it is entered at the UI? Or any other type of external storage which allows non-costly directory scans?

Many thanks to all those who have contributed to ownCloud - a great product getting better!

[PVince81]

@timreeves I think the current code already does a shallow scan whenever the user navigates into a directory. Shallow scan means that it will ask the ext storage whether the folder's mtime has changed. If yes, it will go into that dir and update the cache with the changed files.

One problem though is that if a file is modified but not renamed, the folder mtime will not change, so the scanner won't pick up the changes. It might be possible to improve this by force-scanning not only the folder but also its contents. Still, since mtime isn't propagated up to the root by most filesystems, the shallow scanner cannot find out about deeper changes by just scanning the root folder.

Needs further research.

@icewind1991 told me recently that there is a notification API. Not sure how it works. But it makes it possible for external apps to trigger scanning on specific folders. If SMB or other ext storage have a mechanism to send notifications on changes (like inotify on Linux FS), it could be used to trigger the scanner on specific folders.
Might be worth looking into this as well.

[PVince81]

@timreeves and what I forgot to mention, the sync client only checks for changes on the root (etag change), so if deeper change notifications cannot be propagated to the root, the sync client won't know either.

[timreeves]

@PVince81 Ha thanks for that good tip about the shallow scan, which I have now tested and it seems you are right - all I needed to add to my backup script was a "touch" of all the target directories involved!
Still, there are a couple of caveats:

• I "touched" the uppermost directory, but ownCloud browser UI still shows it as "13 hours ago" - but now it gets the directories and files IN that directory right (i.e. showing the real last mtime). So it's a little bit confusing from a user point of view - the modifed mtime triggers a shallow scan of the CONTENTS of that directory, but seems not to update the noted time for the directory ITSELF.
• A major part of my Backups come from other servers via scp, where I have no simple option to do a "touch" to the directories on my local host. I could duplicate the algorithm which directories are written and "touch" them with a nightly cron-job on my local host - but that's getting rather "unnatural" - so it would still in fact be very helpful if the modified CONTENTS of local external storage directories could be noticed when stepping into them.

[PVince81]

@icewind1991 @DeepDiver1975 if we introduce external storage options as per #12216 we could add an option "do a full rescan of this storage on a regular interval" ? The interval should probably be configurable so admins could decide how important/timely the change detection needs to be.

Then the full rescan would be done in a background job (cron).

[PVince81]

This would at least solve the potential performance issue mentionned here #5036 (comment) in case of multiple storages. Assuming that not all ext storages are important.

Additionally this means that the option is only needed for storages that are used outside of ownCloud. If some storages are using exclusively through ownCloud, they wouldn't need that option and wouldn't cause any performance issue.

[timm2k]

@PVince81 thanks for your reply. I can't tell 1000+ users to use selective sync to disable their whatever-named external storage. I need a centrally managed workaround.

[PVince81]

@icewind1991 how about a new option for "occ files:scan" to scan a specific storage id ? (like an external storage mount point). Then admins could put this in a cron job as a workaround to force rescan every 10-15 minutes.

[PVince81]

If I remember well this issue here was mostly about calling "hasUpdated()". So at least the CLI tool (or whichever solution we use) could at least run hasUpdated() on every known external storage roots.
Whether to force a full scan could be an additional option.

[theroch]

Workaround for local SMB shares from #8892:
The workaround is to run the command line tool ./occ files:scan --all in a cron job to force rescanning / checking for updates on the remote server.

[PVince81]

Well thing is that the scanner will rescan all home folders, not only the remote ones.
Ideal would be at least to have what I suggested here: #5036 (comment)

[khlem]

this method : ./occ files:scan --all does not function when the share is mounted with SMB / CIFS.
The files modification are not update in Owncloud Client, we must add a new file or navigate in Website to force the update of Client.

[directict]

We only use SMB for WEB access outside company + sharing with ppl outside our company, having it synchronised is not possible casue its never up2date on your laptop because changes are not seen by the client.

Just wondering, how does the enterprise version do this? (in case of SMB?)

[khlem]

"Just wondering, how does the enterprise version do this? (in case of SMB?)"

Same problem between enterprise and community version :( .

[directict]

OK.. i thought in the enterprise edition it was not an issue... i meant the Windows Shares plugin in the enterprise, does this function have similar issues?

[khlem]

Nop i tested this version and the same error appears.

I think, the core is the same between the community and enterprise edition.

[directict]

Ok, i thought those were 2 different plugins:

https://owncloud.com/features/#WindowsNetworkDrive

where the "SMB/CIFS" feature of the Community Edition is available here:
https://doc.owncloud.org/server/8.0/admin_manual/configuration_files/external_storage_configuration_gui.html#smb-cifs

[PVince81]

They are different plugins, but detecting remote updates is a general problem for every external storages, nothing specific to the plugin implementations, more something in core.

[directict]

ok that makes sense thanks, in 8.1 there will be improvement right?

[PVince81]

8.1 will contain some new switches for external storages to tell how often the rescan needs to be done (never, detect changes (default), on every access).

But I'm not sure whether this will fix update detection when someone changed a file very deep in the folder structure of an external storage (from outside ownCloud).

@icewind1991 will the "Always" mode rescan the whole external storage regardless of update detection ?

[directict]

Yes, i read everywhere that the SCAN function for SMB will not fix it indeed. its good to see you are working on this.. i think loads of people use ownCloud for SMB access.. atleast, thats what we like the most!

[icewind1991]

8.1 will not fix this issue

[khlem]

This is not a good news.

When do you think this issue will be resolved ?

Same issue in enterprise version, we can't use the system with SMB/CIFS and Owncloud Client.

Strange problem like this in a cloud system.

Thank you for your efforts.

Regards,

Clément.

[PVince81]

The main ticket about update detection #11797 has been scheduled for 8.2, so there is hope.

[RobinMcCorkell]

We have a ticket about update detection -> close