Health check of an instance

[MorrisJobke]

@butonic and I had thought about this many times: It would be nice to have a health check for an instance. This could check for stuff like:

• weird entries in the DB
• duplicate entries
• zombie entries - with no reference to others
• not connected subtrees in the filecache
• duplicate share entries - see remove duplicate shares before mounting them #18835 for code to detect this
• ...

This is maybe something for a cleanup step or just for info to start investigations.

I started this ticket to collect possible candidates of weird entries and to be able to check other existing instances for the same symptoms.

cc @karlitschek @PVince81 @nickvergessen @schiesbn @icewind1991 @rullzer @Xenopathic Opinions on this?

[oparoz]

• wrong media types
• broken thumbnails
• files which think they're folders

[schiessle]

Great idea. Some more ideas:

• encrypted files where "encrypted" is set to '0' at the file cache
• unencrypted files where "encrypted" is set to '1' at the file cache

[butonic]

Some of the checks do take a while. For the implementation I would recommend a section in the admin settings that has buttons to trigger individual checks as well as a check all button.
It should be possible to get the SQL that is executed to check for problems as well as SQL that might have been generated to clean up inconsistencies before executing it. Thet won;t be possible for all checks but anyway.

Also checks for:

• files for not existing storages
• share entries for not existing files
• home storages starting with local::
• incorrectly formatted etags (should not start or end with double quotes)

[karlitschek]

Very good idea. Should this be part of the repair script? I guess some of the issues are repairable and others not?

[MorrisJobke]

Very good idea. Should this be part of the repair script? I guess some of the issues are repairable and others not?

This is more for non-repairable stuff. If it is possible to repair we can do this instead of showing it, but this should primarily help to detect problems that not yet occured, but could be bad.

[karlitschek]

true. but maybe it should be added to the same occ command. a user does't know if something can be repaired or not.

[tflidd]

• remove old tables (i.e. from old apps that are not used any more)

[MorrisJobke]

remove old tables (i.e. from old apps that are not used any more)

There is already a repair step for this. Any specific tables, then we can add them.

[nickvergessen]

@butonic

share entries for not existing files

This is done from 8.1 or 8.2 in onwards with a cron job

[rullzer]

Yes this would be great. But! We must be careful here. Automatic repairs need to be very very well analyzed before they are done! And in some cases it might be best just to advice people to post an issue here if the step reports something is broken.

I would suggest to make sure this can only be run for the CLI. Timeouts are dangerous when handling potentially complex/long running tasks.

[butonic]

Agreed. Let us start with a health check first. Keep repair steps in occ.

[MorrisJobke]

• detecting filecache entries that has a parent in a different storage -> we have seen this, but can't find the reason

[PVince81]

• detect whether all indices from db_structure.xml do exist. I have the feeling that in some situations people somehow managed to skip index creation, see upgrade 8.1.1-3 -> 8.1.3-13.1 'CREATE UNIQUE INDEX' failed #19142 (comment)

[oparoz]

• Find a way to detect id oc_file_locks is full of rubbish by comparing the list of cached files to the actual list of files

[nickvergessen]

@oparoz that is more of a repair step? 😉

[oparoz]

@nickvergessen It's both. First you need to know that something is completely wrong.

1. As a user, you get all the weird messages about files being locked, so you suspect something is wrong
2. You notify the admin
3. As an admin, you test the health of you system
4. You see that something is wrong
5. You run the repair step

No?

[nickvergessen]

Well repair steps dont need "first you need to know", you just run them and they fix it.
The health check is for cases where we can't fix stuff automatically.

[MorrisJobke]

• LDAP entries that are created before a change was done in the LDAP settings (i.e username attribute, homefolder naming rule, ...)

[PVince81]

• child share entries without parent (repair step raised here: Repair step: delete shares where the parent doesn't exist any more #20130)

[MorrisJobke]

child share entries without parent (repair step raised here: #20130)

Nice one :)

[MorrisJobke]

• unique share tokens Sharing tokens should be unique #20741

[MorrisJobke]

• duplicate tags per user Duplicated entries in the oc_vcategory table cause favourite tags to not be permanently deleted #20952

[nickvergessen]

duplicate tags per user #20952

That should not be a health check, it needs a fix in the db layer and a pre-update script to fix it.

[MorrisJobke]

That should not be a health check, it needs a fix in the db layer and a pre-update script to fix it.

Correct - but we should also find out the reason. We can drop it from here if it is not needed anymore - just collecting stuff.

[cdamken]

@bboule This is related to https://github.com/owncloud/enterprise/issues/832#issuecomment-143771260 Shouldn't it have the same milestone?

[nickvergessen]

@cdamken this is an overview ticket with things that could be implemented in little steps...

[bboule]

I think we need to loop @MTRichards into this as well as it seems product related

[MorrisJobke]

I have some alpha grade code ... needs some love: https://github.com/owncloud/serverhealth and has already first tests.

[PVince81]

• repair filecache parents + path_hash if inconsistent: Repair filecache parents + path_hash #22866 add a repair script for path_hash mismatching md5(path)  #10705

[PVince81]

• detect unmigrated legacy storages, in case the warnings were ignored...

[PVince81]

@pmaier1

[PVince81]

repair parent-child relationships (non-matching path or storage): #28253

[PVince81]

• repair mime type of non-folders that do have children

@jvillafanez FYI ^

[PVince81]

• delete stray file cache entries that have no matching files and are not accessible through any parents. Normally entries accessible through parents are already cleared by occ files:scan --all, but inaccessible ones aren't and need a different approach.

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[PVince81]

• deal with orphaned storages (see Maintenance tool to deal with orphaned storages #23364)

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)