Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) #29209

pep29 · 2017-10-11T13:52:56Z

Steps to reproduce

Dowload and unzip owncloud 10.0.3
Copy data and config from owncloud 9.1.6 to ownlouc 10.0.3
Upgrade as explained here https://doc.owncloud.org/server/latest/admin_manual/maintenance/upgrade.html

Expected behaviour

should upgrade smoothless

Actual behaviour

It all went well, but then i am stuck at the moving avatars step (keep showing 0%)

Server configuration

Operating system: opensuse 13.1

Web server: apache2

Database: mysql 5.6.12

PHP version: 5.6.9 (cli)

ownCloud version: 10.0.3.3

Updated from an older ownCloud or fresh install: 9.1.6.2

Where did you install ownCloud from: /srv/www/owncloud

Signing status (ownCloud 9.0 and above):

List of activated apps:

comments: 0.3.0
- configreport: 0.1.1
- dav: 0.3.0
- federatedfilesharing: 0.3.1
- federation: 0.1.0
- files: 1.5.1
- files_external: 0.7.1
- files_trashbin: 0.9.1
- files_versions: 1.3.0
- files_videoplayer: 0.9.8
- firstrunwizard: 1.1
- market: 0.2.2
- notifications: 0.3.1
- provisioning_api: 0.5.0
- systemtags: 0.3.0
- updatenotification: 0.2.1
  Disabled:
- encryption
- external
- files_antivirus
- files_sharing
- templateeditor
- theme-example
- user_external

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

ghost · 2017-10-11T15:02:58Z

So closing as a duplicate of #28763 which you have already found.

PVince81 · 2017-10-12T08:31:43Z

Reopening as there are no federated shares involved, it seems to be related to broken local shares.

PVince81 · 2017-10-12T08:33:48Z

@pep29 please pick one of the shares that are known to be broken and do a select * from oc_share where file_target like '%NAME%' and replace "NAME" with the name of the share as seen duplicated. So if you see "test" and "test (2)" then replace "NAME" with "test" so we can see both in the results.

pep29 · 2017-10-12T08:41:33Z

First i will tell you more about the duplicate case:

A creates a directory xxx and share it with B. So B sees the directory xxx in his shared folder, but due to unknow issues to us (maybe during the synchronisation with owncloud client), a copy of xxx is created in the local shared directory of B, at the end B got xxx and xxx(2), only xxx(2) is shows as a shared folder while xxx is not

PVince81 · 2017-10-12T08:43:46Z

@pep29 is only this one user affected or is there more ?

pep29 · 2017-10-12T08:44:24Z

@PVince81 There are more and most of them use owncloud client on Mac for the sync

PVince81 · 2017-10-12T08:48:56Z

@pep29 please run the SQL queries listed in the first post in #28253 and see if any of the affected users appear in the list.

pep29 · 2017-10-12T08:50:48Z

@PVince81 you mean these one:

select fc.storage,fc.fileid,fc.parent as "wrongparent",fc3.fileid "correctparent",fc.path,fc.etag from oc_filecache fc, oc_filecache fc3 where fc.parent <> -1 and fc.parent != fc3.fileid and fc3.storage=fc.storage and fc3.path = substring(fc.path, 1, length(fc.path) - length(substring_index(fc.path, '/', -1)) - 1) and not exists (select 1 from oc_filecache fc2 where fc2.fileid = fc.parent) order by path;
SELECT fc.`storage`, fc.`fileid`, fc.`path` as `path`, fc.`name`, fcp.`storage` as `parentstorage`, fcp.`path` as `parentpath`     FROM `oc_filecache` fc, `oc_filecache` fcp     WHERE (fc.`parent` = fcp.`fileid`)     AND (         (CONCAT(fcp.`path`, '/', fc.`name`) <> fc.`path`)         OR (fc.`storage` <> fcp.`storage`)     )     AND (fcp.`path` <> '')     AND (fc.`fileid` <> fcp.`fileid`);
select fc.storage,fc.fileid,fc.parent as "wrongparent",fc.path,fc.etag from oc_filecache fc where fc.parent <> -1 and not exists (select 1 from oc_filecache fc2 where fc2.fileid = fc.parent) order by path;
select * from oc_filecache where parent=fileid;

pep29 · 2017-10-12T09:02:50Z

@PVince81 Query: 1,3,4 return empty.

However query 2
SELECT fc.storage, fc.fileid, fc.pathaspath, fc.name, fcp.storageasparentstorage, fcp.pathasparentpath FROMoc_filecachefc,oc_filecache fcp WHERE (fc.parent = fcp.fileid) AND ( (CONCAT(fcp.path, '/', fc.name) <> fc.path) OR (fc.storage <> fcp.storage) ) AND (fcp.path <> '') AND (fc.fileid <> fcp.fileid);

return some sets, the files with whom we have been having issues as well are show, these folder, files are wrongly coded (we suspected the user created it on their mac and uploaded it using the owncloud client. The special characters on the mac have a different code)

e.g.: 151123_Liste_tats�?chlich_anwesende_TN_JT 16.11.15.pdf
but it should be 151123_Liste_tatsächlich_anwesende_TN_JT 16.11.15.pdf

However not all files with those special char are affected, that why we suppected it must be the way mac works

PVince81 · 2017-10-13T07:05:07Z

@pep29

About encoding:

If using the desktop client, uploading files that are encoded with NFD will be normalized to NFC, so I don't see how it could happen that you have these encoding issues. The weird chars "�?" look more like they are in ISO format, not NFD. NFD umlaut looks like this: #21365 (comment)

Now I have no idea why some files would be ISO, unless maybe your server/apache/PHP was not configured properly to use en_US.UTF-8.

That said, I'm not sure whether the encoding is causing the upgrade issue.

About the SQL result:

Since you got results it means there is likely some broken filecache entries that need repairing. The code for this is currently on the stable10 branch and not released yet: #29074. The problem I see here is that you're stuck at the update process, which means the repair should already be done on the previous version, 9.1, which is currently still in the works.

Now there is also no guarantee that fixing these entries would unstuck the upgrade as the problem could still be related to broken shares.

pep29 · 2017-10-13T08:47:12Z

@PVince81:

i tried to test the change from #29074

But when i tried to repair with the command: occ files:scan --all --repair

i get the error:

PHP Fatal error: Class 'OC\Files\Utils\Scanner' not found in /srv/www/owncloud/apps/files/lib/Command/Scan.php on line 154

PVince81 · 2017-10-16T07:46:05Z

@pep29 maybe something went wrong with the patching ? Does the file "lib/private/Files/Utils/Scanner.php" still exist ?

For the encoding issue, please check that you have the PHP module "intl" enabled. (ref #29247)

pep29 · 2017-10-16T08:06:45Z

@PVince81 yes, the file "lib/private/Files/Utils/Scanner.php" still exist

pep29 · 2017-10-16T08:10:17Z

@PVince81 the "intl" module is enabled

pep29 · 2017-10-16T09:08:44Z

@PVince81 I also noticed there is an improvment with the above mentioned SQL result, before the upgrade attempt, the query gives 188 rows and after applying the patch, the query gives now 33 rows but then i am stuck using the command occ files:scan --all --repair

PVince81 · 2017-10-17T15:36:09Z

I'm a bit out of ideas. I don't see why it couldn't find a class that clearly exists, unless you did something wrong when patching. How did you patch ?

pep29 · 2017-10-18T07:27:15Z

@PVince81 you were right, i indeed did something wrong with the patching. The scan now works as it should, however i still have to disable the files_sharing, in order for the upgrade to success. Otherwise i am still stuck in the above mentioned avatar step

PVince81 · 2017-10-18T09:25:46Z

How long have you let the upgrade running ? It is likely that if you let it run long enough, the avatar step will finally exit at some point.

A similar issue about long running upgrade was reported before and is still being investigated. It seemed that setting up the FS for a specific user who received many shares somehow run in a lot of unnecessary loops.

PVince81 · 2017-12-21T14:34:50Z

All this time I was actually expecting SQLSessionInit to do this UTF-8 thing and now you discovered that it doesn't. I wonder whether it worked maybe with a previous version of Doctrine/DBAL and stopped working at some point. I remember that in the past we had "SET NAMES" explicitly and it seems we removed it because it was then done implicitly by Doctrine/DBAL.

The reason could be that now DB tables have a collation so it would use UTF-8 anyway as said in #29209 (comment). However I wonder now whether the DB connection itself (the data transfer) is actually done with UTF-8 or whether some setups would still transfer the data with the wrong encoding...

pep29 · 2017-12-21T14:42:24Z

I don't know exactly where the problem might be with this utf8 issue, since the update worked before prior to version 10. But when i put SQLSessionInit back after the upgrade, i am again stuck in the while loop.

I think, we could rename the ticket by now, since the update error was not actually due to the broken local shares, but some SQLSessionInit vs MysqlSessionInit issue :-D

PVince81 · 2018-01-23T15:37:34Z

@DeepDiver1975 @butonic any idea why the change from #29209 (comment) would fix encoding ? I expected DB encoding to be enforced by collations already

butonic · 2018-01-24T09:24:36Z

@pep29 what does SHOW VARIABLES WHERE Variable_Name LIKE 'collation%' OR Variable_Name LIKE 'character%' show when connecting with mysql, what does

<?php
include 'lib/base.php';
$result = \OC::$server->getDatabaseConnection()->executeQuery("SHOW VARIABLES WHERE Variable_Name LIKE 'collation%' OR Variable_Name LIKE 'character%'");
?>
<table>
	<?php foreach ($result as $row) { ?>
		<tr><td><?php echo $row['Variable_name']; ?></td><td><?=$row['Value']; ?></td></tr>
	<?php } ?>
</table>

show?

In my case CLI shows

+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
| collation_connection     | utf8mb4_general_ci         |
| collation_database       | utf8_bin                   |
| collation_server         | utf8mb4_general_ci         |
+--------------------------+----------------------------+

and via browser I get

character_set_client	utf8
character_set_connection	utf8
character_set_database	utf8
character_set_filesystem	binary
character_set_results	utf8
character_set_server	utf8mb4
character_set_system	utf8
character_sets_dir	/usr/share/mysql/charsets/
collation_connection	utf8_general_ci
collation_database	utf8_bin
collation_server	utf8mb4_general_ci

@PVince81 I don't know if browser as client not being utf8mb4 is a problem ... need to check with my google fu ...

pep29 · 2018-01-24T10:43:17Z

@butonic, my settings appear to be different:

CLI shows:

+--------------------------+---------------------------------------------+
| Variable_name            | Value                                       |
+--------------------------+---------------------------------------------+
| character_set_client     | latin1                                      |
| character_set_connection | latin1                                      |
| character_set_database   | latin1                                      |
| character_set_filesystem | binary                                      |
| character_set_results    | latin1                                      |
| character_set_server     | latin1                                      |
| character_set_system     | utf8                                        |
| character_sets_dir       | /usr/share/mysql-community-server/charsets/ |
| collation_connection     | latin1_german2_ci                           |
| collation_database       | latin1_german2_ci                           |
| collation_server         | latin1_german2_ci                           |
+--------------------------+---------------------------------------------+

and via browser:

character_set_client | utf8
character_set_connection | utf8
character_set_database | latin1
character_set_filesystem | binary
character_set_results | utf8
character_set_server | latin1
character_set_system | utf8
character_sets_dir | /usr/share/mysql-community-server/charsets/
collation_connection | utf8_general_ci
collation_database | latin1_german2_ci
collation_server | latin1_german2_ci

butonic · 2018-01-24T12:03:32Z

@pep29 ok, let us see what the colums look like:

SELECT table_name, column_name, character_set_name FROM information_schema.`COLUMNS`  WHERE table_schema = "owncloud" AND character_set_name IS NOT NULL;

they are all utf8 here.

pep29 · 2018-01-24T12:15:04Z

@butonic

SELECT table_name, column_name, character_set_name FROM information_schema.`COLUMNS`  WHERE table_schema = "owncloud" AND character_set_name IS NOT NULL;

They are all utf8 as well

ownclouders · 2018-03-04T01:07:53Z

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

PVince81 · 2018-04-23T15:16:20Z

@butonic is that enough or should all the DB settings have the correct collation ?

ownclouders · 2018-07-14T01:00:34Z

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

lock · 2019-07-30T11:00:26Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

pep29 mentioned this issue Oct 11, 2017

Upgrade 9.1.6.2 -> 10.0.2.1 null cache when moving avatars outside home (fed shares) #28763

Closed

ghost closed this as completed Oct 11, 2017

PVince81 reopened this Oct 12, 2017

PVince81 added the bug-analysis label Oct 12, 2017

PVince81 added this to the triage milestone Oct 12, 2017

PVince81 changed the title ~~Upgrade error owncloud 9.1.6.2 -> 10.0.3.3~~ Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) Oct 12, 2017

PVince81 added p2-high Escalation, on top of current planning, release blocker sev3-medium sev2-high feature:sharing and removed sev3-medium labels Oct 12, 2017

PVince81 self-assigned this Oct 12, 2017

ownclouders added the status/STALE label Jan 21, 2018

ownclouders removed the status/STALE label Jan 23, 2018

PVince81 mentioned this issue Jan 26, 2018

Renamed files on read-only local external storage cause "File was deleted from server" error #27550

Closed

ownclouders added the status/STALE label Feb 24, 2018

ownclouders closed this as completed Mar 4, 2018

ownclouders removed the status/STALE label Mar 4, 2018

PVince81 reopened this Mar 5, 2018

ownclouders added the status/STALE label Apr 5, 2018

felixboehm removed the sev2-high label Apr 9, 2018

ownclouders removed the status/STALE label Apr 9, 2018

PVince81 added p3-medium Normal priority and removed p2-high Escalation, on top of current planning, release blocker labels Apr 23, 2018

PVince81 modified the milestones: triage, maybe some day May 22, 2018

PVince81 removed their assignment Jun 5, 2018

ownclouders added the status/STALE label Jul 6, 2018

ownclouders closed this as completed Jul 14, 2018

ownclouders removed the status/STALE label Jul 14, 2018

lock bot locked as resolved and limited conversation to collaborators Jul 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) #29209

Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) #29209

pep29 commented Oct 11, 2017 •

edited

Loading

ghost commented Oct 11, 2017

PVince81 commented Oct 12, 2017

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 •

edited

Loading

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 •

edited

Loading

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 •

edited

Loading

pep29 commented Oct 12, 2017 •

edited

Loading

PVince81 commented Oct 13, 2017

pep29 commented Oct 13, 2017

PVince81 commented Oct 16, 2017

pep29 commented Oct 16, 2017

pep29 commented Oct 16, 2017

pep29 commented Oct 16, 2017

PVince81 commented Oct 17, 2017

pep29 commented Oct 18, 2017

PVince81 commented Oct 18, 2017

PVince81 commented Dec 21, 2017

pep29 commented Dec 21, 2017

PVince81 commented Jan 23, 2018

butonic commented Jan 24, 2018 •

edited

Loading

pep29 commented Jan 24, 2018 •

edited

Loading

butonic commented Jan 24, 2018 •

edited

Loading

pep29 commented Jan 24, 2018

ownclouders commented Mar 4, 2018

PVince81 commented Apr 23, 2018

ownclouders commented Jul 14, 2018

lock bot commented Jul 30, 2019

Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) #29209

Upgrade error owncloud 9.1.6.2 -> 10.0.3.3 (broken local shares) #29209

Comments

pep29 commented Oct 11, 2017 • edited Loading

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

ghost commented Oct 11, 2017

PVince81 commented Oct 12, 2017

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 • edited Loading

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 • edited Loading

PVince81 commented Oct 12, 2017

pep29 commented Oct 12, 2017 • edited Loading

pep29 commented Oct 12, 2017 • edited Loading

PVince81 commented Oct 13, 2017

pep29 commented Oct 13, 2017

PVince81 commented Oct 16, 2017

pep29 commented Oct 16, 2017

pep29 commented Oct 16, 2017

pep29 commented Oct 16, 2017

PVince81 commented Oct 17, 2017

pep29 commented Oct 18, 2017

PVince81 commented Oct 18, 2017

PVince81 commented Dec 21, 2017

pep29 commented Dec 21, 2017

PVince81 commented Jan 23, 2018

butonic commented Jan 24, 2018 • edited Loading

pep29 commented Jan 24, 2018 • edited Loading

butonic commented Jan 24, 2018 • edited Loading

pep29 commented Jan 24, 2018

ownclouders commented Mar 4, 2018

PVince81 commented Apr 23, 2018

ownclouders commented Jul 14, 2018

lock bot commented Jul 30, 2019

pep29 commented Oct 11, 2017 •

edited

Loading

pep29 commented Oct 12, 2017 •

edited

Loading

pep29 commented Oct 12, 2017 •

edited

Loading

pep29 commented Oct 12, 2017 •

edited

Loading

pep29 commented Oct 12, 2017 •

edited

Loading

butonic commented Jan 24, 2018 •

edited

Loading

pep29 commented Jan 24, 2018 •

edited

Loading

butonic commented Jan 24, 2018 •

edited

Loading