Cannot Upgrade from 4.5.13 to 5.0.x

[kneiser]

Problem Description:

After upgrading (not update) from ownCloud v4.5.13 to v5.0.x folders and files appear in the ownCloud "admin" user account. While a "normal user" account some folders and files owned have disappeared. Symptoms such as this have been reported on the ownCloud forum, normal course of action in the past has been to recommend the "truncation" of records in the ownCloud file cache table(s). Doing this resolves the folder and file assignments issue but then all sharing of folders by security group and files by link or direct assignment setups are destroyed. Breaking the sharing setups have to be setup again, in larger ownCloud instances this is not good as there can be a lot of complex sharing setups.

Test Machine:

• Fedora 19 operating system.
• Apache V2.4.6
• PHP V5.5.3
• MariaDB V5.5.32
• Upgrade ownCloud V4.5.13 to V5.0.x.
• After installation of V5.0.x the V4.5.13 data is restored from backup using rsync. Only the data files, database dump, and OC config file are restored.
• Data configuration:
  • OC admin user.
  • normal user, account of the tester, contains different types of data files of which only one is shared out by public link.
  • three test user accounts. Main test user owns 4 different file types that are shared only by folder or individual file sharing to other two test users.
  • one test user assigned full read-write access at folder level via OC security group.
  • one test user assigned read only to 3 file types and read-write access to a text file.
    
    

Testing:

• Remove ownCloud and database instance from server.
• Install ownCloud on server. MariaDB has ownCloud instance with 36 tables .
• Restore owncloud from backups per specifications required for an "upgrade".
• Do database dump of ownCloud instance.
• Start Internet Browser, ownCloud maintenance mode starts.
• Login as ownCloud "admin user", navigate to "Admin".
• Folders from other users (normal user)appear. Navigate in to one of the folders, nothing happens. Cannot delete them. "admin" user not part of sharing setup in this test setup. Cannot delete them.
• Can set maximum upload size ( to 2GB) and save with no problem.
• All other settings seem fine, including external storage setting from prior V4.5.x instance.
• Login to an existing "normal" user (not test users). Folders are missing (they appear in admin user). Only one folder appears (set up for external storage folder).
• Normal user, local storage folder empty (sharing for videos). Worked fine in 5.0.12. Navigate in to folder, folder empty.
• admin user, checked users normal user is defined in the correct security group for local storage.
• test0 user, sharing by folder. Sharing seems to be defined correctly.
• test1user, shared folder shared folder from test0 appears. files in folder share correctly.
• test2 user, shared folder appears. shows shared files from test0. files shared correctly.
• test2 user, does not show shared file by link from normal user.
• Open a remote terminal session and run ownCloud file scan script.
  Command-line syntax:
  php console.php files:scan <user_id> #For rescanning a users file
  php console.php files:scan --all #For rescanning the files of all users
• Change directory: cd /var/hda/web-apps/owncloud/html/
• File scan for "all" users: php console.php files:scan --all
• File rescans of all users: php console.php files:scan --all Folders and files for all users appear in the script run.
• Login to admin user. Folders still there.
• Do database dump of ownCloud instance.
• admin still sees normal user folders still there.
• normal user local storage folder still empty.
• admin reset local storage security group to the same one.
• admin reset normal user to security group for local storage.
• normal user local storage
• test0 shared attribute for each file shows correct sharing to other users.
• test1 folder sharing still works fine.
• test2 direct file sharing still works fine.
• test2 still does not show shared file by link from normal user.
• Deleted external storage setup including security group.
• Redefined external storage setup including security group.
• Do database dump of ownCloud instance.
• External storage still does not work.

Testing Completed:

Please provide guidance for helping resolve this issue.

[VicDeo]

Failed to update database structure (MDB2 Error: no such table, _doQuery: [Error message: Could not execute statement][Last executed query: ALTER TABLE oc_calendar_objects RENAME TO oc_clndr_objects, CHANGE calendarid calendarid INT UNSIGNED DEFAULT 0 NOT NULL, CHANGE startdate startdate DATETIME DEFAULT '1970-01-01 00:00:00', CHANGE enddate enddate DATETIME DEFAULT '1970-01-01 00:00:00'][Native code: 1146][Native message: Table 'owncloud.oc_calendar_objects' doesn't exist])

Sounds familiar
#4013
#4085
#4336

[VicDeo]

@kneiser please try to disable calendar app before upgrade.
Workaround for calendar is here #4013 (comment)

[kneiser]

@VicDeo
I do a file restore from backup after reinstalling ownCloud. As so as I start OC the upgrade executes. Is there a way from the command-line to disable an app without going in to OC before the upgrade is executed?

[VicDeo]

@kneiser quick and dirty way is to move calendar files out of apps folder
alternatively execute this query on your DB

UPDATE oc_appconfig SET configvalue='no' WHERE appid='calendar' AND configkey='enabled';

[kneiser]

@VicDeo I disabled the Calendar app as suggested and executed the upgrade and my tests. There is some improvement but in the end previous sharing setups are still broken. During the course of the new testing the OC admin log now shows:
Fatal
core
Failed to update database structure (MDB2 Error: already exists, _doQuery: [Error message: Could not execute statement] [Last executed query: ALTER TABLE oc_bookmarks ADD description VARCHAR(4096) DEFAULT ' ' NOT NULL] [Native code: 1060] [Native message: Duplicate column name 'description'] )
2013-10-01T00:00:10+00:00
Warning
core
starting upgrade from 4.90.14 to 5.0.20
2013-10-01T00:00:16+00:00
Over the next day or so I will update this bug report with the new test results as disabling the Calendar app did help "somewhat". We should not have to disable something in order to get clean upgrades.

[kneiser]

@VicDeo Is there a way I can upload my database dump files to OC Github? I can only upload images that I can find.

[VicDeo]

@kneiser this is not needed if you use php console.php files:scan --all

Per ownCloud Forum recommendations, truncate ownCloud cache tables. There are two cache tables: oc_filecache, oc_fscache (truncate oc_filecache, truncate oc_fscache) (database dump 4,5 CSV files).

[kneiser]

@VicDeo please review the Testing section of my bug issue. I run the "php console.php files:scan --all" twice during the testing. The first run of this command the "file cache" table truncations were not done. The first run is an attempt to clean up the cache tables so that after the "admin" user is logged in that no folders or files that do not belong to this user are displayed. The files or folders that are displayed belong to other ownCloud users. Is there any user documentation posted in the ownCloud guides stating that the truncation of the cache tables is no longer required? On the ownCloud forum it appears to be a common practice recommendation.

[VicDeo]

@kneiser Is there any user documentation posted in the ownCloud guides stating that the truncation of the cache tables is required? :)

1.Truncation of cache table is just a hack invented by users before the console rescan was implemented. It breaks sharing since it doesn't preserve relations between fileids in cache and share fileids. In addition it has a side effect:
#4733
2. Table oc_fscache is not used in 5.x. 5.x uses oc_filecache instead.
3. This issue is probably caused by some glitch in cache migration script (oc_fscache to oc_filecache)

I tried to migrate from 4.5.13 to 5.0.12 and all entries were migrated successfully.
Can you let me know which entries are not visible for user after migration. (I have your dumps)

[kneiser]

@VicDeo I prepared a PDF document containing additional details that you requested. As I do not have your email address I sent the information to Frank.

[kneiser]

@VicDeo did my PDF document get forwarded to you?

[kneiser]

@VicDeo I was chatting with Frank K. on email and I tried an experiment. I tried an upgrade from OC V4.5.13 to V6 Beta 1 (I know it's not a recommended practice), it did not work. I got the same results as this bug report.

[DeepDiver1975]

@kneiser as we assume #5591 / #5630 fix your issue - can I ask you to retest your scenarios with either git branch stable5 or master(this is the upcomding oc6). THX

[kneiser]

@DeepDiver1975 As I understand:

1. Stable5 - new upcoming release of V5.0.x?
2. Upcoming OC6 - OC 6 Beta 2?

If so, I have the ability to test this in both releases. If not please clarify.

[DeepDiver1975]

exactly - this the way to go - THX a lot

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek Gentlemen I have completed my testing on this bug issue on both V5.0.13 and V6.0 Beta 3. I repeated the same testing steps (except I did not disable the calendar app in V5.0.13) with the same data.

Unfortunately I have the same results, it did not upgrade. I even did "php console.php files:scan --all" and did an additional test on both versions with the same results.

So where do we go from here? Is there something I am missing? Did I supply you with enough information from my past testing? Were you able to duplicate the same test case scenarios that I had?

I am no sure what to do next. I am bewildered. :(

[VicDeo]

citing myself: "Truncation of cache table is just a hack invented by users before the console rescan was implemented. It breaks sharing since it doesn't preserve relations between fileids in cache and share fileids"
So don't expect preserving any shares in case of oc_fscache truncation.

[kneiser]

@VicDeo The recent testing no longer included doing the file truncation. As recommend by you testing did include using the file rescan. I have just quickly updated this issue to remove those steps.

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek I have completed my tests on OC 5.0.13 and 6 Beta 3, both resulted in the same test results. Upgrading still does not work. However the latest fixes have changed the test results somewhat. I recently quickly updated my testing steps in this issue to reflect that I am no longer test using database truncation, but the cli php file rescan call.

Now the big question do I close this issue and open a new one to reflect the new test results? Or do I document this call with the new test tests and results? I have also performed a couple of database dumps after maintenance mode ran and after the file recan ran.

PS: I also trimmed down my OC test instance to bare bones files so that the database dump files were cleaner.

Will this new set of results be addressed in the near future? If so which version of OC will receive the updates> Your feedback would be appreciated.

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek I have revised this issue to reflect the new bug symptoms. I have sent database dumps to Frank, requesting they be sent to who ever will be addressing this bug issue.

Please note that I was testing V5.0.13 and had started to test V6 Beta 3 (it was showing same issues as V5.0.13) when all of a sudden Beta 4 came out.

[DeepDiver1975]

@kneiser so - what is the current state as of today? I'm a bit confused. Is the upgrade working now?

As an idea: you can execute the upgrade procedure from the terminal as well by calling php upgrade.php

[kneiser]

@DeepDiver1975 as of now, no, in v5 or v6 of OC we still cannot upgrade from v4.5.13. With the release of V5.0.13 AND V5 Beta 3 things at the end of an upgraded actually degraded more OC features and the end test results were a bit different. From what I understand changes were made in those releases I tested, at least understood, were for the purposes of this bug issue? Consequently since things have changed I decided to revamp my test data and slightly change my testing patterns somewhat. The results still revealed the same upgrade problems plus one more new item (local storage definitions broken). So I also changed the text of this issue to reflect the new test results.

Now you next question.

1. Do you want me to do the usual upgrade process, then test, then run the upgrade.php after that?
2. Do you want me to do my usual test. Followed by a separate upgrade test but manually running the upgrade.php (without initiating from a browser)?

[kneiser]

@DeepDiver1975 I ran the upgrade.php from the cli, I had the same results, upgrade failed to produce the desired results.
During the run the following was displayed:
Turned on maintenance mode
Updated database
Updating filecache, this may take really long...
... 20% done ...
... 40% done ...
... 60% done ...
... 80% done ...
... 100% done ...
Updated filecache
Turned off maintenance mode
Update successful

Also here is the log from the admin user Admin section:

Warning PHP Cannot modify header information - headers already sent by (output started at /var/hda/web-apps/owncloud/html/upgrade.php:39) at /var/hda/web-apps/owncloud/html/lib/user.php#658 2013-11-24T00:13:45+00:00
Warning PHP Cannot modify header information - headers already sent by (output started at /var/hda/web-apps/owncloud/html/upgrade.php:39) at /var/hda/web-apps/owncloud/html/lib/user.php#659 2013-11-24T00:13:45+00:00
Warning PHP session_regenerate_id(): Cannot regenerate session id - headers already sent at /var/hda/web-apps/owncloud/html/apps/calendar/appinfo/update.php#73

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek I have tested V5.0.14a (MariaDB is the database), still cannot successfully upgrade from v4.5.13. Additionally an external storage connection I have defined, and has always worked, is now not working. On this new issue in admin I have a green light on the external storage definition. I even deleted and created the external storage definition and it doesn't work at all (still had a green light). Permissions and ownership are: 775 apache:apache

I also tested v6.0.0a and it has the same set of problems.

Do you know if a developer will ever get assigned to this bug issue? I am willing to spend some time doing testing under a developer's guidance.

Current log in admin user => admin shows:

Warning PHP Undefined index: fileid at /var/hda/web-apps/owncloud/html/lib/files/cache/scanner.php#109 2013-12-28T17:36:41+00:00
Warning PHP Undefined index: fileid at /var/hda/web-apps/owncloud/html/lib/files/cache/scanner.php#109 2013-12-28T17:36:41+00:00
Warning PHP Undefined index: fileid at /var/hda/web-apps/owncloud/html/lib/files/cache/scanner.php#109 2013-12-28T17:36:42+00:00
Warning PHP Undefined index: fileid at /var/hda/web-apps/owncloud/html/lib/files/cache/scanner.php#109 2013-12-28T17:36:40+00:00
Warning PHP Undefined index: fileid at /var/hda/web-apps/owncloud/html/lib/files/cache/scanner.php#109 2013-12-28T17:36:39+00:00
Error core can't remove app files_imageviewer. It is not installed. 2013-12-28T17:23:31+00:00
Error core App "" (files_imageviewer) can't be used because it is not compatible with this version of ownCloud 2013-12-28T17:23:31+00:00
Warning core starting upgrade from 4.90.14 to 5.0.28 2013-12-28T17:23:26+00:00
Fatal core Failed to update database structure (MDB2 Error: no such table, _doQuery: [Error message: Could not execute statement] [Last executed query: ALTER TABLE oc_calendar_objects RENAME TO oc_clndr_objects, CHANGE calendarid calendarid INT UNSIGNED DEFAULT 0 NOT NULL, CHANGE objecttype objecttype VARCHAR(40) DEFAULT ' ' NOT NULL, CHANGE startdate startdate DATETIME DEFAULT '1970-01-01 00:00:00', CHANGE enddate enddate DATETIME DEFAULT '1970-01-01 00:00:00'] [Native code: 1146] [Native message: Table 'owncloud.oc_calendar_objects' doesn't exist] )

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek I recently acquired a Raspberry Pi and set it up as a Raspbian LAMP-like (use SQLite as database app) server. I successfully installed owncloud V5.0.14a on it. I setup the test cases, including a local external storage definition. Everything works as expected.

I have another (Fedora 19 LAMP-like with SQLite as database) test server (similar setup as Raspberry Pi). I have successfully installed and tested V5.0.14a and setup my test data also with an external storage definition. I then manually upgraded to V6.0.0a (v5.0.14a backups) and it works fine!

So far the problems reported on the bug issue seem to point to a problem in the upgrade process for ownCloud (V5.x and V6.x) from V4.5.13 (using MySQL or MariaDB). In the past I was able to upgrade from V4.x to V4.5.x with no problems.

Please Note: I have 3 test servers and one production server.

1. Production: Fedora 14, Amahi 6 LAMP stack, MSQL, ownCloud 4.5.13. Will not upgrade OC at this point.
2. Amahi Test: Fedora 19, Amahi 7 LAMP stack, MariaDB, ownCloud v5.x or v6.x (backup of OC 4.5.13 data) depending on testing.
3. x86 LAMP-like Test: Fedora 19, SQLite, ownCloud V5.0.14a or V6.0.0a (backup of OC 5.0.14a data), currently V6.
4. Raspberry Pi LAMP-like Test: Raspbian, SQLite, ownCloud V5.0.14a.

[kneiser]

@VicDeo @DeepDiver1975 @karlitschek I have just tested the upgrade from V4.5.13 to V6.0.1. The upgrade fails as usual HOWEVER it seems that someone's test data has made it in to v6.0.1. When existing users now get the same test data (photos, test odt file, mp3 file).

I noticed there in the OC "Test Plan" there are a couple of sections on testing upgrades. Did anyone try the test scenario in this bug issue? Those of us that are still on OC V4.5.13 with lots of data stored what do we do?

[karlitschek]

@kneiser I can't see any test data in the release. But we ship some example files starting with ownCloud 6.
This is probably what you see.

[PVince81]

Not sure about 4.5. If upgrading directly to 6.0.2 doesn't work you might try and upgrade to 5.0.15 first. I haven't tested it, so not sure whether it works, but I suppose that upgrading from OC 4 to OC 5 should be possible.
And then upgrade from OC 5 to OC 6.

[kneiser]

@PVince81 @karlitschek I did that upgrade path a long time ago on prior releases of V5.0.x with same results. The last time was V5.0.14a. I have seen nothing in any releases of v5.0.x or v6.0.x to indicate that this bug issue was ever addressed. I am involved in another open source server project where ownCloud is one of the applications used. Many members of our server community have a lot of data in v4.5.x and can't upgrade. As a contingency we are considering providing V6.x and v4.5.x available. Members that want V6.x can move to it but will have to redo their data and configurations.

[rysiekpl]

I also cannot upgrade from 4.5.x to 5.0.x. The upgrade hangs afte "Updated database" with "Please reload the page." After the reload I get "ownCloud is in maintenance mode" and there it hangs. I can set "maintenance" to "false" in the config, but then I will just get the "Updated databse" thing.

I have tried removing PHP session files from the server, changing the browsers, clearing oc_fscache, commenting out line 36 in lib/cache.php (all of these I have found on different fora in the Internet as response to this bug), all to no avail.

Battling this for 2 days now, and a clean install is simply not an option -- we're using ownCloud internally in our organisation and cannot afford to lose the data/settings.

Update: found the bug.

[karlitschek]

fixed