[6.0.0a] lucence jobs store path instead of fileid

[butonic]

Issue by jcfischer from Wednesday Dec 18, 2013 at 07:34 GMT
Originally opened as owncloud/core#6479

---

Expected behaviour

Syncing a directory without any errors

Actual behaviour

SQL Errors when inserting into oc_jobs

Steps to reproduce

1. Sync a directory with JPG files (not sure if that is the root cause)

Server configuration

Operating system: Ubuntu 13.04

Web server: Apache 2.4

Database: Postgres 9.2

PHP version: 5.5

ownCloud version: 6.0.0a

Updated from an older ownCloud or fresh install: fresh install

List of activated app: LDAP,

The content of config/config.php: (Without the database password and passwordsalt)

<?php
$CONFIG = array (
  'instanceid' => 'xxx',
  'passwordsalt' => 'yyy',
  'datadirectory' => '/mnt/data',
  'dbtype' => 'pgsql',
  'version' => '6.0.0.14',
  'dbname' => 'owncloud',
  'dbhost' => '10.0.20.42',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'owncloud',
  'dbpassword' => 'xxx',
  'installed' => true,
  'ldapIgnoreNamingRules' => false,
  'log_type' => 'owncloud',
  'logfile' => '/var/log/owncloud/owncloud.log',
  'loglevel' => '0',
  'logdateformat' => 'F d, Y H:i:s',
  'logtimezone' => 'Europe/Berlin',
  'log_query' => false,
  'log_authfailip' => false,
  'log_rotate_size' => 104857600,
);

Client configuration

Browser: OS X Sync Client 1.4.2

Operating system: OS X 10.8.5

Logs

ownCloud log (data/owncloud.log)

{"app":"OC\\Files\\Cache\\Scanner","message":"!!! No reuse of etag for 'files\/Reading\/Papers2\/Library.papers2\/Thumbnails\/D9903AF2-01E9-48F9-96D1-BA2F367DACB4.jpg' !!! \ncache: Array\n(\n    [fileid] => 3351\n    [storage] => home::b1967ce2-f535-1032-8c41-d1ab18bc04c7\n    [path] => files\/Reading\/Papers2\/Library.papers2\/Thumbnails\/D9903AF2-01E9-48F9-96D1-BA2F367DACB4.jpg\n    [parent] => 63\n    [name] => D9903AF2-01E9-48F9-96D1-BA2F367DACB4.jpg\n    [mimetype] => image\/jpeg\n    [mimepart] => image\n    [size] => 65449\n    [mtime] => 1387351183\n    [storage_mtime] => 1387351183\n    [encrypted] => \n    [unencrypted_size] => 0\n    [etag] => 52b14c93baf11\n)\n \ndata: Array\n(\n    [mimetype] => image\/jpeg\n    [mtime] => 1299615124\n    [size] => 65449\n    [etag] => 52b14c93bc1f9\n    [storage_mtime] => 1299615124\n)\n","level":0,"time":"December 18, 2013 08:19:47"}
{"app":"core","message":"OC_Image::loadFromFileHandle(): Trying","level":0,"time":"December 18, 2013 08:19:48"}
{"app":"core","message":"OC_Image->fixOrientation() No readable file path set.","level":0,"time":"December 18, 2013 08:19:48"}
{"app":"core","message":"OC_Image->fixOrientation() Orientation: -1","level":0,"time":"December 18, 2013 08:19:48"}
{"app":"hook","message":"error while running hook (OCA\\Search_Lucene\\Hooks::indexFile): An exception occurred while executing 'INSERT INTO \"oc_jobs\"(\"class\", \"argument\", \"last_run\") VALUES(?, ?, 0)':\n\nSQLSTATE[22001]: String data, right truncated: 7 ERROR:  value too long for type character varying(256)","level":3,"time":"December 18, 2013 08:19:48"}

[butonic]

Comment by karlitschek from Wednesday Dec 18, 2013 at 08:34 GMT

---

@butonic Can you have a look please?

[butonic]

Comment by butonic from Wednesday Dec 18, 2013 at 13:12 GMT

---

I am trying to store the path to new files as a parameter for the backroundjob. When it cannot be inserted it will be indexed the next time you click in the search field.

I'll prepare a PR that uses storage & fileids instead of the path to get rid of the problem.

In the meanwhile please ignore the error

{"app":"hook","message":"error while running hook (OCA\\Search_Lucene\\Hooks::indexFile): An exception occurred while executing 'INSERT INTO \"oc_jobs\"(\"class\", \"argument\", \"last_run\") VALUES(?, ?, 0)':\n\nSQLSTATE[22001]: String data, right truncated: 7 ERROR:  value too long for type character varying(256)","level":3,"time":"December 18, 2013 08:19:48"}

The other lines are not related to this issue. @DeepDiver1975 might have an idea about the severity of "!!! No reuse of etag for ...

[butonic]

Comment by marcello-lodi from Tuesday Jan 07, 2014 at 11:29 GMT

---

I have the same problem.
Operating system: CentOS 6.5
Web server: Apache 2.2.15
Database: Postgres 8.3
PHP version: 5.3.3
Owncloud: 6.0.0a-4.1

{"app":"hook","message":"error while running hook (OCA\Search_Lucene\Hooks::indexFile): An exception occurred while executing 'INSERT INTO "oc_jobs"("class", "argument", "last_run") VALUES(?, ?, 0)':\n\nSQLSTATE[22001]: String data, right truncated: 7 ERROR: value too long for type character varying(255)","level":3,"time":"2014-01-06T12:47:09+00:00"}

ownCloud version: 6.0.0a

[butonic]

Comment by disconn3ct from Friday Jan 10, 2014 at 21:07 GMT

---

I believe this is the same (or at least related). New Arch linux vhost, clean install. Getting these errors in the log while using the sync client to migrate data in:
Jan 10 16:00:28 foo postgres[3828]: ERROR: value too long for type character varying(256)
Jan 10 16:00:28 foo postgres[3828]: STATEMENT: INSERT INTO "oc_jobs"("class", "argument", "last_run") VALUES($1, $2, 0)

[butonic]

Comment by butonic from Friday Jan 17, 2014 at 17:39 GMT

---

changed name to better reflect the issue

[butonic]

Comment by usterk from Tuesday Jan 21, 2014 at 13:35 GMT

---

I have the same errors:

{"app":"hook","message":"error while running hook (OCA\\Search_Lucene\\Hooks::indexFile): An exception occurred while executing 'INSERT INTO \"oc_jobs\"(\"class\", \"argument\", \"last_run\") VALUES(?, ?, 0)':\n\nSQLSTATE[22001]: String data, right truncated: 7 ERROR:  value too long for type character varying(256)","level":3,"time":"2014-01-21T13:24:53+00:00"}
{"app":"hook","message":"error while running hook (OCA\\Search_Lucene\\Hooks::deleteFile): An exception occurred while executing 'INSERT INTO \"oc_jobs\"(\"class\", \"argument\", \"last_run\") VALUES(?, ?, 0)':\n\nSQLSTATE[22001]: String data, right truncated: 7 ERROR:  value too long for type character varying(256)","level":3,"time":"2014-01-21T13:24:53+00:00"}

Debian 7
x86_64
PHP 5.4.4
PostgreSQL 9.1

[butonic]

@jcfischer @marcello-lodi @disconn3ct @usterk Beware, this is bleeding edge code! Could you try replacing the contents of your search_lucene folder with this zip: https://github.com/owncloud/search_lucene/archive/search_lucene_refactoring.zip and report any issues in #1

Let me know if that solves the Class not found error and still works as expected.

[arachunok]

Is there an update on a fix for this issue and a process to reset the lucene indexes? We are currently encountering this issue when we have deep paths for some of our files. The files are synced to the server, but they fail to index with the error specified in this issue. This causes them to be unavailable in the web client and also causes a subsequent delete of the file on our computers the next time a sync operation occurs in the sync client.

[AceSlash]

From what I understand, the column "argument" is too short in the oc_jobs table, if this is the only issue, why not just extend it?

Something like 1024 char from 256 should work:

ALTER TABLE oc_jobs ALTER COLUMN argument TYPE character varying(1024);

[butonic]

@AceSlash the correct fix has been implemented in the https://github.com/owncloud/search_lucene/tree/search_lucene_refactoring branch. Unfortunately, my time gets soaked up in other projects. I'll try to polish it on friday.

[roelWuyts]

Hello,
I installed OwnCloud yesterday and notice the same errors in the log files. I assume the fix has not been included yet ? Given that I'm new I also don't know the impact of the bug: should this hold me back putting owncloud in actual production ?

Info on my setup:
Operating system: Ubuntu 14.04
Web server: Apache 2.4
Database: MySql 14.14
PHP version: 5.5.9
ownCloud version: 6.0.4 (stable)

[AceSlash]

@roelWuyts The alter from my earlier comment worked for me on a production environment (owncloud 6.0.4), so you should not have any issue if you do the same thing.

[roelWuyts]

Thanks, will do

[cippaciong]

@butonic I upgraded owncloud from version 7.0.2 to 7.0.3 and this error came up in my owncloud.log. I never had this error before.
Let me know if you need additional informations about my setup

[butonic]

@cippaciong please try with the master branch of search_lucene: https://github.com/owncloud/search_lucene/archive/master.zip

[cippaciong]

@butonic I'm having a hard time trying to reproduce it before I try the master branch. Any hint?

[butonic]

@cippaciong It will only appear if you create a new file with a very long path (>256 chars). The old search lucene will try to add an indexing job and fail with this error. The master version of search_lucene no longer does that.

[AceSlash]

In the meantime, extending the column "argument" like I described in my previous post will fix this issue.

[cippaciong]

@butonic I was able to reproduce the error creating a long path and I can confirm that it doesn't show up anymore using the master branch.

This time I had another error but I don't know if it's related to search_lucene, it looks more like a file permission error (I checked and the contents of that directory is owned by www-data while the php5 directory itself is owned by root):
Error PHP session_start(): ps_files_cleanup_dir: opendir(/var/lib/php5) failed: Permission denied (13) at /var/www/owncloud/lib/private/session/internal.php#21

@AceSlash thanks, I will try that later as a workaround

[cippaciong]

@butonic Never mind, that error was caused by a wrong line in my php.ini