Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External file scanning error #28181

Closed
adrienafl opened this issue Jun 21, 2017 · 40 comments
Closed

External file scanning error #28181

adrienafl opened this issue Jun 21, 2017 · 40 comments

Comments

@adrienafl
Copy link

adrienafl commented Jun 21, 2017

Hi ! Just having this error : I tried to fix perms with the strong perm script. I also tried to use update perms but it didn't change anything.

Thank you

Steps to reproduce

  1. Run sudo -u www-data php occ files:scan --all

Expected behaviour

Scan should complete

Actual behaviour

Exception during scan: Undefined index: size
#0 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(416): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/ownclo...', 416, Array)
#1 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(377): OC\Files\Cache\Scanner->handleChildren('files/Backup/PI...', true, 3, '57663', true, 119080535)
#2 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '57663', true)
#3 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '512', true)
#4 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup/PI...', true, 3, '508', true)
#5 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('files/Backup', true, 3, '169', true)
#6 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(311): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 2, true)
#7 /var/www/owncloud/lib/private/Files/Utils/Scanner.php(235): OC\Files\Cache\Scanner->scan('files', true, 3)
#8 /var/www/owncloud/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/xxx/files')
#9 /var/www/owncloud/apps/files/lib/Command/Scan.php(228): OCA\Files\Command\Scan->scanFiles('xxx', '/xxx/files', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false)
#10 /var/www/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/owncloud/core/Command/Base.php(158): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/owncloud/lib/private/Console/Application.php(160): Symfony\Component\Console\Application->run(NULL, NULL)
#16 /var/www/owncloud/console.php(99): OC\Console\Application->run()
#17 /var/www/owncloud/occ(11): require_once('/var/www/ownclo...')
#18 {main}

Server configuration

Operating system:
Debian 8
Web server:
Apache 2.4.10
Database:
mysql Ver 14.14 Distrib 5.5.55, for debian-linux-gnu (x86_64) using readline 6.3
PHP version:
PHP 5.6.30-0+deb8u1
ownCloud version: (see ownCloud admin page)
10.0.2.1
Updated from an older ownCloud or fresh install:
Upgrade from OC 9 (latest one)
Where did you install ownCloud from:
Package
Signing status (ownCloud 9.0 and above):

No error

The content of config/config.php:

{
    "system": {
        "updatechecker": false,
        "asset-pipeline.enabled": true,
        "instanceid": "ocln9v0ykp6k",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "owncloud.xxx.com"
        ],
        "datadirectory": "\/home\/owncloud",
        "overwrite.cli.url": "http:\/\/owncloud.xxx.com",
        "dbtype": "mysql",
        "version": "10.0.2.1",
        "dbname": "ocdb",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379
        }
    }
}

List of activated apps:

Enabled:

  • activity: 2.3.4
  • comments: 0.3.0
  • configreport: 0.1.1
  • dav: 0.2.9
  • federatedfilesharing: 0.3.0
  • federation: 0.1.0
  • files: 1.5.1
  • files_external: 0.7.0
  • files_pdfviewer: 0.8.2
  • files_sharing: 0.10.0
  • files_texteditor: 2.2
  • files_trashbin: 0.9.0
  • files_versions: 1.3.0
  • files_videoplayer: 0.9.8
  • firstrunwizard: 1.1
  • gallery: 16.0.2
  • market: 0.1.0
  • notifications: 0.3.0
  • provisioning_api: 0.5.0
  • systemtags: 0.3.0
  • templateeditor: 0.1
  • updatenotification: 0.2.1
    Disabled:
  • encryption
  • external
  • files_antivirus
  • theme-example
  • user_external
    Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: no

a) The javascript console log
b) The network log
c) ...

@ghost
Copy link

ghost commented Jun 21, 2017

Most likely a duplicate of #28031 . That one is still open but was solved by the OP by setting the correct permissions.

@adrienafl
Copy link
Author

I saw that but as I said i ran the Strong Perm script, every files are www-datat owned...

@ghost
Copy link

ghost commented Jun 22, 2017

Just keep in mind that if this is an external mounted storage (like NTFS) the permission script won't help here and might break more stuff then it solves.

@adrienafl
Copy link
Author

I see. However my data folder is on the same server on the same part just in another folder.

I have a clue : The file where the script fail are files with UTF8 character like "é à ê" etc

Maybe the buffer isn't made to handle very large strings due to UTF8 characters ?

@ghost
Copy link

ghost commented Aug 23, 2017

I performed a short test.
I created a folder named は in the user/files directory via ssh (correct permissions), and the scan failed with "Exception during scan: Undefined index: size"
After I removed the folder, everything worked fine.

@PVince81
Copy link
Contributor

PVince81 commented Sep 7, 2017

In my case it works for SFTP with OC 10.0.3RC1 :

± % occ files:scan --all -vvvv

Starting scan for user 1 out of 1 (admin)
        Folder /admin/
        Folder /admin/files_external
        Folder /admin/thumbnails
        Folder /admin/thumbnails/19
        File   /admin/thumbnails/19/32-32.png
        File   /admin/thumbnails/19/2048-2048-max.png
        Folder /admin/thumbnails/4
        File   /admin/thumbnails/4/32-32.png
        File   /admin/thumbnails/4/2048-2048-max.png
        Folder /admin/files
        File   /admin/files/welcome.txt
        Folder /admin/cache
        Folder /admin/files/sftp/
        File   /admin/files/sftp/y
        Folder /admin/files/sftp/x
        Folder /admin/files/sftp/は
        File   /admin/files/sftp/は/hello.txt

@stuporglue
Copy link

stuporglue commented Sep 22, 2017

I'm using SFTP with OC 10.0.3.2 and am getting the same error (Undefined index: size) when it tries to scan a folder named "Rīgas Doma zēnu koris".

Also:

  • Mel Tormé
  • Les Misérables (Original Broadway Cast Recording)
  • Alô, paixão.mp3

The scan was able to proceed after renaming these files.

--

The scan also produced the same error when it hit a dead symlink in the SFTP directory. Fixing the symlink let the scan complete.

@PVince81
Copy link
Contributor

@IljaN didn't you observe something similar ?

@PVince81 PVince81 added this to the development milestone Sep 23, 2017
@patrickjahns
Copy link
Contributor

@stuporglue
Thank's for your findings - the issue with the dead symlinks is addressed in this PR: #28959

However - the issue with the filenames is not yet addressed @IljaN

@IljaN
Copy link
Contributor

IljaN commented Sep 27, 2017

@stuporglue / @yakuraku Could you please provide the sshfs line which is used to mount the directory?

@stuporglue
Copy link

Unless OwnCloud sets up sshfs automatically, I don't think I'm using sshfs. I think it's just using sftp.

I enabled External Storage and set up the shares as the admin. Users aren't allowed to mount external storage. I chose "SFTP" as the storage type, and used "RSA public key" as the auth method.

I created an "owncloud_sftp" user on my remote machine.

I let OwnCloud generate the SSH key, then added the generated key to ~owncloud_sftp/.ssh/authorizedKeys

image

@ghost
Copy link

ghost commented Oct 1, 2017

@stuporglue Seems SFTP + keypair mode currently doesn't work at all according to #28669 and https://doc.owncloud.org/server/latest/admin_manual/release_notes.html#changes-in-10-0-3

@stuporglue
Copy link

stuporglue commented Oct 1, 2017 via email

@felixboehm felixboehm added the p2-high Escalation, on top of current planning, release blocker label Oct 5, 2017
@patrickjahns
Copy link
Contributor

@stuporglue

And it's working for me, except for files with certain characters. I've
renamed my files though so this isn't affecting me anymore at the moment.

Can you tell me, if you have php-intl installed/enabled? If it is missing - could you try installing / enabling php-intl and check if the error would still occur?

@PVince81 PVince81 modified the milestones: development, planned Nov 8, 2017
@stuporglue
Copy link

@patrickjahns Sorry for the delay responding here. I do have php-intl installed.

$ dpkg -l | grep php | grep intl
ii  php7.0-intl                                   7.0.22-3                              amd64        Internationalisation module for PHP

And here's the relevant part of php --info output:

intl

Internationalization support => enabled
version => 1.1.0
ICU version => 57.1
ICU Data version => 57.1

Directive => Local Value => Master Value
intl.default_locale => no value => no value
intl.error_level => 0 => 0
intl.use_exceptions => 0 => 0

@PVince81
Copy link
Contributor

Can you guys check the database collations ? DB collation, table and column collations. They all should say something like utf8.

@stuporglue
Copy link

Thanks for looking into this.

I'm using Postgres. Default collation is UTF-8...

                                  List of databases
    Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-------------+----------+----------+-------------+-------------+-----------------------
 owncloud    | owncloud | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

And there aren't any tables which override that.

owncloud=# select table_schema,
       table_name,
       column_name,
       collation_name
from information_schema.columns
where collation_name is not null
order by table_schema,
         table_name,
         ordinal_position;
 table_schema | table_name | column_name | collation_name
--------------+------------+-------------+----------------
(0 rows)

@PVince81 PVince81 modified the milestones: planned, development Nov 22, 2017
@PVince81 PVince81 assigned patrickjahns and unassigned IljaN Nov 28, 2017
@PVince81
Copy link
Contributor

Assigning to @patrickjahns who was trying to reproduce this, with no success so far.

@PVince81
Copy link
Contributor

PR to bring back the warning: #29845

@caguiar
Copy link

caguiar commented Dec 24, 2017

I have the same problem, i.e. the file scanner crashes, and the "problem" files which exist on the smb external storage, dosen't appear on the owncloud side.

From what I read above enabling the Mac compatibility in the external storage should have resolved the issue(? did I understand wrong?)
however, I have enabled the Mac compatibility in the external storage settings, which solved 99% of the problems, but I still have at least one file that dos not "sync".

Here is the owncloud.log content:

{"reqId":"NTjJlqJ7xJUeKaugL09z","level":3,"time":"2017-12-24T13:09:36+00:00","remoteAddr":"10.244.6.61","user":"97C5B7E0-4380-41D7-B203-5EC3195AD4CC","app":"PHP","method":"PROPFIND","url":"\/remote.php\/dav\/files\/97C5B7E0-4380-41D7-B203-5EC3195AD4CC\/_OUTRAS-ATIVIDADES_\/Prestacao_servicos\/UBQ%20-%20Prest.servi%C3%A7os\/Pedido%20Remb.%2020-02-2015","message":"Undefined index: size at \/var\/www\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#421"}
{"reqId":"NTjJlqJ7xJUeKaugL09z","level":3,"time":"2017-12-24T13:09:36+00:00","remoteAddr":"10.244.6.61","user":"97C5B7E0-4380-41D7-B203-5EC3195AD4CC","app":"PHP","method":"PROPFIND","url":"\/remote.php\/dav\/files\/97C5B7E0-4380-41D7-B203-5EC3195AD4CC\/_OUTRAS-ATIVIDADES_\/Prestacao_servicos\/UBQ%20-%20Prest.servi%C3%A7os\/Pedido%20Remb.%2020-02-2015","message":"Undefined index: size at \/var\/www\/owncloud\/lib\/private\/Files\/Cache\/Scanner.php#424"}

And here is a zip with a file with a name that issued the error message:

problem_file.zip

@ownclouders
Copy link
Contributor

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

@GregorB54321
Copy link

GregorB54321 commented Mar 16, 2018

Short Braindump of my analysis:

`sudo -uwww-data ./occ files:scan --repair OCLogonUser
[sudo] Passwort für server:
Starting scan (and repair) for user 1 out of 1 (OCLogonUser)
0 [>---------------------------]
Exception during scan: ErrorException: Undefined index: size
#0 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/html/o...', 421, Array)
#1 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('files_versions/...', true, 3, '83097', true, 0)
#2 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83097', true)
#3 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83096', true)
#4 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions/...', true, 3, '83095', true)
#5 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(385): OC\Files\Cache\Scanner->scanChildren('files_versions', true, 3, '24448', true)
#6 /var/www/html/owncloud/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, '32', true)
#7 /var/www/html/owncloud/lib/private/Files/Utils/Scanner.php(238): OC\Files\Cache\Scanner->scan('', true, 3)
#8 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(224): OC\Files\Utils\Scanner->scan('/OCLogonUser', true)
#9 /var/www/html/owncloud/apps/files/lib/Command/Scan.php(310): OCA\Files\Command\Scan->scanFiles('OCLogonUser', '/OCLogonUser', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true)
#10 /var/www/html/owncloud/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/owncloud/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/owncloud/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/owncloud/lib/private/Console/Application.php(162): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/html/owncloud/console.php(106): OC\Console\Application->run()
#17 /var/www/html/owncloud/occ(11): require_once('/var/www/html/o...')
#18 {main}

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 2221 | 28005 | 00:00:58 |
+---------+-------+--------------+
`

MariaDB [owncloud]> select fileid, path from oc_filecache where fileid in (83097 ,83096 ,83095 ,24448 ,32 ); +--------+---------------------------------------------------+ | fileid | path | +--------+---------------------------------------------------+ | 32 | | | 24448 | files_versions | | 83095 | files_versions/.thunderbird | | 83096 | files_versions/.thunderbird/xyz123ab.default | | 83097 | files_versions/.thunderbird/xyz123ab.default/Mail | +--------+---------------------------------------------------+ 5 rows in set (0.00 sec)

server-p560 Mail # pwd /var/www/owncloud_data/OCLogonUser/files_versions/.thunderbird/xyz123ab.default/Mail server-p560 Mail # ls -lah insgesamt 0 drwxr-xr-x 1 www-data www-data 214 Feb 26 21:17 . drwxr-xr-x 1 www-data www-data 26K Mär 15 17:30 .. drwxr-xr-x 1 www-data www-data 5,0K Mär 15 17:30 Surname.Lastname drwxr-xr-x 1 www-data www-data 5,4K Mär 15 17:30 LastnameSurname drwxr-xr-x 1 www-data www-data 4,0K Mär 15 17:27 www.domain.de drwxr-xr-x 1 www-data www-data 4,2K Mär 15 17:27 adress_one drwx------ 1 root root 5,9K Mär 14 11:25 adress_two drwxr-xr-x 1 www-data www-data 6,9K Mär 15 17:30 adress_three drwxr-xr-x 1 www-data www-data 734 Mär 15 17:27 Local Folders drwxr-xr-x 1 www-data www-data 868 Mär 15 11:00 smart mailboxes server-p560 Mail #

server-p560 Mail # pwd /var/www/owncloud_data/OCLogonUser/files_versions/.thunderbird/xyz123ab.default/Mail server-p560 Mail # ls -lah insgesamt 0 drwxr-xr-x 1 www-data www-data 214 Feb 26 21:17 . drwxr-xr-x 1 www-data www-data 26K Mär 15 17:30 .. drwxr-xr-x 1 www-data www-data 5,0K Mär 15 17:30 Surname.Lastname drwxr-xr-x 1 www-data www-data 5,4K Mär 15 17:30 LastnameSurname drwxr-xr-x 1 www-data www-data 4,0K Mär 15 17:27 www.domain.de drwxr-xr-x 1 www-data www-data 4,2K Mär 15 17:27 adress_one drwx------ 1 root root 5,9K Mär 14 11:25 adress_two drwxr-xr-x 1 www-data www-data 6,9K Mär 15 17:30 adress_three drwxr-xr-x 1 www-data www-data 734 Mär 15 17:27 Local Folders drwxr-xr-x 1 www-data www-data 868 Mär 15 11:00 smart mailboxes server-p560 Mail #

`
server@server-p560 /var/www/html/owncloud $ sudo -uwww-data ./occ files:scan --repair OCLogonUser
Starting scan (and repair) for user 1 out of 1 (OCLogonUser)
0 [>---------------------------]

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 2302 | 29225 | 00:00:59 |
+---------+-------+--------------+
`

please reformat anybody my post. i have no clue how to do that!

@PVince81
Copy link
Contributor

Please apply the patch from #30302 to make the "Undefined index" error disappear. Maybe a more useful message will appear.

Please also try setting the options for the external storage to use the encoding compatibility mode. You likely have files which names are encoded with UTF-8 NFD.

@jean151515
Copy link

jean151515 commented Mar 16, 2018

I had the same issue with owncloud-10.0.7 with external storage hooked up using SFTP.

What did the trick for me was mass converting my files using a linux package named convmv:
convmv -f utf8 -t utf8 --nfc --replace --nosmart --notest -r "Your Folder"/

@PVince81
Copy link
Contributor

PVince81 commented Apr 3, 2018

no time to finish this, moving to planned for next

@PVince81 PVince81 modified the milestones: development, planned Apr 3, 2018
@PVince81
Copy link
Contributor

The error will be gone and scanning will go forward. However the NFD error message is still missing.

@PVince81 PVince81 added p3-medium Normal priority and removed p2-high Escalation, on top of current planning, release blocker labels Apr 23, 2018
@IljaN
Copy link
Contributor

IljaN commented Apr 23, 2018

@PVince81 It is a bit tricky to add a message because on a low-level php can`t read the metadata (size) of the file, AFAIR there are different conditions which can lead to this so we would need some reliable way to detect NFD-Filenames beforehand. But need to dig deeper again to have the proper context for this problem.

@PVince81 PVince81 modified the milestones: development, backlog May 28, 2018
@PVince81 PVince81 removed their assignment May 28, 2018
@COArikan
Copy link

COArikan commented Oct 1, 2020

Hi, I've made a temporary fix for this issue in /var/www/owncloud/lib/private/Files/Cache/Scanner.php
Line 421 changed from
} else if ($data['size'] === -1) {
to
} else if ( (isset($data['size'])) && ($data['size'] === -1) ) {

and before Line 424, added a line as:
if (isset($date['size']))
$size += $data['size'];

It is fixed the error and scanning continue..

@phil-davis
Copy link
Contributor

The line numbers look out-of-date compared to current master https://github.com/owncloud/core/blob/master/lib/private/Files/Cache/Scanner.php

PR https://github.com/owncloud/core/pull/32408/files did some changes there in 2018.

@COArikan what version of ownCloud are you running?

Can you make a PR to owncloud/core with the changes?
That will avoid confusion about exactly what changes you are suggesting.

@stale
Copy link

stale bot commented Sep 21, 2021

This issue has been automatically closed.

@stale stale bot closed this as completed Sep 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.