KeyError exception in multiprocessing module

[sschuberth]

Using ScanCode 1ec43b7 on Linux with Python 2.7 I get:

Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/bin/scancode", line 9, in <module>
    load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')()
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/src/scancode/utils.py", line 64, in main
    standalone_mode=standalone_mode, **extra)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/src/scancode/cli.py", line 312, in scancode
    diag, timeout, max_memory)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/src/scancode/cli.py", line 409, in scan
    result = scanned.next()
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/local/lib/python2.7/site-packages/click/_termui_impl.py", line 238, in     
    return next(self.iter)
  File "/var/lib/jenkins/jobs/scancode/workspace/tools/scancode/src/scancode/cli.py", line 41, in wrap
    return func(self, timeout=timeout or 1e10)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 668, in next
    raise value
KeyError: 'sha1'

The command line used was

scancode --timeout 180 -n 2 -f html . /var/lib/jenkins/jobs/scancode/workspace/results/github.com/remy/nodemon/2016-10-06_2cd85b1/nodemon-scancode-1ec43b7.html

and the project scanned was remy/nodemon@2cd85b1.

[pombredanne]

Let me look into this.

[pombredanne]

I can reproduce this with:
$ scancode ../nodemon/test/fixtures/some\ \\file

[pombredanne]

The error comes from here: https://github.com/nexB/scancode-toolkit/blob/645902627cd82a9e90f7f3a82038c5b5c6f8130d/src/scancode/cache.py#L119

and this is because there is a backslash in the file name.

[pombredanne]

FWIW, these are a bunch of test files for nodemon. Fix coming in soon.

[sschuberth]

and this is because there is a backslash in the file name.

Eek. Which means it's not even a valid file name on Windows as also reported here.

[pombredanne]

So the fix is going to be in multiple steps:

1. ensure that exceptions are properly caught in threads and processes and reported as scan errors, always
2. ensure that on POSIX, files that are valid files are scanned alright (including having tests if we are on POSIX or not and handling paths splits accordingly
3. in extractcode and may be elsewhere either handle correctly or report as warnings paths that are eventually non-portable and overall weird (e.g with / or \ or : or other prohibited characters)

[pombredanne]

The PR #414 should fix things... but It need to review the tests on Windows and Mac first too.

[pombredanne]

So almost there with the Mac and Windows tests. I am using archives as test files such that the repo can still be cloned everywhere ... and updating extractcode to properly handle these cases and rename files with illegal file names on any given OS.

[pombredanne]

All the planned fixes have been applied eg.:

• ensure that exceptions are properly caught in threads and processes and reported as scan errors, always
• ensure that on POSIX, files that are valid files are scanned alright (including having tests if we are on POSIX or not and handling paths splits accordingly
• in extractcode and may be elsewhere either handle correctly or report as warnings paths that are eventually non-portable and overall weird (e.g with / or \ or : or other prohibited characters). This also uses a more advanced unicode to ascii transliteration for sanity.