KeyError in read_events while executing shell command

[jajadinimueter]

Hi!

I keep getting this exception while executing the following command

watchmedo shell-command -p "*.rst" -i build --ignore-directories -w -R -c bin/sphinx-build

Traceback (most recent call last):
  File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner
    self.run()
  File "/path/to/watchdog-0.7.1-py3.3.egg/watchdog/observers/api.py", line 187, in run
    self.queue_events(self.timeout)
  File /path/to/watchdog-0.7.1-py3.3.egg/watchdog/observers/inotify.py", line 135, in queue_events
    inotify_events = self._inotify.read_events()
  File "/path/to/watchdog-0.7.1-py3.3.egg/watchdog/observers/inotify_c.py", line 321, in read_events
    wd_path = unicode_paths.encode(self._path_for_wd[wd])
KeyError: 4007

I'm on version 0.7.1.

Do you need more information?

[jajadinimueter]

I did just this for a temporary fix:

jajadinimueter@3c2be97

It works for me

[tamland]

It would be nice to have a reproduceable example to find out why you are getting events for unknown watch descriptors instead of sweeping it under the rug. I have seen these errors before in logs, but not been able to find out what triggers it.

[danilobellini]

This is exactly #117. I've found this issue when running my dose project on my PyScanPrev project root directory with dose tox; that triggers the issue on every single run when using the specific commits in these links.

After spending some time looking for the "root path of all evil", I think I found a reproduceable example to trigger that issue: a really fast create-remove loop for the same directory name a few times (about 30, perhaps less) seems enough for a stochastical approach. I've already add that as a test, and I've found a solution for this issue about crashing. But I think that opened up a Pandora's box.

As it seems, that's an issue about a DELETE-CREATE-IGNORE sequence in the inotify buffer for the very same directory. To really force that in a non-stochastic way I'd need to mock events in this particular order, as the IN_DELETE_SELF and IN_IGNORE are tied together by inotify, not by different commands.

The loop really created 30 IN_CREATE | IN_DIR and 30 IN_DELETE | IN_DIR InotifyEvent instances, besides the IN_DELETE_SELF ones, which are less than 30, perhaps because some were deleted before a new wd was assigned to them. The Pandora's box is: when testing for watchdog events, that's what happens:

E         Differing items:
E         {<class 'watchdog.events.DirDeletedEvent'>: 37} != {<class 'watchdog.events.DirDeletedEvent'>: 30}
E         {<class 'watchdog.events.DirCreatedEvent'>: 28} != {<class 'watchdog.events.DirCreatedEvent'>: 30}
E         {<class 'watchdog.events.DirModifiedEvent'>: 55} != {<class 'watchdog.events.DirModifiedEvent'>: 60}

Testing again, it's clear that the numbers are random:

E         Differing items:
E         {<class 'watchdog.events.DirModifiedEvent'>: 51} != {<class 'watchdog.events.DirModifiedEvent'>: 60}
E         {<class 'watchdog.events.DirCreatedEvent'>: 26} != {<class 'watchdog.events.DirCreatedEvent'>: 30}
E         {<class 'watchdog.events.DirDeletedEvent'>: 43} != {<class 'watchdog.events.DirDeletedEvent'>: 30}

Actually, that's what I've got after implementing the straightforward solution for fixing this issue. I'm using tox -e py35 -- -k fast_sub to test based on #368.

[DonyorM]

Fixed by #375.