Segmentation fault: 11

[yocontra]

Test case:

• Use chokidar to watch a large folder (like node_modules)
• Run rm -rf node_modules && npm install

Process will log Segmentation fault: 11 and die. This happens to me all the time when using watchify, which uses chokidar

[es128]

I can reproduce, although interestingly it doesn't happen if rm -rf node_modules and npm i are done separately. How about for you?

The segfault is obviously coming from underlying C stuff in node core or fsevents, but I'll see what I can do to track it down.

[yocontra]

@es128 There is a node module you can require that provides better stacktraces for segfaults - that might give you a good head start

[es128]

It doesn't happen when using either io.js or node 0.11

[es128]

Oh spoke too soon, I just got it to happen once with io.js (1.0.1), but that was like 1 of 10 attempts. I can make it happen pretty consistently with node 0.10

[es128]

I've narrowed it down to fsevents, but I'd be out of my depth in trying to actually fix it. Hopefully the folks at strongloop will be able to pitch in, as they've recently agreed to take over maintenance of the fsevents module and are already responsible for its io.js compatibility: https://github.com/bajtos/fsevents/pull/1#issuecomment-70133793.

[noahgrant]

I'm glad to see that (a) someone else has been seeing this and (b) that it might be getting worked on. I've seen the same errors with Gaze for months now, and it's pretty sporadic. Here's the stacktrace (the top entry is from the segfault-handler library):

PID 8783 received SIGSEGV for address: 0x10
0   segfault-handler.node               0x0000000100cf647f _ZL16segfault_handleriP9__siginfoPv + 287
1   libsystem_platform.dylib            0x00007fff87e98f1a _sigtramp + 26
2   ???                                 0x0000000101038ec0 0x0 + 4311977664
3   node                                0x000000010043c721 uv__async_event + 62
4   node                                0x000000010043c894 uv__async_io + 136
5   node                                0x0000000100449229 uv__io_poll + 1427
6   node                                0x000000010043ccd9 uv_run + 239
7   node                                0x00000001003f758a _ZN4node5StartEiPPc + 365
8   node                                0x0000000100001134 start + 52

i'm not totally sure how to interpret this, but it's at least a start.

I should also note that I'm using both watchify and gulp-watch, and I get this segfault both when I use the old gulp-watch with gaze as well as the new gulp-watch with chokidar.

[es128]

@noahgrant you can track progress on this in the fsevents repo's issue/PR tracker. Very grateful that @bnoordhuis has been giving it some attention. Waiting on him to come up with a new patch (or rewrite) as the next step. Unless someone else comes along with the requisite expertise and desire to work on this seems like there is not much else that can be done in the meantime.

[noahgrant]

Thanks, @es128!

[noahgrant]

by the way, i should note that the above stack trace is from Gaze, not Chokidar, but also presents much less information than the one you posted, @es128, on the fsevents repo, which has two lines referencing CoreFoundation and one referencing fse.node. The stack trace above seems pretty generic. What does the ??? even mean?

I guess my point is that if I've seen it for both Gaze and Chokidar, only the latter of which uses fsevents, maybe fsevents isn't the issue? Maybe it's node core, from libuv?

clarification: i don't actually know which one it's coming from (i'm using gaze from gulp-watch and chokidar from watchify), but it's not Gaze directly. If the null pointer bug causing this is from fsevents, then my stacktrace above should also be from chokidar. If it's from libuv/node core, then it's possible the stacktrace was from fs.watch/watchFile (and from Gaze indirectly)

[es128]

Yep that seems like a possibility, although I don't know enough to do much more than speculate.

[mcollina]

As a side note, I built https://github.com/mcollina/chokidar-child to work around this issue. It 100% solves it for me by spawning a child process for each watcher. This also helps a lot if you run multiple watcher instance from the same node process.

[es128]

@mcollina This prevents any crashes that are otherwise reproducible? Or the child process wrapper just recovers gracefully after a crash?

If the former, why do you think running in a child process makes a difference?

[mcollina]

The child process wrapper recovers gracefully after a crash.

Given that I was using some instances of the watcher to watch hundreds of files, the process wrapper solves my issues in two ways:

1. reduces the numbers of file watched per process, so that the segfault happens much less frequently
2. restarting the watch when the process segfault

I am on Mac, and it is related to the fsevents bug.

It makes chokidar working reliably for my particular use case, which was not before due to the segfault and the high number of watched files.

[es128]

Makes sense. Thanks for the insight.

[mcollina]

Thank you for maintaining this.

[phated]

Thanks for working on this. We are getting so close to releasing gulp4

[noahgrant]

@phated, just curious, how is gulp4 related to this? It looks like gulp4 is still using gaze ^0.5.x via vinyl-fs->glob-watcher@2.0.0. Gaze 0.5.x is just a wrapper around fs.watch/fs.watchFile, and doesn't have its own watcher (like Gaze 0.6.x) and doesn't use fsevents.

[phated]

@noahgrant gulpjs/glob-watcher#13 and gulpjs/gulp#651 (comment)

[noahgrant]

ahhh gotcha. awesome!

[es128]

fsevents/fsevents#61

If there's anyone who can help evaluate this - would be appreciated.

[es128]

This has likely been fixed for the described use-case (at least, vastly improved) with the release of fsevents@0.3.7.

Closing for now, but will reopen if someone can demonstrate a realistic and reproducible case that still exhibits this problem.

[xzyfer]

We've started seeing people experience this again today with the release node-sass@3.3.0.

@es128 appears to be correct that this was resolved in fsevents@0.3.7. However my research shows that it was reintroduced in fsevents@0.3.8 with their move to nan@2.

My investigation is in sass/node-sass#1107 (comment)

The segfaults stacktrace is

PID 47259 received SIGSEGV for address: 0x46
0   segfault-handler.node               0x00000001033c390f _ZL16segfault_handleriP9__siginfoPv + 287
1   libsystem_platform.dylib            0x00007fff8e80a5aa _sigtramp + 26
2   ???                                 0x0000000103429870 0x0 + 4349663344
3   fse.node                            0x00000001035d8616 _ZN3Nan3imp10GetWrapperIPFvRKNS_20FunctionCallbackInfoIN2v85ValueEEEENS0_15FunctionWrapperEEEPT0_T_ + 142
4   fse.node                            0x00000001035d7a40 _ZN3fse8FSEvents10InitializeEN2v86HandleINS1_6ObjectEEE + 194
5   node                                0x0000000100601047 _ZN4node6DLOpenERKN2v820FunctionCallbackInfoINS0_5ValueEEE + 725
6   node                                0x00000001001749ee _ZN2v88internal25FunctionCallbackArguments4CallEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEE + 158
7   node                                0x000000010019c97b _ZN2v88internalL19HandleApiCallHelperILb0EEENS0_11MaybeHandleINS0_6ObjectEEEPNS0_7IsolateERNS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgumentsE1EEE + 811
8   node                                0x000000010019f1cd _ZN2v88internalL21Builtin_HandleApiCallEiPPNS0_6ObjectEPNS0_7IsolateE + 61
9   ???                                 0x0000340e442060bb 0x0 + 57235877159099

My system

$ node -v
v2.5.0
$ npm -v
2.13.2
$ node -p process.versions
{ http_parser: '2.5.0',
  node: '2.5.0',
  v8: '4.2.77.21',
  uv: '1.6.1',
  zlib: '1.2.8',
  ares: '1.10.1-DEV',
  modules: '44',
  openssl: '1.0.2d' }
$ node -p process.platform
darwin
$ node -p process.arch
x64

[xzyfer]

I've opened an issue with fsevents (fsevents/fsevents#82).

[es128]

Reopening... although I don't think there's any immediate action to be taken here because unfortunately pinning chokidar to 0.3.7 would break io.js v3 compat.

Working toward #318 remains attractive

[phated]

@es128 it looks like the linked issue has been resolved, has this also?

[es128]

@phated good point. We'll keep an eye out, but closing again for now.