"Too many open files" errors on 2.17 #19765
Comments
|
Hmmm also seeing with With: But this runs fine: |
Good thinking: the env var is (You may need to |
|
@huonw I'm no longer able to consistently repro the problem on 2.17 (it still happens often, but not every time). I can say that I haven't seen the error yet when using |
|
The rust based parser consumes snapshots/digests (as did the Python parser, albeit via forked processes), and so shouldn't involve any "new" sources of open file handles. @danxmoran : What is |
Could the problem be from a difference in concurrency limits? The Python parser would have been constrained in some way by |
|
You've read this presumably? Your ulimit is very low indeed. |
That may be true! But even with Color's large codebase I haven't needed to bump the limit before 2.17, and it's especially fishy (to me, not knowing all the details of how it works) that goals like |
No, because in both cases, the way that those parsers actually get files to operate on is by capturing One thing that you could do is compare |
|
So it's not lost to the ether, from Slack Leads to this too many open files error. Can be re-produced on pantsbuild/example-adhoc javascript example by replacing the deps with: and running |
|
I did some experimenting, and @sureshjoshi's specific case is due to trying to hold open over 9500 files under |
|
To clarify, this is |
|
From Stu's PR, it looks like it's related to concurrent access to what are now files (previously in the LMDB store). I thought that was only for "large" files - but in any case, I'd call this a pretty small reproduction repo. They get far nuttier very quickly. |
stuhood
added a commit
that referenced
this issue
Oct 24, 2023
…ts (#20055) As described in #19765, `2.17.x` uses more file handles than previous versions. Based on the location of the reported error, I suspect that this is due to the move from using the LMDB store for all files, to using the filesystem-based store for large files (#18153). In particular: rather than digesting files inside of `spawn_blocking` while capturing them into the LMDB store (imposing the [limit of blocking threads](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads) from the tokio runtime), `fn store` moved to digesting them using tokio's async file APIs, which impose no such limit. This change adds a semaphore to (some) file opens to provide a best-effort limit on files opened for the purposes of being captured. It additionally (in the first commit) fixes an extraneous file handle that was being kept open during capture. Fixes #19765.
WorkerPants
pushed a commit
that referenced
this issue
Oct 24, 2023
…ts (#20055) As described in #19765, `2.17.x` uses more file handles than previous versions. Based on the location of the reported error, I suspect that this is due to the move from using the LMDB store for all files, to using the filesystem-based store for large files (#18153). In particular: rather than digesting files inside of `spawn_blocking` while capturing them into the LMDB store (imposing the [limit of blocking threads](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads) from the tokio runtime), `fn store` moved to digesting them using tokio's async file APIs, which impose no such limit. This change adds a semaphore to (some) file opens to provide a best-effort limit on files opened for the purposes of being captured. It additionally (in the first commit) fixes an extraneous file handle that was being kept open during capture. Fixes #19765.
WorkerPants
pushed a commit
that referenced
this issue
Oct 24, 2023
…ts (#20055) As described in #19765, `2.17.x` uses more file handles than previous versions. Based on the location of the reported error, I suspect that this is due to the move from using the LMDB store for all files, to using the filesystem-based store for large files (#18153). In particular: rather than digesting files inside of `spawn_blocking` while capturing them into the LMDB store (imposing the [limit of blocking threads](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads) from the tokio runtime), `fn store` moved to digesting them using tokio's async file APIs, which impose no such limit. This change adds a semaphore to (some) file opens to provide a best-effort limit on files opened for the purposes of being captured. It additionally (in the first commit) fixes an extraneous file handle that was being kept open during capture. Fixes #19765.
|
Can confirm that the javascript repro above is fixed by #20055 |
huonw
pushed a commit
that referenced
this issue
Oct 24, 2023
…ts (Cherry-pick of #20055) (#20078) As described in #19765, `2.17.x` uses more file handles than previous versions. Based on the location of the reported error, I suspect that this is due to the move from using the LMDB store for all files, to using the filesystem-based store for large files (#18153). In particular: rather than digesting files inside of `spawn_blocking` while capturing them into the LMDB store (imposing the [limit of blocking threads](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads) from the tokio runtime), `fn store` moved to digesting them using tokio's async file APIs, which impose no such limit. This change adds a semaphore to (some) file opens to provide a best-effort limit on files opened for the purposes of being captured. It additionally (in the first commit) fixes an extraneous file handle that was being kept open during capture. Fixes #19765. Co-authored-by: Stu Hood <stuhood@gmail.com>
huonw
pushed a commit
that referenced
this issue
Oct 24, 2023
…ts (Cherry-pick of #20055) (#20077) As described in #19765, `2.17.x` uses more file handles than previous versions. Based on the location of the reported error, I suspect that this is due to the move from using the LMDB store for all files, to using the filesystem-based store for large files (#18153). In particular: rather than digesting files inside of `spawn_blocking` while capturing them into the LMDB store (imposing the [limit of blocking threads](https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.max_blocking_threads) from the tokio runtime), `fn store` moved to digesting them using tokio's async file APIs, which impose no such limit. This change adds a semaphore to (some) file opens to provide a best-effort limit on files opened for the purposes of being captured. It additionally (in the first commit) fixes an extraneous file handle that was being kept open during capture. Fixes #19765. Co-authored-by: Stu Hood <stuhood@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
While testing out v2.17.0, I ran the following to test the perf of Rust-based dependency inference:
This failed with:
I thought it might be a problem with the new Rust-based system, so I ran the following to find the environment variable that would revert back to the Python-based implementation:
But it also crashed with:
Strangely, toggling pantsd on/off fixes the problem, but in different directions for the two goals:
Pants version
2.17.0
OS
MacOS
The text was updated successfully, but these errors were encountered: