100% CPU when opening terminal and takes lot of time #835

adangel · 2024-06-21T11:21:59Z

Describe the bug
Debian Testing recently fixed #1029152 which has the consequence that the hard limit for NOFILE is now much bigger than before. That means that _SC_OPEN_MAX is now as big as 1073741816 instead of only 1048576.

When closing the file descriptors of the forked child process, the for-loop now takes a lot of time:

cdt/core/org.eclipse.cdt.core.native/native_src/unix/exec_pty.c

Lines 110 to 117 in 0583a05

    
           { 
        
               int fdlimit = sysconf(_SC_OPEN_MAX); 
        
               int fd = 3; 
        
               while (fd < fdlimit) { 
        
                   close(fd++); 
        
               } 
        
           }

Eventually it finished and the terminal is working. But every time a new terminal window is opened, it takes again a lot of time (couple of minutes).

To Reproduce
Steps to reproduce the behavior:

Make sure you have a big number in /proc/sys/fs/nr_open: echo 1073741816 > /proc/sys/fs/nr_open
Start eclipse
Verify the limits of the eclipse process, e.g. prlimit --nofile -p <PID OF eclipse> should output:
```
RESOURCE DESCRIPTION                    SOFT       HARD UNITS
NOFILE   max number of open files 1073741816 1073741816 files
```
If the soft limit is lower, you can set it with prlimit --nofile=1073741816:1073741816 -p <PID OF eclipse>
Open a new local terminal
It takes a long time until the prompt is shown

Expected behavior
The prompt is shown instantly and the terminal is directly usable.

Version Information (please complete the following information):

OS and OS Version/extra details: Linux, Debian Testing
Eclipse Version from Help: 2024-06 (4.32.0)
CDT Version from Help -> About -> Installation Details -> Features tab: 11.6.0.202403071917

Additional context

This is the same problem as described in JetBrains/pty4j#147 .
It seems, that pty4j and the terminal functionality from eclipse-cdt stem from the same origin - ELT. Maybe the same fix (JetBrains/pty4j@04685d8) from pty4j can be reused here?

The text was updated successfully, but these errors were encountered:

jonahgraham · 2024-06-21T14:08:08Z

Thanks @adangel for this detailed report. Since pty4j uses code originally from Eclipse CDT I hope we can bring the fix back to CDT easily enough.

Do you have the knowledge/capacity to apply the fix to CDT? I can help support the development.

It seems, that pty4j and the terminal functionality from eclipse-cdt stem from the same origin - ELT. Maybe the same fix (JetBrains/pty4j@04685d8) from pty4j can be reused here?

FWIW I think all of it was derived from the CDT version. The code's blame is from 2004 and the code was introduced to fix https://bugs.eclipse.org/bugs/show_bug.cgi?id=27663

adangel · 2024-06-22T07:34:49Z

@jonahgraham see #840 - hope this is OK copying from EPL 1.0 to EPL 2.0...

…ipse-cdt#835) When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com>

jonahgraham · 2024-06-23T16:10:00Z

Thank you @adangel for applying the fix to CDT's code base. I have tested #840 on my machine and I can see the before and after behaviour.

reopening because it auto-closed when applied to main branch, I want to backport this to 11.6 branch, which is the current target milestone of this issue.

haukehoffmann · 2024-06-24T10:45:13Z

Hi adangel, many thanks for your post. It saved my day.

My eclipse (CDT) got stucked with the command:
CDT Scanner Discovery: Invoking Command: gcc -E -P -v -dD spec.c
and a echo 1048576 > /proc/sys/fs/nr_open fixed it quickly for now.

I'm using the following versions:
Debian Testing Linux
Eclipse Version: 2024-06 (4.32.0), Build id: 20240606-1231

markusdrescher · 2024-06-27T21:46:27Z

Hi @jonahgraham @adangel,

thanks for the fix. I also stumbled across this issue on Debian unstable. I've downloaded libspawner.so from the PR into my eclipse installation, i.e. into configuration/org.eclipse.osgi/102/0/.cp/os/linux/x86_64/
to test if the issue is gone with Eclipse 2024-06. That is the case for instance when running a program via CDT. However, trying to debug it via CDT, it times out with a popup message: "Could not determine GDB version using command: gdb --version"

So using the workaround echo 1048576 > /proc/sys/fs/nr_open helps in that case.
Is there something similar in the codebase of CDT for debugging?

jonahgraham · 2024-06-27T22:59:30Z

Thanks @markusdrescher for highlighting this. I have reproduced the error launching debug session with the unfixed CDT and with the fix in #840. I see the CPU at 100% for 10 seconds, and then the error. The 10 seconds is because of:

cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/launching/LaunchUtils.java

Line 338 in 5654112

timeoutJob.schedule(10000);

I am now looking to see if we need to apply the same fix from #840 to somewhere else in CDT.

jonahgraham · 2024-06-27T23:09:30Z

Indeed - the same fix is needed here:

cdt/core/org.eclipse.cdt.core.native/native_src/unix/exec_unix.c

Line 85 in c598eed

/* Close all the fd's in the child */

…ipse-cdt#835) _This commit adds to eclipse-cdt#840 7bd8d52 to apply the same fix to another place that does the same operation on all fds._ When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com>

jonahgraham · 2024-06-27T23:26:12Z

@markusdrescher You can pull the updated libspawner out of PR #846 libspawner.so (x86_64)

Once I get a clean build I should be able to merge this and then backport all the changes to CDT 11.6 release branch.

…ipse-cdt#835) When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com> (cherry picked from commit 7bd8d52)

…ipse-cdt#835) _This commit adds to eclipse-cdt#840 7bd8d52 to apply the same fix to another place that does the same operation on all fds._ When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com> (cherry picked from commit 3875408)

…ipse-cdt#835) When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com> cherry picked and squashed from commits: - 7bd8d52 - 24d9bd1 - 3875408

When _SC_OPEN_MAX (max nr of open files limit per process) is a very big number, then closing all possible file handles can take a while. This change first tries to use the syscall close_range() if available, falling back to use /proc/self/fd to close only open file handles, and lastly falling back to the old way of closing all possible handles one after another. In general, the first or second approach should be available which speeds up the pty spawning. Refs JetBrains/pty4j#147 Copied from JetBrains/pty4j@04685d8 (which is EPL 1.0) Co-authored-by: Sergey Simonchik <sergey.simonchik@jetbrains.com> cherry picked and squashed from commits: - 7bd8d52 - 24d9bd1 - 3875408

hansiglaser · 2024-06-29T12:44:58Z

Also STM32CubeIDE suffers from this problem, see https://community.st.com/t5/stm32cubeide-mcus/cubeide-does-not-start-and-spawner-reaper/m-p/690290
Thanks for fixing!

rgrr · 2024-07-01T07:03:27Z

I can confirm that this also fixes long delays in "Build Targets".

Thank you for the fix.

markusdrescher · 2024-07-01T19:09:38Z

@markusdrescher You can pull the updated libspawner out of PR #846 libspawner.so (x86_64)

Once I get a clean build I should be able to merge this and then backport all the changes to CDT 11.6 release branch.
@jonahgraham Fix confirmed on my side for the Debug issue.

Thank you for fixing these issues on such a short notice. Remarkable. Keep up the good work!

jonahgraham · 2024-07-02T00:44:39Z

Thank you all for confirming the fix - it is now in the 11.6 branch and I will release as soon as practical (It is Canada Day here so slightly delayed today)

The pre-release can be installed from https://ci.eclipse.org/cdt/job/cdt/job/cdt_11_6/lastSuccessfulBuild/artifact/releng/org.eclipse.cdt.repo/target/repository/ in the meantime (zip option)

The release means promoting the next build to the official download location so that Check for Updates picks it up.

jonahgraham · 2024-07-03T01:16:18Z

Once the mirrors do their thing Check for Updates in Eclipse will pick up this fix (assuming that https://download.eclipse.org/tools/cdt/releases/11.6/ or https://download.eclipse.org/tools/cdt/releases/latest are listed in available software sites)

adangel mentioned this issue Jun 22, 2024

Speed up closing all file descriptors in the child process (Fixes #835) #840

Merged

jonahgraham added this to the 11.6.1 milestone Jun 23, 2024

jonahgraham closed this as completed in #840 Jun 23, 2024

jonahgraham closed this as completed in 7bd8d52 Jun 23, 2024

jonahgraham reopened this Jun 23, 2024

jonahgraham closed this as completed in 3875408 Jun 28, 2024

jonahgraham reopened this Jun 28, 2024

ghentschke mentioned this issue Jun 28, 2024

JSON Compilation Database Generator properties page is visible in non C/C++ projects #847

Closed

jonahgraham linked a pull request Jun 28, 2024 that will close this issue

[cdt_11_6] Speed up closing all file descriptors in the child process (Fixes #835) #853

Merged

jonahgraham closed this as completed Jul 3, 2024

jld01 mentioned this issue Jul 31, 2024

Eclipse CDT 2024-06 on Debian is too slow #877

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

100% CPU when opening terminal and takes lot of time #835

100% CPU when opening terminal and takes lot of time #835

adangel commented Jun 21, 2024

jonahgraham commented Jun 21, 2024

adangel commented Jun 22, 2024

jonahgraham commented Jun 23, 2024

haukehoffmann commented Jun 24, 2024 •

edited

Loading

markusdrescher commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

hansiglaser commented Jun 29, 2024

rgrr commented Jul 1, 2024

markusdrescher commented Jul 1, 2024 •

edited

Loading

jonahgraham commented Jul 2, 2024

jonahgraham commented Jul 3, 2024

100% CPU when opening terminal and takes lot of time #835

100% CPU when opening terminal and takes lot of time #835

Comments

adangel commented Jun 21, 2024

jonahgraham commented Jun 21, 2024

adangel commented Jun 22, 2024

jonahgraham commented Jun 23, 2024

haukehoffmann commented Jun 24, 2024 • edited Loading

markusdrescher commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

jonahgraham commented Jun 27, 2024

hansiglaser commented Jun 29, 2024

rgrr commented Jul 1, 2024

markusdrescher commented Jul 1, 2024 • edited Loading

jonahgraham commented Jul 2, 2024

jonahgraham commented Jul 3, 2024

haukehoffmann commented Jun 24, 2024 •

edited

Loading

markusdrescher commented Jul 1, 2024 •

edited

Loading