Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to run in parallel with more than 14 nodes #1588

Closed
cyberjaime45 opened this issue Feb 21, 2020 · 22 comments · Fixed by getgauge/gauge-java#433
Closed

Unable to run in parallel with more than 14 nodes #1588

cyberjaime45 opened this issue Feb 21, 2020 · 22 comments · Fixed by getgauge/gauge-java#433

Comments

@cyberjaime45
Copy link

Expected behavior

Previous Gauge versions used to run fine with up to 30 in parallel

Actual behavior

We run our tests using Saucelabs headless feature. Previous versions had no problems running in parallel with up to 30-35 in parallel

Currently, If select to run with more than 10 in parallel I get this following error bellow. Running with 10. works fine.

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[Flash] Starting progress reporting at http://127.0.0.1:39277
Executing in 30 parallel streams.
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Failed to start runner. Timed out connecting to java
Successfully generated html-report to => ............./reports/staging/html-report/index.html

Successfully generated json-report to => ..............t/reports/staging/json-report

Specifications: 0 executed 0 passed 0 failed 0 skipped
Scenarios: 0 executed 0 passed 0 failed 0 skipped

Total time taken: 30.353s
34mINFO�[m] �[1m-----------------------------------------------------------------------
34mINFO�[m] �[1;32mBUILD SUCCESS�[m
34mINFO�[m] �[1m-----------------------------------------------------------------------
34mINFO�[m] Total time: 40.399 s
34mINFO�[m] Finished at: 2020-02-21T12:48:33-05:00
34mINFO�[m] Final Memory: 28M/393M
34mINFO�[m] �[1m------------------------------------------------------------------------�

Gauge version

Gauge version: 1.0.7
Commit Hash: ed7b4fd6

Plugins
-------
flash (0.0.2)
html-report (4.0.9)
java (0.7.4)
json-report (0.3.3)
screenshot (0.0.1)
@cyberjaime45
Copy link
Author

Here's the Gauge logs
21-02-2020 13:19:44.390 [Gauge] [DEBUG] Created gauge_screenshots_dir at /home/....../.gauge/screenshots
21-02-2020 13:19:44.396 [Gauge] [DEBUG] Plugin java is already installed.
21-02-2020 13:19:44.398 [Gauge] [DEBUG] Plugin html-report is already installed.
21-02-2020 13:19:44.402 [Gauge] [DEBUG] Plugin json-report is already installed.
21-02-2020 13:19:44.409 [Gauge] [DEBUG] Plugin flash is already installed.
21-02-2020 13:19:44.409 [Gauge] [DEBUG] Parsing started.
21-02-2020 13:19:44.470 [Gauge] [DEBUG] Started concepts parsing.
21-02-2020 13:19:44.587 [Gauge] [DEBUG] 95 concepts parsing completed.
21-02-2020 13:19:44.590 [Gauge] [DEBUG] Started specifications parsing.
21-02-2020 13:19:44.675 [Gauge] [DEBUG] 95 specifications parsing completed.
21-02-2020 13:19:44.675 [Gauge] [DEBUG] Parsing completed.
21-02-2020 13:19:44.906 [Gauge] [DEBUG] Checking updates...
21-02-2020 13:19:44.940 [Gauge] [DEBUG] Downloading https://downloads.getgauge.io/plugin/flash?l=java&p=flash,html-report,java,json-report,screenshot&o=linux&a=amd64
21-02-2020 13:19:45.334 [Gauge] [DEBUG] Downloading https://downloads.getgauge.io/plugin/html-report?l=java&p=flash,html-report,java,json-report,screenshot&o=linux&a=amd64
21-02-2020 13:19:45.458 [Gauge] [DEBUG] Downloading https://downloads.getgauge.io/plugin/java?l=java&p=flash,html-report,java,json-report,screenshot&o=linux&a=amd64
21-02-2020 13:19:45.587 [Gauge] [DEBUG] Downloading https://downloads.getgauge.io/plugin/json-report?l=java&p=flash,html-report,java,json-report,screenshot&o=linux&a=amd64
21-02-2020 13:19:45.703 [Gauge] [DEBUG] Downloading https://downloads.getgauge.io/plugin/screenshot?l=java&p=flash,html-report,java,json-report,screenshot&o=linux&a=amd64
21-02-2020 13:19:47.006 [Gauge] [DEBUG] Attempting to connect to grpc server at port: 34289
21-02-2020 13:19:47.259 [Gauge] [DEBUG] Successfully made the connection with runner with port: 34289
21-02-2020 13:19:47.259 [Gauge] [DEBUG] Validation started.
21-02-2020 13:19:48.423 [Gauge] [DEBUG] Validation completed.
21-02-2020 13:19:48.832 [Gauge] [DEBUG] Runner with PID:26011 has exited
21-02-2020 13:19:48.832 [Gauge] [DEBUG] Run started
21-02-2020 13:19:48.833 [Gauge] [DEBUG] Starting Html Report plugin
21-02-2020 13:19:49.027 [Gauge] [DEBUG] Attempting to connect to grpc server at port: 44105
21-02-2020 13:19:49.027 [Gauge] [DEBUG] Successfully made the connection with plugin with port: 44105
21-02-2020 13:19:49.028 [Gauge] [DEBUG] Starting JSON Report plugin
21-02-2020 13:19:49.220 [Gauge] [DEBUG] Attempting to connect to grpc server at port: 45931
21-02-2020 13:19:49.221 [Gauge] [DEBUG] Successfully made the connection with plugin with port: 45931
21-02-2020 13:19:49.221 [Gauge] [DEBUG] Starting Flash plugin
21-02-2020 13:19:49.601 [Gauge] [DEBUG] Attempting to connect to grpc server at port: 35441
21-02-2020 13:19:49.623 [Gauge] [DEBUG] Successfully made the connection with plugin with port: 35441
21-02-2020 13:19:49.623 [Gauge] [INFO] Executing in 30 parallel streams.
21-02-2020 13:20:19.632 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.632 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.632 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.632 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.642 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.642 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.650 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.650 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.654 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.654 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.658 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.658 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.659 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.660 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.662 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.662 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.669 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.669 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.671 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.671 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.676 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.676 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.683 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.683 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.687 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.687 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.692 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.692 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.695 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.695 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.700 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.700 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.723 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.723 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.723 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.723 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.729 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.729 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.751 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.751 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.788 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.788 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.801 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.801 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.819 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.819 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.845 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.845 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.863 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.863 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.885 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.885 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.904 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.904 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.922 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.922 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:19.979 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:19.979 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [DEBUG] Skipping 95 specifications
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.003 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.004 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
21-02-2020 13:20:20.005 [html-report] [DEBUG] Transformed SuiteResult to report structure
21-02-2020 13:20:20.018 [html-report] [CRITICAL] write /home/qauser/workspace/QA_automation_test/reports/staging/html-report/index.html: file already closed
21-02-2020 13:20:20.103 [Gauge] [ERROR] Unable to connect to plugin Html Report 4.0.9. rpc error: code = Unavailable desc = transport is closing
21-02-2020 13:20:20.103 [Gauge] [DEBUG] Killing Plugin Html Report 4.0.9
21-02-2020 13:20:20.104 [Gauge] [ERROR] Error occurred while waiting for plugin process to finish.
Error : exit status 1
21-02-2020 13:20:20.106 [json-report] [INFO] Successfully generated json-report to => /home/....../reports/staging/json-report
21-02-2020 13:20:20.113 [Gauge] [DEBUG] Sending kill message to JSON Report plugin.
21-02-2020 13:20:20.113 [Gauge] [DEBUG] Sending kill message to Flash plugin.
21-02-2020 13:20:20.114 [Gauge] [INFO] Specifications: 0 executed 0 passed 0 failed 0 skipped
21-02-2020 13:20:20.115 [Gauge] [INFO] Scenarios: 0 executed 0 passed 0 failed 0 skipped
21-02-2020 13:20:20.115 [Gauge] [INFO]
Total time taken: 31.17s

@negiDharmendra
Copy link
Contributor

@cyberjaime45 I tried running gauge specs in parallel with more than 30 stream and it works fine.
Can you please share more info.

  • which os your working on
  • java version

Note: Please make sure that the issues mentioned here are resolved first.

@cyberjaime45
Copy link
Author

I'm still investigating this issue, I'll get back to you when I get more info.

@cyberjaime45
Copy link
Author

cyberjaime45 commented Feb 29, 2020 •
edited
Loading

Still getting the same error when trying to run with 30 streams in parallel
Error:
Error occurred while waiting for plugin process to finish.
Error : exit status 1
Unable to connect to plugin Html Report 4.0.9. rpc error: code = Unavailable desc = transport is closing

Failed to kill plugin Html Report 4.0.9. os: process already finished
Server:
Ubuntu
Java 11

I can run in parallel with up to 15 streams. More than this will result in error mentioned above. It looks it needs more time. Any help will be greatly appreciated
@negiDharmendra

@sriv
Copy link
Member

sriv commented Mar 19, 2020

Unable to connect to plugin Html Report 4.0.9. rpc error: code = Unavailable desc = transport is closing

This error is unhelpful, and masks the underlying error. @cyberjaime45 - is there any additional information in logs/gauge.log?

@cyberjaime45
Copy link
Author

cyberjaime45 commented Apr 1, 2020 •
edited
Loading

I'm going to create a sample project that points to our Saucelabs account with 30 nodes. In this way you can run it from your side and debug it.

It'll take few days to set up.

@cyberjaime45
Copy link
Author

@sriv how can I send you a sample project to you guys for debugging purposes?

Email will be preferable

@cyberjaime45 cyberjaime45 changed the title Unable to run in parallel with more than 10 nodes Unable to run in parallel with more than 14 nodes Apr 11, 2020
@NivedhaSenthil NivedhaSenthil self-assigned this Apr 15, 2020
@NivedhaSenthil
Copy link
Member

@cyberjaime45 can you please mention what is the version of gauge and gauge-java that worked for you with 30 nodes lastly ? will help bisecting the issue

@cyberjaime45
Copy link
Author

Hi @NivedhaSenthil

I believe it was:
Gauge: gauge 1.0.5
gauge-java 0.7.3
html-report 4.0.5

@NivedhaSenthil
Copy link
Member

Given sample project works fine on my local machine with following config.

  Model Name:	MacBook Pro
  Model Identifier:	MacBookPro15,1
  Processor Name:	Intel Core i7
  Processor Speed:	2.2 GHz
  Number of Processors:	1
  Total Number of Cores:	6
  L2 Cache (per Core):	256 KB
  L3 Cache:	9 MB
  Memory:	16 GB

when running it consumed about 3GB of swap space. I could replicate the issue mentioned above in a docker container with swap space of about 2GB, by the moment the free space becomes 0 the error starts popping up. Since the behaviour is same even with older version of gauge and gauge-java for me, I believe it could be similar memory issue you are facing. Can you please try if having more memory solves the issue ?

@cyberjaime45
Copy link
Author

My current Ubuntu server:
Linux Machine:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
8GB of Ram

Before it used to work just fine. This is a wild guess, but it kind of started failing after
#134

@NivedhaSenthil
Copy link
Member

Unfortunately I couldn't get it working with the lower version which do not the changes mentioned, with a system having low memory. There could be various factors affecting memory usage... like other processes taking up space etc. Please try increasing the memory with increase in number of nodes.

@cyberjaime45
Copy link
Author

Thanks. I'll have to ask Devops to increase memory. I'll get back to you with some results very soon.

@cyberjaime45
Copy link
Author

cyberjaime45 commented Apr 17, 2020 •
edited
Loading

Devops adde more memory, with total of 15GB. I found something interesting. I recently updated to:
java (0.7.6).

With 0.7.6 I can run just 10 parallel with 15GB of RAM, however if I downgrade to 0.74 I can run up to 20 in parallel.

When the script is running, I see that java dk is taking all the memory, way above 10GB.

There's definitely an issue with memory consumption that it need to looked over with plugin gauge-java 0.7.4+

@NivedhaSenthil
Copy link
Member

Thanks for the detailed investigation, we will try profiling to see what could be the cause.

@NivedhaSenthil NivedhaSenthil removed their assignment Apr 21, 2020
sriv added a commit to getgauge/gauge-java that referenced this issue Apr 28, 2020
@sriv
use singleThreadExecutor instead of custom executor, fizes getgauge/g…
f07d5cc 
…auge#1588
@sriv sriv mentioned this issue Apr 28, 2020
Merged
sriv added a commit to getgauge/gauge-java that referenced this issue Apr 28, 2020
@sriv
use singleThreadExecutor instead of custom executor, fixes getgauge/g…
5598f49 
…auge#1588

Signed-off-by: sriv <srikanth.ddit@gmail.com>
@sriv
Copy link
Member

sriv commented Apr 28, 2020

This is an issue with gauge-java. PR for the fix: getgauge/gauge-java#433

negiDharmendra pushed a commit to getgauge/gauge-java that referenced this issue Apr 28, 2020
@sriv
use singleThreadExecutor instead of custom executor, fixes getgauge/g…
4e26cc8 
…auge#1588 (#433)

Signed-off-by: sriv <srikanth.ddit@gmail.com>
@cyberjaime45
Copy link
Author

@sriv @negiDharmendra

Gauge Java does not fix my issue. It actually made it worse. With 0.7.7 I can't event run 10 in parallel. I have to go back to Gauge java 0.7.4.

With 0.7.4, I can run a lot more up to 20 in parallel.
Machine:
Ubuntu
Memory 16GB
Core: 4
Java version: 11.0.7

Can you guys please, take a look at the differences between 0.7.4 and newer versions?

29-04-2020 06:04:12.101 [Gauge] [INFO] Executing in 10 parallel streams.
29-04-2020 06:04:42.107 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.107 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.107 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.107 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.110 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.110 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.112 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.112 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.113 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.113 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.118 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.118 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.122 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.122 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.122 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.122 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.128 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.128 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [DEBUG] Skipping 57 specifications
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.130 [Gauge] [ERROR] Failed to start runner. Timed out connecting to java
29-04-2020 06:04:42.132 [json-report] [INFO] Successfully generated json-report to => .........../reports/staging/json-report
29-04-2020 06:04:42.137 [html-report] [DEBUG] Transformed SuiteResult to report structure
29-04-2020 06:04:42.146 [html-report] [CRITICAL] write ........./reports/staging/html-report/index.html: file already closed
29-04-2020 06:04:42.174 [Gauge] [ERROR] Unable to connect to plugin Html Report 4.0.10. rpc error: code = Unavailable desc = transport is closing
29-04-2020 06:04:42.175 [Gauge] [DEBUG] Killing Plugin Html Report 4.0.10
29-04-2020 06:04:42.175 [Gauge] [ERROR] Failed to kill plugin Html Report 4.0.10. os: process already finished
29-04-2020 06:04:42.175 [Gauge] [DEBUG] Sending kill message to JSON Report plugin.
29-04-2020 06:04:42.175 [Gauge] [DEBUG] Sending kill message to Flash plugin.
29-04-2020 06:04:42.174 [Gauge] [ERROR] Error occurred while waiting for plugin process to finish.
Error : exit status 1
29-04-2020 06:04:42.176 [Gauge] [INFO] Specifications: 0 executed 0 passed 0 failed 0 skipped
29-04-2020 06:04:42.176 [Gauge] [INFO] Scenarios: 0 executed 0 passed 0 failed 0 skipped
29-04-2020 06:04:42.176 [Gauge] [INFO]
Total time taken: 30.046s

@sriv
Copy link
Member

sriv commented Apr 29, 2020

This is strange, because we noticed improvements in our own builds:

prior to the change - https://github.com/getgauge/gauge-java/actions/runs/88826219 (41 mins runtime)

post this change - https://github.com/getgauge/gauge-java/actions/runs/90043656 (23 mins)

Can you guys please, take a look at the differences between 0.7.4 and newer versions?

That's what I've been doing. See getgauge/gauge-java#433

Perhaps obvious, but is the pom.xml/build.gradle pointing to 0.7.7 of gauge-java?

@cyberjaime45
Copy link
Author

cyberjaime45 commented Apr 29, 2020 •
edited
Loading

Yes, both local machine and pom file have the same version of gauge java
Since I have installed in our machines Gauge version: 1.0.8 I can't install gauge java 0.7.3. However when I had installed Gauge < 1.0.8. I was able to run up to 30 in parallel with a machine that only had 7gb of memory.

With Gauge 1.0.8 (I had to install gauge java 0.7.4+). I noticed I was only able to run up to 14 in parallel with machine with 7gb of memory. With gauge java 0.7.5 and 0.7.6 I noticed I was only able to run with 10 in parallel.
So I asked Devops to increase memory to 16GB. It didn't help too much. So I downgraded to gauge java 0.7.4, then I was surprised I was able to run up 20 in parallel.

With gauge java 0.7.7, I can't even run 10 in parallel with machine with 16gm of memory.

Gauge java 0.7.4 works for me.

Machine Ubuntu
Cores: 4
Memory: 16gb
Java 11.0.7

Gauge version: 1.0.8
Plugins
flash (0.0.2)
html-report (4.0.10)
java (0.7.4) -> I can run up to 20-25 in parallel
java (0.7.5 & 0.7.6) -> I can run up to 10-14 in parallel
java (0.7.7) -> Less than 10 in parallel
json-report (0.3.3)
screenshot (0.0.1)
spectacle (0.1.4)

When you have time, please take a look. Thanks. For now I'll stick with 0.7.4. it simply works for me.

@sriv
Copy link
Member

sriv commented Apr 29, 2020

@cyberjaime45 - Can you please check some details for me:

  • is enable_multithreading set in the properties file for the environment you are running against?
  • what is the version of gauge-java in your pom.xml?

I don't think memory should be an issue, unless your tests need it. It's more likely that CPU is getting hogged (pure guess).

Given this is a 4 core machine, by default gauge would launch only 2 streams. You can override it, and set the streams (which you do), but when you don't have enable_multithreading=true, 10 streams would try to bring up 10 jvm processes (i.e. 1 process per stream).

Here is a quick benchmark that I did:

Steps:

  • build the image: docker build -t gauge-java:0.7.7 .
  • run the tests: docker run --cpus=4 -m=4G -v pwd:/gauge gauge-java:0.7.7 -Dnodes=30

I set the constraints to use 4 CPU and 4GB memory for 30 nodes. The peak memory usage was 1GB. [note: this project has enable_multithreading=true]

@sriv sriv reopened this Apr 29, 2020
@cyberjaime45
Copy link
Author

cyberjaime45 commented Apr 29, 2020 •
edited
Loading

@sriv Bingo!!!

That was it!!! I had enable_multithreading set to false. (I missed this one, my bad)

Now works fine. The below is running 25 in parallel with enable_multithreading set to true.

          total        used        free      shared  buff/cache   available
          15G        3.3G         10G        1.6M        1.4G         12G

We can put this issue to rest once and for all

@sriv
Copy link
Member

sriv commented Apr 29, 2020

Excellent. Glad to hear that.

I'll close this issue.

@sriv sriv closed this as completed Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

use singleThreadExecutor instead of custom executor getgauge/gauge-java
4 participants
@sriv @NivedhaSenthil @cyberjaime45 @negiDharmendra