Limit memory overhead #6822

tylersmalley · 2016-04-07T21:25:27Z

V8 will not trigger a full mark-sweep & mark-compact until there is memory pressure for the max-old-space-size, which is 1.6GB on a 64bit system. This means that Kibana will, at times, consume over 1.6GB of RSS memory.

We will be setting max-old-space-size to 256MB unless it is ran with --dev, in which case we will use the node defaults.

Over-writing the max-old-space-size with NODE_OPTIONS is still possible.

tylersmalley · 2016-04-07T21:26:29Z

I am going to build and deploy this to DigitalOcean to get a definitive idea of what the usage is. But for now, let's discuss the language used in the documentation and the method for setting the default.

tylersmalley · 2016-04-08T16:25:36Z

bin/kibana.bat

@@ -22,7 +22,7 @@ If Not Exist "%NODE%" (
 )

 TITLE Kibana Server
-"%NODE%" %NODE_OPTIONS% "%DIR%\src\cli" %*
+"%NODE%" --max-old-space-size=256 %NODE_OPTIONS% "%DIR%\src\cli" %*


I need to update this batch file to reflect the changes made in bin/kibana. Downloading a Windows VM to verify before adding to the PR.

Here's two ways. I think the first is the more common (but longer);

@REM Loop through any command line parameters and if we find "--dev" add "--max-old-space-size=256" :parse IF "%~1"=="" GOTO endparse IF "%~1"=="--dev" set NODE_OPTIONS="--max-old-space-size=256 %NODE_OPTIONS%" SHIFT GOTO parse :endparse

And a single line way;

@REM If the command line parameters do not contain "--dev" then add "--max-old-space-size=256" echo "%*" | findstr /V /C:"--dev" && set NODE_OPTION="--max-old-space-size=256 %NODE_OPTION%"

tylersmalley · 2016-04-08T16:35:12Z

With max-old-space-size=256 the process' RSS is staying under 300M.

tylersmalley · 2016-04-08T22:45:31Z

@LeeDr - would you mind reviewing/testing the changes made to the batch file in this PR?

Bargs · 2016-04-09T17:33:24Z

Why remove the limit in dev? Dev should be as prod like as possible, otherwise we're just asking for pain down the road.

tylersmalley · 2016-04-10T01:41:56Z

@Bargs - the optimize process requires much more memory to run and is not used in production, except for plugin installation which goes through bin/kibana-plugin

Bargs · 2016-04-11T15:25:34Z

Hmmm, I'm guessing you've seen the optimizer crash with --max-old-space=256? I just started it up with 256 without trouble but it hasn't been running long. The optimizer gets started with cluster.fork so I think it should get its own heap. I thought we might be able to configure max-old-space separately if it's just the optimizer that needs more memory, but unfortunately that doesn't seem possible.

tylersmalley · 2016-04-11T15:44:40Z

This illustrates the memory usage as compared to 5.0.0-alpha1.

rashidkpc · 2016-04-14T23:41:21Z

README.md

@@ -6,6 +6,7 @@ Kibana is an open source ([Apache Licensed](https://github.com/elastic/kibana/bl

 - Elasticsearch master
 - Kibana binary package
+- 512 MB of available RAM


512 here? I assume this is because some amount of memory will of course be used that isn't old space

That is correct, it's just a recommendation. We could probably run on lower, but we do have things which are not old space.

tylersmalley · 2016-04-25T21:51:44Z

docs/production.asciidoc

+
+=== Memory management
+
+Kibana is built on Node.js which doesn't tune its heap size based on the amount of memory available. To combat this, we set defaults based on the requirements of Kibana needs, while allowing overhead for additional plugins. These defaults can be overridden at runtime, for example `NODE_OPTIONS="--max-old-space-size=512" bin/kibana`.


@rashidkpc, thoughts this verbiage?

rashidkpc · 2016-04-25T21:55:51Z

In total: LGTM

V8 will not trigger a full mark-sweep & mark-compact until there is memory pressure for the max-old-space-size, which is 1.6GB on a 64bit system. This means that Kibana will, at times, consume over 1.6GB of RSS memory. Bypassing this can be done by starting with `--dev` or explicitally setting `max-old-space-size`. For example: `NODE_OPTIONS="--max-old-space-size=1024" ./bin/kibana` Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

LeeDr · 2016-04-26T23:19:01Z

LGTM
I re-tested on Windows with no NODE_OPTIONS setting, and memory seemed to stay around 256M or lower.

With --dev and no NODE_OPTIONS, it went to about 614M while running functional tests.

With --max-old-space-size=50 it crashes with and without --dev.

With --max-old-space-size=5000 without --dev I saw the memory get to about 1G but then dropped down to around 500M while the functional tests were running.

tylersmalley · 2016-04-27T06:00:19Z

@epixa - what do you think about getting this into alpha2? It's ready to merge, but I can hold off if you would like to wait for beta1.

I also did further testing on Linux 32bit today.

epixa · 2016-04-27T13:40:59Z

Yep, let's do it.

epixa · 2016-04-28T14:16:39Z

I found a pretty big problem with this:

Optimizing does happen on the server in production in at least one scenario: If you remove a plugin, the next time you start up the server, the optimizer is run on the remaining plugins. I just removed x-pack on an alpha2 build and now I can't start kibana with the default settings because the optimizer makes it crash due to out-of-memory.

@kqualters-elastic

## Summary `eui@81.2.0` ⏩ `eui@81.3.0` ✨ Highlights: - fixes #158644 - Adds a new Timeline icon for use within `EuiMarkdownEditor` (cc @kqualters-elastic) - Expandable rows used within `EuiBasicTable` and `EuiInMemoryTable` now have a faster and snappier expand animation ___ ## [`81.3.0`](https://github.com/elastic/eui/tree/v81.3.0) - Added `timelineWithArrow` glyph to `EuiIcon` ([#6822](elastic/eui#6822)) **Bug fixes** - Fixed `EuiCodeBlock` potentially incorrectly ignoring lines ending with a question mark when using the Copy button. ([#6794](elastic/eui#6794)) - Fixed `EuiCodeBlock` to not include line numbers when copying content ([#6824](elastic/eui#6824)) - Fixed the expanded row animation on `EuiBasicTable` causing cross-browser Safari issues ([#6826](elastic/eui#6826))

tylersmalley force-pushed the default-node-arguments branch 3 times, most recently from 37fc9b8 to 0888ec7 Compare April 8, 2016 16:23

tylersmalley reviewed Apr 8, 2016
View reviewed changes

tylersmalley changed the title ~~Set default GC limit for V8~~ Limit memory overhead Apr 8, 2016

tylersmalley force-pushed the default-node-arguments branch 3 times, most recently from e133e0c to f91d316 Compare April 8, 2016 22:42

tylersmalley added the review label Apr 8, 2016

tylersmalley mentioned this pull request Apr 8, 2016

Lets talk about v8's memory settings #5595

Closed

rashidkpc reviewed Apr 14, 2016
View reviewed changes

rashidkpc assigned tylersmalley Apr 14, 2016

tylersmalley force-pushed the default-node-arguments branch from f91d316 to ea81885 Compare April 22, 2016 22:41

tylersmalley mentioned this pull request Apr 25, 2016

[node] Bump node.js to 4.4.3 #6966

Closed

tylersmalley removed their assignment Apr 25, 2016

tylersmalley reviewed Apr 25, 2016
View reviewed changes

tylersmalley force-pushed the default-node-arguments branch from ea81885 to 1ef98b1 Compare April 26, 2016 17:44

Tyler Smalley added 2 commits April 26, 2016 13:44

Adds documentation for memory requirements

e8e9f58

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley force-pushed the default-node-arguments branch from 1ef98b1 to e8e9f58 Compare April 26, 2016 21:51

epixa merged commit 855faff into elastic:master Apr 27, 2016

epixa added v5.0.0 v5.0.0-alpha2 labels Apr 27, 2016

tylersmalley deleted the default-node-arguments branch April 27, 2016 15:18

epixa mentioned this pull request Apr 28, 2016

Out-of-memory when trying to start kibana after removing plugin #7082

Closed

tylersmalley mentioned this pull request Apr 28, 2016

Revert "Limit memory overhead" #7088

Merged

epixa added the reverted label Oct 26, 2016

cjcenizal mentioned this pull request Jun 7, 2017

Add --max-old-space-size=8192 argument to bin/kibana script to avoid running out of memory during optimize step. #12232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit memory overhead #6822

Limit memory overhead #6822

tylersmalley commented Apr 7, 2016

tylersmalley commented Apr 7, 2016

tylersmalley Apr 8, 2016

LeeDr Apr 8, 2016

tylersmalley commented Apr 8, 2016

tylersmalley commented Apr 8, 2016

Bargs commented Apr 9, 2016

tylersmalley commented Apr 10, 2016

Bargs commented Apr 11, 2016

tylersmalley commented Apr 11, 2016

rashidkpc Apr 14, 2016

tylersmalley Apr 14, 2016

tylersmalley Apr 25, 2016

rashidkpc Apr 25, 2016

rashidkpc commented Apr 25, 2016

LeeDr commented Apr 26, 2016

tylersmalley commented Apr 27, 2016

epixa commented Apr 27, 2016

epixa commented Apr 28, 2016


		=== Memory management

		Kibana is built on Node.js which doesn't tune its heap size based on the amount of memory available. To combat this, we set defaults based on the requirements of Kibana needs, while allowing overhead for additional plugins. These defaults can be overridden at runtime, for example `NODE_OPTIONS="--max-old-space-size=512" bin/kibana`.

Limit memory overhead #6822

Limit memory overhead #6822

Conversation

tylersmalley commented Apr 7, 2016

tylersmalley commented Apr 7, 2016

tylersmalley Apr 8, 2016

Choose a reason for hiding this comment

LeeDr Apr 8, 2016

Choose a reason for hiding this comment

tylersmalley commented Apr 8, 2016

tylersmalley commented Apr 8, 2016

Bargs commented Apr 9, 2016

tylersmalley commented Apr 10, 2016

Bargs commented Apr 11, 2016

tylersmalley commented Apr 11, 2016

rashidkpc Apr 14, 2016

Choose a reason for hiding this comment

tylersmalley Apr 14, 2016

Choose a reason for hiding this comment

tylersmalley Apr 25, 2016

Choose a reason for hiding this comment

rashidkpc Apr 25, 2016

Choose a reason for hiding this comment

rashidkpc commented Apr 25, 2016

LeeDr commented Apr 26, 2016

tylersmalley commented Apr 27, 2016

epixa commented Apr 27, 2016

epixa commented Apr 28, 2016