[0.9.4.1] runtime/cgo: out of memory in thread_start

[brettdh]

Crash log file (raw)

Search for "out of memory" or "cannot allocate memory"; the log contains several such failures, after about a day's worth of writes every 10 seconds, with successful queries interspersed. Some of the panics occurred immediately after a query; some of them occurred in server restart loops; some occurred during WAL flushes.

There was also a random "unexpected fault address" panic in this log, but I have no idea if it's related.

[brettdh]

Note: in our setup, we impose a 1GB memory limit on our daemons by default, including influxd. It's currently hovering at 956m virtual, 35m resident, under the same write volume and occasional queries, so I see no obvious leak there.

[jwilder]

1GB may not be enough RAM for your dataset plus working room for queries and other tasks. I would try increasing it.

It's also possible that you are hitting a Go 1.5 bug that seems to be causing #4548 too. Go 1.5.2 might fix this.

[brettdh]

Bumped to 4GB and restarted influxd. Got stuck in the same startup loop for a while; realized new rlimit was not in place; did it right the second time. No restart loop with 4GB limit, and now it's running again. Will check it periodically for more query-related panics.

[nathanielc]

we have upgraded to go1.5.2, closing