Retention policy deletion check make influxdb out of memory

[liuyue2656]

I used Influxdb 1.6.3 with docker container. I set the docker memory limit 16G, the influx used 5.6G memory on usually. but when the Retention policy deletion check start, influxdb will be out of memory.

InfluxDB shell version: 1.6.3

The secondary times with this problem,

on first times, i just drop shard *** fix it, but two days ago, this problem reappeared.

logs like this:

ts=2018-11-23T11:14:10.531351Z lvl=info msg="Retention policy deletion check (start)" log_id=0Bx2XxtG000 service=retention trace_id=0Bx4HLtl000 op_name=retention_delete_check op_event=start
...
ts=2018-11-23T11:14:45.362340Z lvl=info msg="InfluxDB starting" log_id=0Bx4JTxW000 version=1.6.3 branch=1.6 commit=389de31c961831de0a9f4172173337d4a6193909
ts=2018-11-23T11:14:45.362379Z lvl=info msg="Go runtime" log_id=0Bx4JTxW000 version=go1.10.3 maxprocs=4
ts=2018-11-23T11:14:45.477967Z lvl=info msg="Using data dir" log_id=0Bx4JTxW000 service=store path=/var/lib/influxdb/data

oom logs:

Nov 23 08:21:49 internal01 kernel: [5667645.229997] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 08:52:51 internal01 kernel: [5669507.320670] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 09:23:55 internal01 kernel: [5671370.871603] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 09:55:03 internal01 kernel: [5673239.425317] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 10:26:29 internal01 kernel: [5675124.963615] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 10:57:32 internal01 kernel: [5676987.736887] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 11:28:53 internal01 kernel: [5678869.204816] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 11:59:55 internal01 kernel: [5680731.005487] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 12:30:58 internal01 kernel: [5682594.168392] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 13:02:01 internal01 kernel: [5684456.581666] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 13:33:03 internal01 kernel: [5686318.619853] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 14:04:08 internal01 kernel: [5688183.801135] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 14:35:10 internal01 kernel: [5690045.601794] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 15:06:23 internal01 kernel: [5691918.455915] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 15:37:30 internal01 kernel: [5693785.261420] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 16:08:31 internal01 kernel: [5695646.521986] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 16:39:32 internal01 kernel: [5697507.109775] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 17:10:31 internal01 kernel: [5699366.354329] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 17:41:35 internal01 kernel: [5701229.986819] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 18:12:36 internal01 kernel: [5703091.510759] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 18:43:43 internal01 kernel: [5704958.154558] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
Nov 23 19:14:43 internal01 kernel: [5706818.021524] influxd invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0

My config:

[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  engine = "tsm1"
  wal-dir = "/var/lib/influxdb/wal"
  cache-max-memory-size = "8g"
  cache-snapshot-memory-size = "128m"

Retention policies:

> show retention policies;
name    duration  shardGroupDuration replicaN default
----    --------  ------------------ -------- -------
autogen 1440h0m0s 168h0m0s           1        true

shards:

> show shards
name: _internal
id  database  retention_policy shard_group start_time           end_time             expiry_time          owners
--  --------  ---------------- ----------- ----------           --------             -----------          ------
251 _internal monitor          251         2018-11-16T00:00:00Z 2018-11-17T00:00:00Z 2018-11-24T00:00:00Z
252 _internal monitor          252         2018-11-17T00:00:00Z 2018-11-18T00:00:00Z 2018-11-25T00:00:00Z
253 _internal monitor          253         2018-11-18T00:00:00Z 2018-11-19T00:00:00Z 2018-11-26T00:00:00Z
254 _internal monitor          254         2018-11-19T00:00:00Z 2018-11-20T00:00:00Z 2018-11-27T00:00:00Z
257 _internal monitor          257         2018-11-20T00:00:00Z 2018-11-21T00:00:00Z 2018-11-28T00:00:00Z
258 _internal monitor          258         2018-11-21T00:00:00Z 2018-11-22T00:00:00Z 2018-11-29T00:00:00Z
259 _internal monitor          259         2018-11-22T00:00:00Z 2018-11-23T00:00:00Z 2018-11-30T00:00:00Z
260 _internal monitor          260         2018-11-23T00:00:00Z 2018-11-24T00:00:00Z 2018-12-01T00:00:00Z

name: quotes
id  database retention_policy shard_group start_time           end_time             expiry_time          owners
--  -------- ---------------- ----------- ----------           --------             -----------          ------
183 quotes   autogen          183         2018-09-24T00:00:00Z 2018-10-01T00:00:00Z 2018-11-30T00:00:00Z
192 quotes   autogen          192         2018-10-01T00:00:00Z 2018-10-08T00:00:00Z 2018-12-07T00:00:00Z
201 quotes   autogen          201         2018-10-08T00:00:00Z 2018-10-15T00:00:00Z 2018-12-14T00:00:00Z
210 quotes   autogen          210         2018-10-15T00:00:00Z 2018-10-22T00:00:00Z 2018-12-21T00:00:00Z
219 quotes   autogen          219         2018-10-22T00:00:00Z 2018-10-29T00:00:00Z 2018-12-28T00:00:00Z
228 quotes   autogen          228         2018-10-29T00:00:00Z 2018-11-05T00:00:00Z 2019-01-04T00:00:00Z
237 quotes   autogen          237         2018-11-05T00:00:00Z 2018-11-12T00:00:00Z 2019-01-11T00:00:00Z
246 quotes   autogen          246         2018-11-12T00:00:00Z 2018-11-19T00:00:00Z 2019-01-18T00:00:00Z
255 quotes   autogen          255         2018-11-19T00:00:00Z 2018-11-26T00:00:00Z 2019-01-25T00:00:00Z

name: sandbox_quotes
id  database       retention_policy shard_group start_time           end_time             expiry_time          owners
--  --------       ---------------- ----------- ----------           --------             -----------          ------
256 sandbox_quotes autogen          256         2018-11-19T00:00:00Z 2018-11-26T00:00:00Z 2018-12-03T00:00:00Z

[aslobodskoy-fiksu]

#10453

[dgnorton]

@liuyue2656 have you tried switching to tsi1 index instead of inmem as mentioned in the possible duplicate issue linked in the comment above?

[liuyue2656]

I think it's ok now, I updated the Shard Duration from 168h0m8s to 24h0m0s, it's ok.

If this problem appear again, I'll try change the index to tsi1.