Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.0.7 Compacting file result in OOM on Raspberry Pi 4B #22124

Open
ProBackup-nl opened this issue Aug 9, 2021 · 0 comments
Open

2.0.7 Compacting file result in OOM on Raspberry Pi 4B #22124

ProBackup-nl opened this issue Aug 9, 2021 · 0 comments
Labels
area/memory area/performance area/storage area/2.x OSS 2.0 related issues and PRs

Comments

@ProBackup-nl
Copy link

ProBackup-nl commented Aug 9, 2021 •
edited
Loading

Steps to reproduce:

  1. Disable swap
  2. Insert new measurement every second using socat to /api/v2/write via python script
  3. Wait 23 days

Expected behavior:
No OOM

Actual behavior:
influxd OOM

Environment info:

  • Linux 5.10.48-1-ARCH aarch64
  • InfluxDB version: InfluxDB 2.0.7 (git: 2a45f0c) build_date: 2021-06-04T19:17:40Z
  • Disk usage:
dev             665M     0  665M   0% /dev
run             925M  744K  925M   1% /run
/dev/mmcblk0p2  117G   11G  101G  10% /
tmpfs           925M     0  925M   0% /dev/shm
tmpfs           925M     0  925M   0% /tmp
/dev/mmcblk0p1  240M   43M  198M  18% /boot
tmpfs           185M     0  185M   0% /run/user/0

Logs:

Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357097Z lvl=info msg="TSM compaction (start)" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357871Z lvl=info msg="Beginning compaction" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=l>
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357929Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.357967Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358003Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358039Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358074Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358113Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358148Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:24 influxPi4B influxd[17654]: ts=2021-07-31T09:01:24.358182Z lvl=info msg="Compacting file" log_id=0VY3A5a0000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level >
Jul 31 11:01:27 influxPi4B kernel: influxd invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Jul 31 11:01:27 influxPi4B kernel: CPU: 3 PID: 17748 Comm: influxd Tainted: G         C        5.10.48-1-ARCH #1
Jul 31 11:01:27 influxPi4B kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
Jul 31 11:01:27 influxPi4B kernel: Call trace:
Jul 31 11:01:27 influxPi4B kernel:  dump_backtrace+0x0/0x1a0
Jul 31 11:01:27 influxPi4B kernel:  show_stack+0x18/0x24
Jul 31 11:01:27 influxPi4B kernel:  dump_stack+0xc8/0x120
Jul 31 11:01:27 influxPi4B kernel:  dump_header+0x48/0x1f4
Jul 31 11:01:27 influxPi4B kernel:  oom_kill_process+0x204/0x20c
Jul 31 11:01:27 influxPi4B kernel:  out_of_memory+0xe4/0x320
Jul 31 11:01:27 influxPi4B kernel:  __alloc_pages_slowpath.constprop.0+0x850/0xba0
Jul 31 11:01:27 influxPi4B kernel:  __alloc_pages_nodemask+0x298/0x2f0
Jul 31 11:01:27 influxPi4B kernel:  handle_mm_fault+0x4dc/0xc90
Jul 31 11:01:27 influxPi4B kernel:  do_page_fault+0x140/0x3d4
Jul 31 11:01:27 influxPi4B kernel:  do_translation_fault+0xb0/0xcc
Jul 31 11:01:27 influxPi4B kernel:  do_mem_abort+0x44/0xa4
Jul 31 11:01:27 influxPi4B kernel:  el0_da+0x28/0x34
Jul 31 11:01:27 influxPi4B kernel:  el0_sync_handler+0x168/0x1b0
Jul 31 11:01:27 influxPi4B kernel:  el0_sync+0x180/0x1c0
Jul 31 11:01:27 influxPi4B kernel: Mem-Info:
Jul 31 11:01:27 influxPi4B kernel: active_anon:206675 inactive_anon:243243 isolated_anon:0
                                    active_file:239 inactive_file:308 isolated_file:32
                                    unevictable:0 dirty:0 writeback:0
                                    slab_reclaimable:4988 slab_unreclaimable:5648
                                    mapped:442 shmem:179 pagetables:1405 bounce:0
                                    free:3532 free_pcp:31 free_cma:215
Jul 31 11:01:27 influxPi4B kernel: Node 0 active_anon:826700kB inactive_anon:972972kB active_file:956kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1768>
Jul 31 11:01:27 influxPi4B kernel: DMA free:8144kB min:6648kB low:7524kB high:8400kB reserved_highatomic:0KB active_anon:529016kB inactive_anon:332500kB active_file:200kB inactive_file:204kB u>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 984 984 984
Jul 31 11:01:27 influxPi4B kernel: DMA32 free:5984kB min:10340kB low:11344kB high:12348kB reserved_highatomic:0KB active_anon:297424kB inactive_anon:640448kB active_file:876kB inactive_file:14>
Jul 31 11:01:27 influxPi4B kernel: lowmem_reserve[]: 0 0 0 0
Jul 31 11:01:27 influxPi4B kernel: DMA: 358*4kB (UMEC) 241*8kB (UMEC) 129*16kB (UMEC) 59*32kB (UEC) 16*64kB (UEC) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8336kB
Jul 31 11:01:27 influxPi4B kernel: DMA32: 364*4kB (UM) 161*8kB (UME) 174*16kB (UE) 24*32kB (UME) 2*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6424kB
Jul 31 11:01:27 influxPi4B kernel: 725 total pagecache pages
Jul 31 11:01:27 influxPi4B kernel: 0 pages in swap cache
Jul 31 11:01:27 influxPi4B kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 31 11:01:27 influxPi4B kernel: Free swap  = 0kB
Jul 31 11:01:27 influxPi4B kernel: Total swap = 0kB
Jul 31 11:01:27 influxPi4B kernel: 504832 pages RAM
Jul 31 11:01:27 influxPi4B kernel: 0 pages HighMem/MovableOnly
Jul 31 11:01:27 influxPi4B kernel: 31338 pages reserved
Jul 31 11:01:27 influxPi4B kernel: 65536 pages cma reserved
Jul 31 11:01:27 influxPi4B kernel: Tasks state (memory values in pages):
Jul 31 11:01:27 influxPi4B kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul 31 11:01:27 influxPi4B kernel: [    225]     0   225     3952      190    61440        0             0 systemd-userdbd
Jul 31 11:01:27 influxPi4B kernel: [    229]     0   229     4366      403    73728        0         -1000 systemd-udevd
Jul 31 11:01:27 influxPi4B kernel: [    233]   981   233     5397      264    73728        0             0 systemd-network
Jul 31 11:01:27 influxPi4B kernel: [    311]   980   311     4973      481    69632        0             0 systemd-resolve
Jul 31 11:01:27 influxPi4B kernel: [    312]   979   312    22392      205    65536        0             0 systemd-timesyn
Jul 31 11:01:27 influxPi4B kernel: [    315]    81   315     3554      337    57344        0          -900 dbus-daemon
Jul 31 11:01:27 influxPi4B kernel: [    316]     0   316     1880      134    49152        0             0 socat
Jul 31 11:01:27 influxPi4B kernel: [    318]     0   318     4067      225    57344        0             0 systemd-homed
Jul 31 11:01:27 influxPi4B kernel: [    320]     0   320     4088      245    65536        0             0 systemd-logind
Jul 31 11:01:27 influxPi4B kernel: [    322]     0   322    99907     5174   131072        0             0 python
Jul 31 11:01:27 influxPi4B kernel: [    323]     0   323     2403      198    49152        0         -1000 sshd
Jul 31 11:01:27 influxPi4B kernel: [    325]     0   325     2116      129    45056        0             0 agetty
Jul 31 11:01:27 influxPi4B kernel: [  17654]   976 17654  2059639   437575  4632576        0             0 influxd
Jul 31 11:01:27 influxPi4B kernel: [  17658]     0 17658     8115      227    69632        0          -250 systemd-journal
Jul 31 11:01:27 influxPi4B kernel: [  38555]     0 38555     4079      188    61440        0             0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [  38556]     0 38556     4079      187    65536        0             0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: [  38557]     0 38557     4079      188    57344        0             0 systemd-userwor
Jul 31 11:01:27 influxPi4B kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=influxd,pid=17654,uid=976
Jul 31 11:01:27 influxPi4B kernel: Out of memory: Killed process 17654 (influxd) total-vm:8238556kB, anon-rss:1749696kB, file-rss:604kB, shmem-rss:0kB, UID:976 pgtables:4524kB oom_score_adj:0
Jul 31 11:01:27 influxPi4B kernel: oom_reaper: reaped process 17654 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 31 11:01:27 influxPi4B systemd[1]: influxdb.service: Main process exited, code=killed, status=9/KILL

Environment:

# systemctl show influx.service | grep ^Limit
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1073741816
LimitNOFILESoft=1073741816
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=6081
LimitNPROCSoft=6081
LimitMEMLOCK=242425856
LimitMEMLOCKSoft=242425856
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=6081
LimitSIGPENDINGSoft=6081
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
# systemctl show influxdb.service | grep ^Memory
MemoryCurrent=[not set]
MemoryAvailable=infinity
MemoryAccounting=yes
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
MemoryDenyWriteExecute=no

Performance:
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

# Commands should be run when the bug is actively happening.
# Note: This command will run for ~30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=30s"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz` and `iostat.txt` output files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/memory area/performance area/storage area/2.x OSS 2.0 related issues and PRs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ProBackup-nl @danxmoran