Kernel OOM killer issue

Forum

Sign In Here

Sign in to your account

If you are new here please register for an account

Username:

Password:

Remember Me

Forgot Password?

Resolved Kernel OOM killer issue

Ajdin Lupcević
Tuesday, 03 January 2017
Subscribe via email

Hi,

we have issue with cloudlinux kernel OOM killer. Server was freez and we need to hardcore reset it. before this happend we have errors and it's continue after server is restarted:

Jan 3 20:26:52 enigma kernel: [ 3000.177353] 156765 (php) invoked oom-killer in ub 1201 generation 56 gfp 0x200d2
Jan 3 20:26:52 enigma kernel: [ 3000.177357] UB-1201-Mem-Info:
Jan 3 20:26:52 enigma kernel: [ 3000.177363] Node 0 Normal prio:41 portion:9491 scan:0 a_anon:114074 125500ms i_anon:155028 2957865ms a_file:17 5500ms i_file:10 5500ms unevictable:0 reclaim_stat: 24760649 14024226 645878 442457
Jan 3 20:26:52 enigma kernel: [ 3000.177370] Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177378] Node 1 DMA prio:21 portion:5 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177384] Node 1 DMA/shadow prio:21 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177391] Node 1 DMA32 prio:21 portion:1269 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177397] Node 1 DMA32/shadow prio:21 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177405] Node 1 Normal prio:41 portion:8205 scan:64 a_anon:116589 185495ms i_anon:137909 2957865ms a_file:9 5495ms i_file:10 5495ms unevictable:0 reclaim_stat: 16843130 9503293 406270 288830
Jan 3 20:26:52 enigma kernel: [ 3000.177412] Node 1 Normal/shadow prio:41 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan 3 20:26:52 enigma kernel: [ 3000.177418] Total 523646 anon:523600 file:46 a_anon:230663 i_anon:292937 a_file:26 i_file:20 unevictable:0
Jan 3 20:26:52 enigma kernel: [ 3000.177422] RAM: 524288 / 524288 [562] SWAP: 0 / 0 [562] KMEM: 0 / inf [0] DCSZ: 0 / inf [0] OOMG: 519182 / inf [58] Dirty 0 Wback 0 Dche 0 Prnd 0
Jan 3 20:26:52 enigma kernel: [ 3000.178190] Out of memory in UB 1201: OOM killed process 152583 (php) score 0 vm:399468kB, rss:42000kB, swap:0kB
Jan 3 20:26:52 enigma kernel: [ 3000.181892] oom-killer in ub 1201 generation 56 ends: task died
Jan 3 20:26:52 enigma kernel: [ 3000.181897] RAM: 516279 / 524288 [562] SWAP: 0 / 0 [562] KMEM: 0 / inf [0] DCSZ: 0 / inf [0] OOMG: 519182 / inf [59] Dirty 0 Wback 0 Dche 0 Prnd 0
Jan 3 20:26:56 enigma kernel: [ 3003.748802] Out of memory in UB 1201: OOM killed process 152613 (php) score 0 vm:399656kB, rss:42244kB, swap:0kB
Jan 3 20:26:57 enigma kernel: [ 3004.459685] Out of memory in UB 1201: OOM killed process 152639 (php) score 0 vm:399616kB, rss:42152kB, swap:0kB
Jan 3 20:26:57 enigma kernel: [ 3005.179026] Out of memory in UB 1201: OOM killed process 152686 (php) score 0 vm:399748kB, rss:42208kB, swap:0kB
Jan 3 20:26:58 enigma kernel: [ 3005.657632] Out of memory in UB 1201: OOM killed process 152792 (php) score 0 vm:399620kB, rss:41748kB, swap:0kB
Jan 3 20:26:58 enigma kernel: [ 3006.393931] Out of memory in UB 1201: OOM killed process 157740 (httpd) score 0 vm:209616kB, rss:128260kB, swap:0kB
Jan 3 20:26:59 enigma kernel: [ 3006.676679] Out of memory in UB 1201: OOM killed process 152805 (php) score 0 vm:399744kB, rss:42144kB, swap:0kB

Jan 3 20:26:59 enigma kernel: [ 3006.952813] Out of memory in UB 1201: OOM killed process 157773 (httpd) score 0 vm:209512kB, rss:128136kB, swap:0kB
Jan 3 20:26:59 enigma kernel: [ 3007.010944] Out of memory in UB 1201: OOM killed process 157778 (httpd) score 0 vm:209656kB, rss:128260kB, swap:0kB
Jan 3 20:26:59 enigma kernel: [ 3007.019966] Out of memory in UB 1201: OOM killed process 157782 (httpd) score 0 vm:209500kB, rss:128224kB, swap:0kB
Jan 3 20:26:59 enigma kernel: [ 3007.166487] Out of memory in UB 1201: OOM killed process 152851 (php) score 0 vm:399748kB, rss:41788kB, swap:0kB

Jan 3 20:27:00 enigma kernel: [ 3008.219167] Out of memory in UB 1201: OOM killed process 152858 (php) score 0 vm:399620kB, rss:41672kB, swap:0kB
Jan 3 20:27:01 enigma kernel: [ 3008.546078] Out of memory in UB 1201: OOM killed process 152913 (php) score 0 vm:399744kB, rss:41780kB, swap:0kB
Jan 3 20:27:01 enigma kernel: [ 3008.857261] Out of memory in UB 1201: OOM killed process 152928 (php) score 0 vm:399620kB, rss:41788kB, swap:0kB

Jan 3 20:27:02 enigma kernel: [ 3009.436137] Out of memory in UB 1201: OOM killed process 153007 (php) score 0 vm:399596kB, rss:41744kB, swap:0kB
Jan 3 20:27:02 enigma kernel: [ 3010.009724] Out of memory in UB 1201: OOM killed process 153076 (php) score 0 vm:399620kB, rss:41824kB, swap:0kB

Jan 3 20:27:03 enigma kernel: [ 3010.630382] Out of memory in UB 1201: OOM killed process 153077 (php) score 0 vm:399620kB, rss:41708kB, swap:0kB
Jan 3 20:27:03 enigma kernel: [ 3010.752142] Out of memory in UB 1201: OOM killed process 153080 (php) score 0 vm:399616kB, rss:41812kB, swap:0kB
Jan 3 20:27:03 enigma kernel: [ 3011.170634] Out of memory in UB 1201: OOM killed process 158051 (httpd) score 0 vm:209636kB, rss:128424kB, swap:0kB
Jan 3 20:27:03 enigma kernel: [ 3011.197717] Out of memory in UB 1201: OOM killed process 153082 (php) score 0 vm:399748kB, rss:41924kB, swap:0kB
Jan 3 20:27:04 enigma kernel: [ 3011.575533] Out of memory in UB 1201: OOM killed process 153083 (php) score 0 vm:399620kB, rss:41736kB, swap:0kB
Jan 3 20:27:04 enigma kernel: [ 3012.039548] OOM kill in rage task 156672 (php) score 20 in ub 1201
Jan 3 20:27:04 enigma kernel: [ 3012.039553] OOM killer in rage, 1 tasks killed in ub 1201
Jan 3 20:27:04 enigma kernel: [ 3012.039725] OOM killer in rage, 1 tasks killed
Jan 3 20:27:04 enigma kernel: [ 3012.039858] Out of memory in UB 1201: OOM killed process 153085 (php) score 0 vm:399620kB, rss:41796kB, swap:0kB
Jan 3 20:27:05 enigma kernel: [ 3012.708630] OOM kill in rage task 158137 (httpd) score 61 in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.711706] OOM killer in rage, 1 tasks killed in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.711834] OOM killer in rage, 1 tasks killed
Jan 3 20:27:05 enigma kernel: [ 3012.713262] Out of memory in UB 1201: OOM killed process 158137 (httpd) score 0 vm:209560kB, rss:128432kB, swap:0kB
Jan 3 20:27:05 enigma kernel: [ 3012.774450] OOM kill in rage task 156674 (php) score 20 in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.774512] OOM kill in rage task 156462 (php) score 20 in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.774526] OOM kill in rage task 156444 (php) score 20 in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.774564] OOM kill in rage task 156370 (php) score 20 in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.774567] OOM killer in rage, 4 tasks killed in ub 1201
Jan 3 20:27:05 enigma kernel: [ 3012.774685] OOM killer in rage, 4 tasks killed
Jan 3 20:27:05 enigma kernel: [ 3012.774802] Out of memory in UB 1201: OOM killed process 153154 (php) score 0 vm:399744kB, rss:42352kB, swap:0kB
Jan 3 20:27:06 enigma kernel: [ 3013.921334] Out of memory in UB 1201: OOM killed process 153162 (php) score 0 vm:399748kB, rss:41900kB, swap:0kB
Jan 3 20:27:06 enigma kernel: [ 3014.130608] Out of memory in UB 1201: OOM killed process 153333 (php) score 0 vm:399620kB, rss:41784kB, swap:0kB
Jan 3 20:27:08 enigma kernel: [ 3015.482295] Out of memory in UB 1201: OOM killed process 153339 (php) score 0 vm:399616kB, rss:41808kB, swap:0kB
Jan 3 20:27:08 enigma kernel: [ 3015.767738] Out of memory in UB 1201: OOM killed process 153369 (php) score 0 vm:399616kB, rss:41808kB, swap:0kB
Jan 3 20:27:08 enigma kernel: [ 3016.344224] Out of memory in UB 1201: OOM killed process 153419 (php) score 0 vm:399744kB, rss:41928kB, swap:0kB

Jan 3 20:27:10 enigma kernel: [ 3018.081365] Out of memory in UB 1201: OOM killed process 153479 (php) score 0 vm:399616kB, rss:41828kB, swap:0kB

Jan 3 20:27:11 enigma kernel: [ 3019.346131] Out of memory in UB 1201: OOM killed process 153552 (php) score 0 vm:399764kB, rss:42016kB, swap:0kB
Jan 3 20:27:11 enigma pure-ftpd: ([email protected]) [INFO] Logout.
Jan 3 20:27:12 enigma kernel: [ 3020.002983] Out of memory in UB 1201: OOM killed process 153611 (php) score 0 vm:399748kB, rss:41964kB, swap:0kB
Jan 3 20:27:13 enigma kernel: [ 3020.597724] Out of memory in UB 1201: OOM killed process 158491 (httpd) score 0 vm:229456kB, rss:148100kB, swap:0kB

Does anyone have idea how to solve this?

Best regards.

Rate this post:

Responses (2)

14.01.2019 08:01:51
# 1

Hogne Vevle Accepted Answer

Posts: 0
Joined: 04.06.2020

Votes

Undo

@bogdan
After disabling oom-killer as described, the log entries no longer reveal which user the killed process belongs to.

For example:

Jan 14 09:49:12 xxx kernel: [8164156.726262] out_of_memory_in_ub: task php-cgi will be killed due to oom

How can we determine the LVE ID that owns the offending process?

15.01.2019 10:01:58
# 2