Kernel OOM killer issue
Forum
Toggle Submenu
  1. Forums
  2. Recent
  3. Tags
Add a reply View Replies (2)
  1. Forums
  2. General
  3. General Discussion

Resolved Kernel OOM killer issue

  1. Ajdin Lupcević
    Ajdin Lupcević
  2. Tuesday, 03 January 2017
  3.  Subscribe via email
Hi,

we have issue with cloudlinux kernel OOM killer. Server was freez and we need to hardcore reset it. before this happend we have errors and it's continue after server is restarted:


Jan  3 20:26:52 enigma kernel: [ 3000.177353] 156765 (php) invoked oom-killer in ub 1201 generation 56 gfp 0x200d2
Jan  3 20:26:52 enigma kernel: [ 3000.177357] UB-1201-Mem-Info:
Jan  3 20:26:52 enigma kernel: [ 3000.177363]  Node 0 Normal prio:41 portion:9491 scan:0 a_anon:114074 125500ms i_anon:155028 2957865ms a_file:17 5500ms i_file:10 5500ms unevictable:0 reclaim_stat: 24760649 14024226 645878 442457
Jan  3 20:26:52 enigma kernel: [ 3000.177370]  Node 0 Normal/shadow prio:41 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177378]  Node 1 DMA prio:21 portion:5 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177384]  Node 1 DMA/shadow prio:21 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177391]  Node 1 DMA32 prio:21 portion:1269 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177397]  Node 1 DMA32/shadow prio:21 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177405]  Node 1 Normal prio:41 portion:8205 scan:64 a_anon:116589 185495ms i_anon:137909 2957865ms a_file:9 5495ms i_file:10 5495ms unevictable:0 reclaim_stat: 16843130 9503293 406270 288830
Jan  3 20:26:52 enigma kernel: [ 3000.177412]  Node 1 Normal/shadow prio:41 portion:0 scan:0 a_anon:0 2957865ms i_anon:0 2957865ms a_file:0 2957865ms i_file:0 2957865ms unevictable:0 reclaim_stat: 0 0 0 0
Jan  3 20:26:52 enigma kernel: [ 3000.177418] Total 523646 anon:523600 file:46 a_anon:230663 i_anon:292937 a_file:26 i_file:20 unevictable:0
Jan  3 20:26:52 enigma kernel: [ 3000.177422] RAM: 524288 / 524288 [562] SWAP: 0 / 0 [562] KMEM: 0 / inf [0] DCSZ: 0 / inf [0] OOMG: 519182 / inf [58] Dirty 0 Wback 0 Dche 0 Prnd 0
Jan  3 20:26:52 enigma kernel: [ 3000.178190] Out of memory in UB 1201: OOM killed process 152583 (php) score 0 vm:399468kB, rss:42000kB, swap:0kB
Jan  3 20:26:52 enigma kernel: [ 3000.181892] oom-killer in ub 1201 generation 56 ends: task died
Jan  3 20:26:52 enigma kernel: [ 3000.181897] RAM: 516279 / 524288 [562] SWAP: 0 / 0 [562] KMEM: 0 / inf [0] DCSZ: 0 / inf [0] OOMG: 519182 / inf [59] Dirty 0 Wback 0 Dche 0 Prnd 0
Jan  3 20:26:56 enigma kernel: [ 3003.748802] Out of memory in UB 1201: OOM killed process 152613 (php) score 0 vm:399656kB, rss:42244kB, swap:0kB
Jan  3 20:26:57 enigma kernel: [ 3004.459685] Out of memory in UB 1201: OOM killed process 152639 (php) score 0 vm:399616kB, rss:42152kB, swap:0kB
Jan  3 20:26:57 enigma kernel: [ 3005.179026] Out of memory in UB 1201: OOM killed process 152686 (php) score 0 vm:399748kB, rss:42208kB, swap:0kB
Jan  3 20:26:58 enigma kernel: [ 3005.657632] Out of memory in UB 1201: OOM killed process 152792 (php) score 0 vm:399620kB, rss:41748kB, swap:0kB
Jan  3 20:26:58 enigma kernel: [ 3006.393931] Out of memory in UB 1201: OOM killed process 157740 (httpd) score 0 vm:209616kB, rss:128260kB, swap:0kB
Jan  3 20:26:59 enigma kernel: [ 3006.676679] Out of memory in UB 1201: OOM killed process 152805 (php) score 0 vm:399744kB, rss:42144kB, swap:0kB

Jan  3 20:26:59 enigma kernel: [ 3006.952813] Out of memory in UB 1201: OOM killed process 157773 (httpd) score 0 vm:209512kB, rss:128136kB, swap:0kB
Jan  3 20:26:59 enigma kernel: [ 3007.010944] Out of memory in UB 1201: OOM killed process 157778 (httpd) score 0 vm:209656kB, rss:128260kB, swap:0kB
Jan  3 20:26:59 enigma kernel: [ 3007.019966] Out of memory in UB 1201: OOM killed process 157782 (httpd) score 0 vm:209500kB, rss:128224kB, swap:0kB
Jan  3 20:26:59 enigma kernel: [ 3007.166487] Out of memory in UB 1201: OOM killed process 152851 (php) score 0 vm:399748kB, rss:41788kB, swap:0kB

Jan  3 20:27:00 enigma kernel: [ 3008.219167] Out of memory in UB 1201: OOM killed process 152858 (php) score 0 vm:399620kB, rss:41672kB, swap:0kB
Jan  3 20:27:01 enigma kernel: [ 3008.546078] Out of memory in UB 1201: OOM killed process 152913 (php) score 0 vm:399744kB, rss:41780kB, swap:0kB
Jan  3 20:27:01 enigma kernel: [ 3008.857261] Out of memory in UB 1201: OOM killed process 152928 (php) score 0 vm:399620kB, rss:41788kB, swap:0kB

Jan  3 20:27:02 enigma kernel: [ 3009.436137] Out of memory in UB 1201: OOM killed process 153007 (php) score 0 vm:399596kB, rss:41744kB, swap:0kB
Jan  3 20:27:02 enigma kernel: [ 3010.009724] Out of memory in UB 1201: OOM killed process 153076 (php) score 0 vm:399620kB, rss:41824kB, swap:0kB

Jan  3 20:27:03 enigma kernel: [ 3010.630382] Out of memory in UB 1201: OOM killed process 153077 (php) score 0 vm:399620kB, rss:41708kB, swap:0kB
Jan  3 20:27:03 enigma kernel: [ 3010.752142] Out of memory in UB 1201: OOM killed process 153080 (php) score 0 vm:399616kB, rss:41812kB, swap:0kB
Jan  3 20:27:03 enigma kernel: [ 3011.170634] Out of memory in UB 1201: OOM killed process 158051 (httpd) score 0 vm:209636kB, rss:128424kB, swap:0kB
Jan  3 20:27:03 enigma kernel: [ 3011.197717] Out of memory in UB 1201: OOM killed process 153082 (php) score 0 vm:399748kB, rss:41924kB, swap:0kB
Jan  3 20:27:04 enigma kernel: [ 3011.575533] Out of memory in UB 1201: OOM killed process 153083 (php) score 0 vm:399620kB, rss:41736kB, swap:0kB
Jan  3 20:27:04 enigma kernel: [ 3012.039548] OOM kill in rage task 156672 (php) score 20 in ub 1201
Jan  3 20:27:04 enigma kernel: [ 3012.039553] OOM killer in rage, 1 tasks killed in ub 1201
Jan  3 20:27:04 enigma kernel: [ 3012.039725] OOM killer in rage, 1 tasks killed
Jan  3 20:27:04 enigma kernel: [ 3012.039858] Out of memory in UB 1201: OOM killed process 153085 (php) score 0 vm:399620kB, rss:41796kB, swap:0kB
Jan  3 20:27:05 enigma kernel: [ 3012.708630] OOM kill in rage task 158137 (httpd) score 61 in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.711706] OOM killer in rage, 1 tasks killed in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.711834] OOM killer in rage, 1 tasks killed
Jan  3 20:27:05 enigma kernel: [ 3012.713262] Out of memory in UB 1201: OOM killed process 158137 (httpd) score 0 vm:209560kB, rss:128432kB, swap:0kB
Jan  3 20:27:05 enigma kernel: [ 3012.774450] OOM kill in rage task 156674 (php) score 20 in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.774512] OOM kill in rage task 156462 (php) score 20 in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.774526] OOM kill in rage task 156444 (php) score 20 in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.774564] OOM kill in rage task 156370 (php) score 20 in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.774567] OOM killer in rage, 4 tasks killed in ub 1201
Jan  3 20:27:05 enigma kernel: [ 3012.774685] OOM killer in rage, 4 tasks killed
Jan  3 20:27:05 enigma kernel: [ 3012.774802] Out of memory in UB 1201: OOM killed process 153154 (php) score 0 vm:399744kB, rss:42352kB, swap:0kB
Jan  3 20:27:06 enigma kernel: [ 3013.921334] Out of memory in UB 1201: OOM killed process 153162 (php) score 0 vm:399748kB, rss:41900kB, swap:0kB
Jan  3 20:27:06 enigma kernel: [ 3014.130608] Out of memory in UB 1201: OOM killed process 153333 (php) score 0 vm:399620kB, rss:41784kB, swap:0kB
Jan  3 20:27:08 enigma kernel: [ 3015.482295] Out of memory in UB 1201: OOM killed process 153339 (php) score 0 vm:399616kB, rss:41808kB, swap:0kB
Jan  3 20:27:08 enigma kernel: [ 3015.767738] Out of memory in UB 1201: OOM killed process 153369 (php) score 0 vm:399616kB, rss:41808kB, swap:0kB
Jan  3 20:27:08 enigma kernel: [ 3016.344224] Out of memory in UB 1201: OOM killed process 153419 (php) score 0 vm:399744kB, rss:41928kB, swap:0kB

Jan  3 20:27:10 enigma kernel: [ 3018.081365] Out of memory in UB 1201: OOM killed process 153479 (php) score 0 vm:399616kB, rss:41828kB, swap:0kB

Jan  3 20:27:11 enigma kernel: [ 3019.346131] Out of memory in UB 1201: OOM killed process 153552 (php) score 0 vm:399764kB, rss:42016kB, swap:0kB
Jan  3 20:27:11 enigma pure-ftpd: ([email protected]) [INFO] Logout.
Jan  3 20:27:12 enigma kernel: [ 3020.002983] Out of memory in UB 1201: OOM killed process 153611 (php) score 0 vm:399748kB, rss:41964kB, swap:0kB
Jan  3 20:27:13 enigma kernel: [ 3020.597724] Out of memory in UB 1201: OOM killed process 158491 (httpd) score 0 vm:229456kB, rss:148100kB, swap:0kB


Does anyone have idea how to solve this?

Best regards.
Rate this post:
0
Responses (2)
  1. 04.01.2017 06:01:03
  2. # 1
Bogdan Accepted Answer
Bogdan
Posts: 709
Joined: 26.06.2013
0
Votes
Undo
The messages shows that LVE ID 1201 used up all available memory and oom-killer was actually killing it\'s processes to keep in memory limit. The server could really be locked up if huge number of memory faults is done, however, we have a way to solve it: http://docs.cloudlinux.com/index.html?oom_killer_for_lve_processes.html

Disabling oom-killer by that documentation will use lightweight sigkill signal.
  1. 14.01.2019 08:01:51
  2. # 2
Hogne Vevle Accepted Answer
Guest
Posts: 0
Joined: 15.11.2019
0
Votes
Undo
@bogdan
After disabling oom-killer as described, the log entries no longer reveal which user the killed process belongs to.

For example:

Jan 14 09:49:12 xxx kernel: [8164156.726262] out_of_memory_in_ub: task php-cgi will be killed due to oom

How can we determine the LVE ID that owns the offending process?
  1. 15.01.2019 10:01:58
  2. # 3
Vladimir Accepted Answer
Vladimir
Posts: 108
Joined: 04.07.2017
0
Votes
Undo
Hello,

LVE statistics should still work for all users. You can check memory faults with lveinfo utility:
https://docs.cloudlinux.com/lveinfo.html
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Cite your Sources with a URL link to help readers verify facts or find more details.
Add Another Link
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.

General
General
  1. 503 posts
  2. 2 subcategories
CloudLinux and Control Panels
CloudLinux and Control Panels
  1. 1038 posts
  2. 5 subcategories
KernelCare
KernelCare
  1. 61 posts
  2. 1 subcategory
Imunify360
Imunify360
  1. 172 posts
  2. 1 subcategory

Stay in touch

CloudLinux Facebook Facebook 9K+ followers CloudLinux Twitter Twitter 10K+ followers CloudLinux YouTube Channel YouTube official channel CloudLinux on LinkedIn LinkedIn Follow us

Products
Success Stories
Partners
Support
About Us

© 2019 All rights reserved. CloudLinux Inc.
Terms of Use | Privacy Policy | Vulnerability Reporting | Legal 

enru

Call Sales at +1 (800) 231-7307

Email [email protected]