kernel panics and servers restart
Forum
  1. Forums
  2. CloudLinux and Control Panels
  3. CloudLinux and cPanel
  1. Havri
  2. Tuesday, 22 May 2018
  3.  Subscribe via email
Hello,

We started getting for about 2-3 weeks some random restarts on some of our servers. The vmcore-dmesg.txt from abrt in /var/crash outputs the following:


[4650602.942215] OOM kill timeout: 892429840 (u3E,\x81f\xb8T\x83\xc4\xe1uJO\xa4\xb1>\x92\x84\x7fO\xeb\xee\xa6\xe1|\xcf\x1e\x9d\x16\xcc\x0e\xaad\x1e\x9e\xab\xb0t\xb2
O\xb9A\x19\xad\x8el\x95^L\x0eY\x91W\xee\xbcMO!\x04i\xe1\x1b\xd2~\xbc\x1f\xa1\xe5\xb4\x9d\xab\xa4f\x98=)\x10\x83\xc4q\xc8\x1f\x9f^\xa72c\xe2\xb1\xd4\x8a\xb3G+\xc6\xa0\xdf\xca\x8dp\x1d\x97\xd4\x14\x8f\xf1\xbf\xe3\xdbRU\xde\x9dx\x13\x18\xa7\xcf\xa7\xf7\xab\x87%\x8e\xadY#\x89\xcdS\x07\xa6\x9d\x9c\x86\x96h\x01(\xa9k\x05\xb9\xb8b?\xd8\xfbBb\xf0\xa6\x04yt\xabV\xb4#\x8dzj\xc2\xd4SE[S\x8f\xc9F\x8bO[I\x1dl\xb24p\x95\xa6\x9a\x99KC\x10\x95\xf8Df\xb0y)
[4650602.945748] Call Trace:
[4650602.945777] general protection fault: 0000 [#1] SMP
[4650602.946829] Modules linked in: xt_time sctp_diag sctp libcrc32c dccp_diag dccp udp_diag unix_diag af_packet_diag netlink_diag binfmt_misc xt_set ip_set_hash_net ip_set nfnetlink kcare(OE) tcp_diag inet_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_conntrack nf_conntrack_ftp xt_LOG xt_limit ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 xt_REDIRECT nf_nat_redirect iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_owner xt_multiport kmodlve(O) vzdev loop vfat fat intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel
[4650602.953476] lrw gf128mul glue_helper ablk_helper cryptd sb_edac pcspkr edac_core i2c_i801 mei_me lpc_ich mei sg shpchp wmi ipmi_devintf ipmi_si ipmi_msghandler acpi_pad acpi_power_meter ip_tables ext4 mbcache jbd2 raid10 raid1 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper ttm igb ahci drm libahci ptp pps_core dca libata i2c_algo_bit i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
[4650602.958022] CPU: 0 PID: 5561 Comm: lsphp ve: 0 Tainted: G OE ------------ 3.10.0-714.10.2.lve1.5.12.el7.x86_64 #1 29.2
[4650602.960293] Hardware name: Supermicro Super Server/X10DRW-i, BIOS 2.0b 04/13/2017
[4650602.961433] task: ffff88015f772fd0 ti: ffff8803f7a28000 task.ti: ffff8803f7a28000
[4650602.962642] RIP: 0010:[<ffffffff8102e52f>] [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[4650602.963807] RSP: 0000:ffff8803f7a2bcb0 EFLAGS: 00010206
[4650602.964959] RAX: ab9db4e5a11fbc7e RBX: ffffffff816ba7a0 RCX: 0000000000000000
[4650602.966070] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8811c99b3fc0
[4650602.967198] RBP: ffff8803f7a2bd20 R08: ffff88103fa00000 R09: ffffffff81917c27
[4650602.968290] R10: 0000000000000000 R11: ffff8803f7a2ba1e R12: ffff8811c99b3fc0
[4650602.969364] R13: ffff8803f7a2bd20 R14: ab9db4e5a11fbc7e R15: 0000000000000000
[4650602.970460] FS: 00007f52e5d42880(0000) GS:ffff88103fa00000(0000) knlGS:0000000000000000
[4650602.971569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4650602.972703] CR2: 0000000001259a68 CR3: 00000003c7154000 CR4: 00000000003607f0
[4650602.973771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4650602.974904] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[4650602.975973] Call Trace:
[4650602.977024] [<ffffffff8108bcd6>] ? vprintk_default+0x36/0x50
[4650602.978118] [<ffffffff8102f6cd>] show_trace_log_lvl+0x4d/0x60
[4650602.979223] [<ffffffff8102f734>] show_stack+0x34/0x70
[4650602.980376] [<ffffffff81198221>] oom_trylock+0x1d1/0x1e0
[4650602.981394] [<ffffffff8120bed1>] mem_cgroup_oom_synchronize+0xe1/0x4e0
[4650602.982426] [<ffffffff811c849c>] ? handle_mm_fault+0x9ac/0x14c0
[4650602.983495] [<ffffffff81232dc4>] ? dput+0x24/0x180
[4650602.984529] [<ffffffff81198fe3>] pagefault_out_of_memory+0x13/0x50
[4650602.985566] [<ffffffff816965e5>] mm_fault_error+0x68/0x12b
[4650602.986593] [<ffffffff816a8df5>] __do_page_fault+0x395/0x450
[4650602.987616] [<ffffffff816a8ee5>] do_page_fault+0x35/0x90
[4650602.988662] [<ffffffff816a4cf8>] page_fault+0x28/0x30
[4650602.989668] Code: 8b b7 e0 06 00 00 4d 85 ed 0f 85 bf fe ff ff 65 48 8b 04 25 00 0e 01 00 49 89 ed 48 39 c7 0f 84 aa fe ff ff 48 8b 87 e0 06 00 00 <4c> 8b 28 e9 9b fe ff ff 66 0f 1f 84 00 00 00 00 00 8b 45 9c 0f
[4650602.991825] RIP [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[4650602.993004] RSP <ffff8803f7a2bcb0>




[2744413.666430] OOM kill timeout: -1329536358 (T\xe5q\x18\xe4*\x12v\x83\xb4aB\x81SS\xff)
[2744413.671483] Call Trace:
[2744413.671514] general protection fault: 0000 [#1] SMP
[2744413.676273] Modules linked in: xt_time tcp_diag inet_diag xt_set kcare(OE) ip_set_hash_net ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_conntrack nf_conntrack_ftp xt_LOG xt_limit ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 xt_REDIRECT nf_nat_redirect iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat kmodlve(O) nf_conntrack vzdev xt_owner xt_multiport loop intel_powerclamp vfat coretemp fat intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core i2c_i801 mei_me lpc_ich
[2744413.706243] mei sg shpchp wmi ipmi_devintf ipmi_si nfit ipmi_msghandler libnvdimm acpi_power_meter acpi_pad ip_tables ext4 mbcache jbd2 raid10 raid1 sd_mod crc_t10dif crct10dif_generic ast crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper ttm igb ahci ptp libahci drm pps_core dca libata i2c_algo_bit i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
[2744413.728430] CPU: 1 PID: 12206 Comm: lsphp ve: 0 Tainted: G OE ------------ 3.10.0-714.10.2.lve1.5.15.el7.x86_64 #1 29.2
[2744413.740034] Hardware name: Supermicro Super Server/X10DRW-i, BIOS 3.0a 02/08/2018
[2744413.745907] task: ffff8808b6ac9fe0 ti: ffff88019f74c000 task.ti: ffff88019f74c000
[2744413.751633] RIP: 0010:[<ffffffff8102e52f>] [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[2744413.757528] RSP: 0018:ffff88019f74fcb0 EFLAGS: 00010283
[2744413.763301] RAX: 3afdd0db33704016 RBX: ffffffff816bc7a0 RCX: 0000000000000001
[2744413.769209] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff880141f5ef90
[2744413.775101] RBP: ffff88019f74fd20 R08: ffff88103fa40000 R09: ffffffff81919c27
[2744413.780890] R10: 0000000000000000 R11: ffff88019f74fa1e R12: ffff880141f5ef90
[2744413.786521] R13: ffff88019f74fd20 R14: 3afdd0db33704016 R15: 0000000000000000
[2744413.792242] FS: 00007f4a984b9880(0000) GS:ffff88103fa40000(0000) knlGS:0000000000000000
[2744413.798116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2744413.803973] CR2: 00007f4a6674e000 CR3: 000000019f74a000 CR4: 00000000003607e0
[2744413.809672] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2744413.815334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2744413.820991] Call Trace:
[2744413.826480] [<ffffffff8108bcd6>] ? vprintk_default+0x36/0x50
[2744413.832111] [<ffffffff8102f6cd>] show_trace_log_lvl+0x4d/0x60
[2744413.837843] [<ffffffff8102f734>] show_stack+0x34/0x70
[2744413.843312] [<ffffffff81198241>] oom_trylock+0x1d1/0x1e0
[2744413.848815] [<ffffffff8120bef1>] mem_cgroup_oom_synchronize+0xe1/0x4e0
[2744413.854262] [<ffffffff811a34af>] ? put_page+0x4f/0x60
[2744413.859792] [<ffffffff811c7fe7>] ? handle_mm_fault+0x4d7/0x14c0
[2744413.865205] [<ffffffff81232db4>] ? dput+0x24/0x180
[2744413.870642] [<ffffffff81199003>] pagefault_out_of_memory+0x13/0x50
[2744413.876029] [<ffffffff81696625>] mm_fault_error+0x68/0x12b
[2744413.881406] [<ffffffff816a99f5>] __do_page_fault+0x395/0x450
[2744413.886809] [<ffffffff816a9ae5>] do_page_fault+0x35/0x90
[2744413.892055] [<ffffffff816a58f8>] page_fault+0x28/0x30
[2744413.897297] Code: 8b b7 e0 06 00 00 4d 85 ed 0f 85 bf fe ff ff 65 48 8b 04 25 00 0e 01 00 49 89 ed 48 39 c7 0f 84 aa fe ff ff 48 8b 87 e0 06 00 00 <4c> 8b 28 e9 9b fe ff ff 66 0f 1f 84 00 00 00 00 00 8b 45 9c 0f
[2744413.908367] RIP [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[2744413.913832] RSP <ffff88019f74fcb0>




[9532455.482317] OOM kill timeout: -1504 (\x16)
[9532455.483196] Call Trace:
[9532455.483243] BUG: unable to handle kernel paging request at 00007f6aa9f966f8
[9532455.483945] IP: [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[9532455.484602] PGD 8000000143b93067 PUD 143b90067 PMD 0
[9532455.485295] Oops: 0000 [#1] SMP
[9532455.486071] Modules linked in: ip6table_mangle ip6table_raw iptable_raw udp_diag binfmt_misc tcp_diag inet_diag kcare(OE) xt_set ip_set_hash_net ip_set nfnetlink ip6table_nat nf_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter ip6_tables nf_nat_ftp xt_conntrack iptable_mangle nf_conntrack_ftp xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 xt_REDIRECT nf_nat_redirect iptable_filter iptable_nat nf_conntrack_ipv4 kmodlve(O) nf_defrag_ipv4 vzdev nf_nat_ipv4 nf_nat nf_conntrack xt_owner xt_multiport loop intel_powerclamp vfat coretemp fat intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core i2c_i801 mei_me
[9532455.490862] mei lpc_ich sg ipmi_devintf shpchp wmi ipmi_si ipmi_msghandler acpi_power_meter acpi_pad ip_tables ext4 mbcache jbd2 raid1 raid10 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper ahci ttm igb libahci drm ptp libata pps_core dca i2c_algo_bit i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: fixup_kcare]
[9532455.494424] CPU: 11 PID: 29493 Comm: lsphp ve: 0 Tainted: G OE ------------ 3.10.0-714.10.2.lve1.5.9.el7.x86_64 #1 29.2
[9532455.496300] Hardware name: Supermicro Super Server/X10DRW-i, BIOS 2.0b 04/13/2017
[9532455.497239] task: ffff880267bf0ff0 ti: ffff880052c18000 task.ti: ffff880052c18000
[9532455.498181] RIP: 0010:[<ffffffff8102e52f>] [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[9532455.499125] RSP: 0000:ffff880052c1bcb0 EFLAGS: 00010287
[9532455.500084] RAX: 00007f6aa9f966f8 RBX: ffffffff816ba7a0 RCX: 000000000000000b
[9532455.501061] RDX: 000000000000000b RSI: 0000000000000000 RDI: ffff88014bf36f90
[9532455.502056] RBP: ffff880052c1bd20 R08: ffff88105f2c0000 R09: ffffffff81917bd7
[9532455.503069] R10: 0000000000000000 R11: ffff880052c1ba1e R12: ffff88014bf36f90
[9532455.504085] R13: ffff880052c1bd20 R14: 00007f6aa9f966f8 R15: 0000000000000000
[9532455.505118] FS: 00007f6abe758880(0000) GS:ffff88105f2c0000(0000) knlGS:0000000000000000
[9532455.506162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[9532455.507182] CR2: 00007f6aa9f966f8 CR3: 0000000007d28000 CR4: 00000000003607e0
[9532455.508224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[9532455.509193] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[9532455.510193] Call Trace:
[9532455.511205] [<ffffffff8108bcd6>] ? vprintk_default+0x36/0x50
[9532455.512244] [<ffffffff8102f6cd>] show_trace_log_lvl+0x4d/0x60
[9532455.513282] [<ffffffff8102f734>] show_stack+0x34/0x70
[9532455.514321] [<ffffffff811981e1>] oom_trylock+0x1d1/0x1e0
[9532455.515352] [<ffffffff8120be91>] mem_cgroup_oom_synchronize+0xe1/0x4e0
[9532455.516375] [<ffffffff811c8480>] ? handle_mm_fault+0x9d0/0x14c0
[9532455.517376] [<ffffffff811cf3c5>] ? do_mmap_pgoff+0x305/0x3c0
[9532455.518354] [<ffffffff81198fa3>] pagefault_out_of_memory+0x13/0x50
[9532455.519375] [<ffffffff81696585>] mm_fault_error+0x68/0x12b
[9532455.520376] [<ffffffff816a8d75>] __do_page_fault+0x395/0x450
[9532455.521327] [<ffffffff816a8e65>] do_page_fault+0x35/0x90
[9532455.522251] [<ffffffff816a4c78>] page_fault+0x28/0x30
[9532455.523179] Code: 8b b7 e0 06 00 00 4d 85 ed 0f 85 bf fe ff ff 65 48 8b 04 25 00 0e 01 00 49 89 ed 48 39 c7 0f 84 aa fe ff ff 48 8b 87 e0 06 00 00 <4c> 8b 28 e9 9b fe ff ff 66 0f 1f 84 00 00 00 00 00 8b 45 9c 0f
[9532455.525197] RIP [<ffffffff8102e52f>] dump_trace+0x1df/0x2d0
[9532455.526193] RSP <ffff880052c1bcb0>
[9532455.527185] CR2: 00007f6aa9f966f8



These examples are from 3 of our servers. We haven't modified anything substantial on the servers. I have to mention that all three servers run kernel-care. Maybe after certain updates, the in-memory kernel-care kernel version had some bugs.


All are running:
- cPanel 11.68.0.39 or newer
- Litespeed 5.2.5 or newer
- kernel-care


Let me know if anything similar has been reported to Cloudlinux.

Also, if you need any other info, let me know.

Thanks.
Rate this post:
  1. 22.05.2018 15:05:18
  2. # 1
Igor Ghertesco Accepted Answer
Posts: 154
Joined: 07.08.2015
0
Votes
Undo
Hello,

This is a known issue, the fix is almost ready. The internal task ID is CLKRN-259, it will be mentioned in our blog: https://www.cloudlinux.com/cloudlinux-os-blog

Also, if you face the similar issue next time, please submit a ticket to us immediately: https://cloudlinux.zendesk.com/hc/en-us/requests/new
  1. 22.05.2018 17:05:54
  2. # 2
Havri Accepted Answer
Posts: 26
Joined: 30.07.2015
0
Votes
Undo
Hello,

Ok. Thanks a lot. I'll keep an eye on your changelog

Regards.
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.