CloudLinux OS Blog

Intel CPU Bug - Meltdown and Spectre - KernelCare and CloudLinux

Intel CPU Bug - Meltdown and Spectre - KernelCare and CloudLinux
Update [May 29, 2018 12:25am PT]

Meltdown fixes for Ubuntu 16.04 are now on test.

To deploy them:
edit /etc/sysconfig/kcare/kcare.conf

Add:
PREFIX=test

Run:
kcarectl --update

TAGNAME: update-2018-05-29-test-1

ubuntu-xenial:
  CVE-2017-5754: Systems with microprocessors utilizing speculative execution and
    indirect branch prediction may allow unauthorized disclosure of information to
    an attacker with local user access via a side-channel analysis of the data cache.
  cvelist: [CVE-2017-5754]
  latest-version: 4.4.0-122.146

Update [Mar 30, 2018 12:25am PT]

Spectre fixes.
We are glad to announce that we've released Specre fixes to the TEST feed for the following distributions:

RHEL 6
Cent OS 6
CentOS 6 Plus
CloudLinux 6
OpenVZ
Proxmox VE 3.10
Proxmox VE 2.6

RHEL 7
CentOS 7

CentOS7 Plus
CloudLinux 7
CloudLinux 6 Hybrid

To deploy them:
edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

Note: It is a must to apply microcode changes. For the instructions follow the link: Microcode update form Intel

Update [Mar 20, 2018 05:25am PT]

Spectre fixes.
We've delayed our fixes for Specre vulnerability, as it is includes  Microcode update form Intel

That fixes was released by Intel last week (20180312 Release). Now we in process of carefull testing our patches with Specre fixes together with that Microcode change. 

ETA for public test release for el7 kernels - next week


Update [Mar 12, 2018 02:21am PT]

We are glad to announce that patches with Meltdown fixes are ready and released for: 

- CentOS 6 (latest-version: 2.6.32-696.20.1.el6)
- Cent OS 6 Plus (latest-version: 2.6.32-696.20.1.el6)
- RHEL 6 (latest-version: 2.6.32-696.20.1.el6)
- CloudLinux 6 (latest-version: 2.6.32-896.16.1.lve1.4.51.el6)
- OpenVZ (latest-version: 2.6.32-042stab127.2)
- Proxmox VE 3.10 ( latest-version: 3.10.0-22-pve_3.10.0-52)
- Cent OS 7  ( latest-version: 3.10.0-693.21.1.el7)
- Cent OS 7 Plus ( latest-version: 3.10.0-693.17.1.el7)
- RHEL 7 (latest-version: 3.10.0-693.21.1.el7)
- CloudLinux 7 (latest-version: 3.10.0-714.10.2.lve1.5.12.el7)
- CloudLinux 6 Hybrid (latest-version: 3.10.0-714.10.2.lve1.5.12.el6h)


Update [Feb 26, 2018 05:59am PT]

Fixed patches with Meltdown fixes for CentOS6 are going to be released till the end of that week. 
For CentOS 6 users: If you facing any problems with current patches, please, don't hesitate to contact our support team. We will get back to you shortly.

Patches with Spectre fixes (part 2) for el7 are ready, so we are in phase of testing that patches. Will be on Test server till the end of that week as well.

Next Spectre fixes (el6, Ubuntu, Debian) will be held on March.

Meltdown fixes for Ubuntu 16.04 and 14.04, Debian 7 are going to be released within next 2 weeks. Patches for Debian 8 and 9 will be held on March.

 

Update [Feb 20, 2018 08:20am PT]

For CentOS 6 users:


We still working on a proper fix for the crashed patch. Finally, we figured out what was the real source of the problem, so fix is coming.

ETA - Feb 22

Update [Feb 16, 2018 10:20am PT]

For CentOS 6 users:


Unfortunately, few clients running CentOS 6 distribution faced crashes after our last patch was applied. That is why we unroll that from a patch server until we do a proper fix. ETA - Feb 19

Update [Feb 16, 2018 00:20am PT]

We have released patches for Meltdown vulnerability for Xen PV hosts for CentOS7, CentOS7 Plus, CloudLinux 6 Hybrid, Proxmox VE 3.10, RHEL 7. Our internal tests, as well as tests with live customers over the past three days, didn't reveal any crashes.


Update [Feb 13, 2018 06:00am PT]

We are actively working on the fixes for others supported distros. Patches for servers that running Xen PV are going to be deployed to TEST today.
Patches with a fix for Meltdown for Ubuntu 16.04 and Ubuntu 14.04 to be added next week.


Update [Feb 10, 2018 11:00am PT]

We are happy to announce that we have released patches for Meltdown vulnerabilit for RedHat, CentOS, CloudLinux 6 as well as Virtuozzo/OpenVZ 2.6.32 kernels. Our internal tests, as well as tests with live customers, didn't reveal any crashes.

* Note: If you are running Xen PV -- they will not work



Update [Feb 6, 2018 1:57pm PT]

We have pushed Meltdown patches for RedHat, CentOS, CloudLinux 6 as well as Virtuozzo/OpenVZ 2.6.32 kernels to TEST channel

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

Update [Feb 2, 2018 11:27am PT]

Patches for CentOS 7 were released.


Update [Feb 2, 2018 8:27am PT]

Patches for CloudLinux 6 Hybrid and CloudLinux 7 were released.  Patches for Cent OS 7 will be released in 2 hours.

Update [Feb 1, 2018 8:25am PT]

Patches for RHEL 7, Cent-OS 7 Plus and Proxmox VE 3.10 were released. Other RHEL 7 related distributions will be released tomorrow.

Update [Jan 30, 2018 7:55am PT]

So far we had no negative reports and several positives. This is your last chance to test it before tomorrow's rollout of patches.

Please, try testing it on at least one machine.

Update [Jan 29, 2018 7:50am PT]

New patchset released to test for the same distributions as bellow with three more issues fixed. Hopefully this will be the last 'test' patchset. Please, help us with testing it. It seems like we removed all issues that might cause crash, and the two out of three issues were minor/didn't result in the crash.

Update [Jan 23, 2018 5:00am PT]

CloudLinux 7, CloudLinux 6h, CentOS/RHEL 7 and Proxmox VE 3.10 patches are ready/waiting for you in test channel. The issue from last round of tests had been resolved.

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

 

 

Update [Jan 22, 2018 5:40am PT]

We have received one crash report, that is related to NMI. We will not be releasing patches today, as we have to fix this issue first.

Thank you, everyone, who helped us test this patch. Please, don't install test patches for now, as we now know they have an issue.

Update [Jan 21, 2018 9:50am PT]

So far we have a few reports that patches are working as expected/well, no negative reports/crashes. We would like few more people to try/test, before we push it out.

Update [Jan 20, 2018 1:30pm PT]

CloudLinux 7, CloudLinux 6h, CentOS/RHEL 7 patches are ready/waiting for you in test channel.

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

 

Update [Jan 18, 2018 9:20pm PT]

CPU Hotplugging has been fixed. Xen PV doesn't work for now, everything else seems to work just fine. Our plan is to release patches to testing tomorrow morning for those who wants to test, followed by general release on Monday.

Note, this is for CL/CentOS/RHEL 7 only. CL/CentOS/RHEL/Virtuozzo 6 should follow shortly after. The rest of the distributions will be covered by the end of the next week.

* If you run on Xen PV -- you might want to disable KernelCare updates https://docs.kernelcare.com/index.html?settings.htm , as new version will not apply, but will remove old version.

PS: We do plan to add support for Xen PV as well, but only after we resolve issues with the rest of distributions.

 

Update [Jan 18, 2018 9:20pm PT]

We have been running extensive tests for CL/CentOS/RHEL 7 patchset, and so far the only issue we found was with CPU hotplugging. This issue is fixed, and we have started next set of tests. 

Update [Jan 17, 2018 1:10pm PT]

We are testing patches CloudLinux 7, CloudLinux 6h, CentOS/RHEL7 on hardware with NMI, and planning to make them available tomorrow on our 'test' channel. We will provide exact instructions on how you can test them.

Once we are done with this platforms, we will start adding all other distributions.

Update [Jan 16, 2018 8:10am PT]

We are getting very close with Meltdown patches. Our KernelCare team got the patchset working & stable. It is going through extensive code review now, and soon we will send it for ~12 hours of testing. If everything goes through well -- we might have patches available for those willing to test them either later today or tomorrow. Stay tunned.

Update [Jan 14, 2018 7:10am PT]

In the past few days we have made a bit more progress with KernelCare patch for Meltdown. We also hit three distinct issues that we are now working to overcome. We hope that once we done with them within next 1-2 days, we will not hit any other major issues.

Update [Jan 11, 2018 7:35am PT]

We are starting to roll out protection against CVE-2017-5753 via KernelCare. This is one of CVEs affecting Spectre. It is significantly smaller than KAISER changes that  are needed to protect against Meltdown.
The defence against Meltdown is still work in progress.

Update [Jan 11, 2018 6:35am PT]

We have resolved one particularly elusive bug yesterday, and had a lot of progress since than. Yet, we still don't have fully working solution. There few small issues that preventing us from moving forward, and we are working on them now. So far the patch size is 79kb, and it includes more than 1000 lines of changes, some of which is asm, but most of it is C. This is a huge sized patch, as typically our patches within 100 lines. The sheer size of the patch, and the number of subsistems it touches is what makes it so hard & causes such a long delay. Yet, we are getting really close.

Update [Jan 10, 2018 5:21am PT]

KernelCare Update -- Sometimes optimizm is a good thing. Sometimes it drives you into attempting something "nearly" imposible, and you spend much more time than you expect, and still don't see the end of the road.
We are still working on the patches, still fighting bugs. A lot of them are quite elusive. It seems that it might be one of those projects where last 10% of the job takes 90% of the work. We made virtually no progress since yesterday. Only few bugs were fixed, and several new bugs were detected. Given that we have one shot at this, and if we get it wrong, the server will crash -- it might take us long time.

I was hopeful that we will be able to pass this stage much quicker - but now it is obvious it is not going to happen. We will continue working, and trying to solve these problems asap. Yet, realistically -- as of now Friday is the best case ETA, and it can easily spill over to the next week :(

I appologize for wrong ETA, major delays with this patch and any inconvenience it might have caused.

Update [Jan 9, 2018 5:40pm PT]

KernelCare update -- We are very close, and now we are hitting small(er) bugs. Yet, realistically -- it will take us at least another day before we release. There is just too much code/to many places where bugs appear.

CloudLinux update -- We will release new CL6 & CL7 kernels either today or tomorrow that should fix some of the issues people are having

 

Update [Jan 8, 2018 6:20pm PT]

KernelCare update - we still don't have a build that doesn't crash right away, but we believe we are getting close. Yet, we have to shift most likely release to tomorrow, with possibility it will shift even further to Wednesday. Sorry about all the delays. We are doing the best we can.

Update [Jan 7, 2018 4:45pm PT]

Last update for the day. We still don't have a KernelCare patchset build, but most components needed are there. Re-addressing of existing processes seems to work (that was our main worry). We are hopeful to have KC patches tomorrow, but given that we still don't have a build, tells me that it might be pushed to Tue before we will start releasing them.

I am very sorry about the delays. We are working full force, and my team who I am very proud off, is doing excellent job. Yet, the complexity of the problem is so high that it is very hard to predict exact timeline.

Update [Jan 7, 2018 9:35am PT]

We received a number of complains about CloudLinux 6 kernels. Most of them are related to Xen, but some are not. We traced some of them to upstream patches (CentOS kernels having similar issues).
We don't have solution for it right now. I would recommend people experiencing this issues to move to hybrid kernel: https://docs.cloudlinux.com/index.html?hybrid_kernel.html

It allows to run CloudLinux 7 kernel on CloudLinux 6 servers, and we haven't had any reports regarding it.

We don't have any KernelCare updates yet. We are still trying to do first build of patches today, but yet to be ready for that so far.

Update [Jan 6, 2018 7:55am PT]

KernelCare updates:

As we are starting to piece the code together, we are are starting to hit first snags. So far nothing major, but today is no longer probable. We are still shooting for Sunday/Monday for first releases for EL7 (RedHat/CentOS/CloudLinux 7) releases. 
Note: This is still maybe :( Few major things are yet to be merged (like pgd_realloc responsible for moving exisitng processes to new addressing scheme)

Update [Jan 6, 2018 7:10am PT]

CloudLinux 6 kernels are moved to production.

# yum clean all && yum update kernel-firmware && yum install kernel-2.6.32-896.16.1.lve1.4.49.el6

It is the same kernel as was in beta yesterday.

Update [Jan 5, 2018 10:10am PT]

CloudLinux 7 Beta kernels with Resellers limits has been patched now as well. We are planning to release this kernels (together with reseller limits) to production on 15th. We will not be bringing in reseller limits update via KC, so you might want to take this time and make a jump into resellers limits anyway if you are planning to reboot.

Update instructions:

CL7:
yum clean all --enablerepo=cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.5.8.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
yum clean all --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.5.8.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing
 
Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
- KMODLVE-142: resync stats before returning IO usage
- KMODLVE-140: fix panic with module loading
- KMODLVE-139: add ability to set debug level in load time
- KMODLVE-138: properly check lve_cgroup_kernel_open return value
- KMODLVE-134: code cleanup for better test coverage
- KMODLVE-131: improve failure IDs handling
- KMODLVE-127: lvp_lve_move implementation
 
 

Update [Jan 5, 2018 7:45am PT]

KernelCare update

I just had update on where we are on individual tasks that people are working on for KernelCare. Here is the summary of what we know, what we expect, and suggestions:

1. Fixes available should prevent 2 out of 3 possible attack vectors. Intel promises microcode updates next week to close all known attack vectors. Applying microcode typically requires reboot (but we will try to allow doing it via KC)
2. Exploit for meltdown is very simple, as long as attacker can execute arbitrary code, they can read everything in RAM
3. The patchset is big & complex. We also need to augment it with a lot of our own code to make it patchable with KernelCare, as we need to change the way existing / running processes access kernel space.

Even though we didn't hit any roadblocks today, I think our initial projection of Saturday was very optimistic. We need to continue not hitting any roadblocks to make it.

More realistic projection is Monday, due to complexity & number of components we need to bring together & make sure they all working.

My recommendation as of now would be:
* If you are a service provider where it is easy for an attacker to run arbitrary code, don't wait
* If you are in full control of your servers / enterprise / corp customer with no imediate access for hackers to run any code on your servers, and your risk tolerance is high enough, you might be able to hold off until we have a patch.

Note: We are still going to continue working through weekend/full force to get this patch out. This is just my suggestions best of the information available to me at this moment

Update [Jan 5, 2018 7:00am PT]

CloudLinux 7 and CloudLinux 6hybrid kernels are in production now. It is the same kernels that were pushed to beta, so if you upgraded earlier, you don't need to upgrade again.

Upgrade Instructions:
CL7:

# yum clean all && yum install linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7
 
CL6h:
# yum clean all && yum install linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h
 
What to expect next:
We will move CloudLinux 6 kernels to production tomorrow morning if no new issues will be found.
 
KernelCare update:

So far we are progressing as expected for tomorrow release. We didn't hit any lucky breaks, nor we hit any walls. Yet, we are still far away to be confident on Tomorrow's release.
There is still a posibility that it will be delayed until Sunday, Monday, or we hit something that we didn't realize to be a blocker, and we might have to delay even further :(
I will try to make another (single) update on KC today, but there is really not much information at this stage beyond: We are writting code/testing what we have.

Update [Jan 5, 2018 5:23am PT]

CL6 kernel released to beta. Update command:

# yum clean all --enablerepo=cloudlinux-updates-testing && yum update kernel-firmware --enablerepo=cloudlinux-updates-testing && yum install kernel-2.6.32-896.16.1.lve1.4.49.el6 --enablerepo=cloudlinux-updates-testing
 
Changelog:
CKSIX-143: fixed deadlock when disk quota is enabled

Update [Jan 4, 2018 5:50pm PT]

Some people are experiencing issues iwth CloudLinux 6 kernel. We will push a new kernel early tomorrow morning. Please, hold off install CloudLinux 6 kernel for now. CloudLinux 7 and CloudLinux 6hybrid kernels should be fine.

Update [Jan 4, 2018 3:50pm PT]

CloudLinux 6 kernel is now available. Please, note that unlike CloudLinux 7 kernel, we didn't have time to fully tested. We will complete our tests early tomorrow morning (probably before 7am PT). So, try testing it first, and post if your tests were successful here.

Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
 
Upgrade instructions

yum clean all --enablerepo=cloudlinux-updates-testing && yum update microcode_ctl && yum install kernel-2.6.32-896.16.1.lve1.4.48.el6 --enablerepo=cloudlinux-updates-testing

Don't forget to reboot.

 

Update [Jan 4, 2018 3:37pm PT]

Corrected upgrade instructions

Upgrade instructions
CL7:
# yum clean all --enablerepo=cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
# yum clean all --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing

 

Update [Jan 4, 2018 1:45pm PT]

CloudLinux 7 and CloudLinux 7 hybrid kernels are out. Please, try them, post how it works for you/what overhead you are seeing. Our tests show single digits overhead on with our syntetic / hosting related workloads.

Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
 
Upgrade instructions
CL7:
# yum clean all --cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
# yum clean all --cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing

 

Don't forget to reboot.
CloudLinux 6 kernel should follow within an hour, stand by for more info.

There will be no more KC status updates until tomorrow.

Update [Jan 4, 2018 10:25am PT]

We need to make one more iteration for CloudLinux 7 kernels - this should be the last one. Another ~3 hours.

We also got patches for CloudLinux 6 (thanks to the great team at our upstream kernel at virtuozzo.com). Hopefully it will be ready in another 4 hours.

Update [Jan 4, 2018 7:37am PT]

Some setback on CloudLinux 7 kernels. We need to make some changes and restart the build/test cycles. Another ~3 hours.

Update [Jan 4, 2018 4:50am PT]

Good News: We have CloudLinux 7 and CloudLinux 7 hybrid kernels being built right now. We hope to have them for you in the next 3 hours. CloudLinux 6 kernels will follow shortly after that.

Bad News: We will not have KernelCare patches until Saturday, and even that is considered optimistic at best. We will also have to release three sets of patches, and it might take us a week to cover it all.

The first set of patches should cover KPTI patchset that fixes a meltdown attack (that is the one we optimistically plan for Saturday). There are multiple complexities with the patch, one of which is to change how addressing works for existing processes, and the other one is how to deal with unpatch (and changing addressing again). This code would have to be written from scratch, as this condition doesn't happen.

The second patch will be focused on speculation control that wasn't part of mainstream, but part of RHEL kernel, and tries to address one of the Spektre attack.

The third patchset will try to load microcode used to protect against second Spectre attack on the fly. Typically microcode is loaded on OS boot, but we will try to safely apply it using KernelCare.

Why is it so bad: There is a chance that we don't understand "everything" yet, and there is something that will prevent us from delivering this patches altogether, or will delay them even more.  We are trying to hot patch six months of work of a fairly big group of kernel developers in a very short time and will be working non stop as long as we can. Yet, this is really complex problems and we foresee that we will hit a few brick walls that we would have to smash before we will have patches. 

Update [Jan 3, 2018 8:03pm PT]

There will be no more updates until Jan 4th. Please, expect more updates around 5 am PT. We are trying to solve it as fast as we can, but the changes are big, intrusive, and we had little prior warning.

Update [Jan 3, 2018 6:54pm PT]

PoC for Spectre attack is publicly available. This is the attack that has no patches yet from any vendor, and might not be even possible to  protect against.

https://pastebin.com/A58h8N7y

Update [Jan 3, 2018 6:10pm PT]

We are mostly done with CloudLinux 7 patches (full kernel, not KernelCare), but there are still several hours of work left before we can start testing them.

We also identified the areas that need to be modified for patches to be applicable by KernelCare and started to work on those areas.

We hope to release some CloudLinux kernels into beta tomorrow. We are yet to have ETA for KernelCare patches due to the large size of the needed patch, and some critical areas that the patch affects.

Update [Jan 3, 2018 3:40pm PT]

We have succesfully downloaded the sources for the patch for RHEL7, and started to work on them. Yet, please, don't expect any KC patches today. The patchset is complex, and it will take time to adopt it.

Update [Jan 3, 2018 3:40pm PT]

RedHat code cannot be downloaded for now (for whatever reason, we are dealing with it)

Xen released Xen Security Advisory - Intel and AMD CPUs affected, no mitigation/solution for now for two out of three ways of attacking the system.

Update [Jan 3, 2018 3:25pm PT]

RedHat released security advisory/fixes for RHEL 6.4. This gives us something to work with.

Update [Jan 3, 2018 3:15pm PT]

The vulnerabilities were finally disclosed: https://spectreattack.com/

Our initial attempt to port mainline patches to CloudLinux 7 & EL 7 kernels is not succesful due to lack of PCID support. Without PCID, the performance drop will be 30%+. We are working on workarounds, and waiting for an update from the upstream.

 

Original Story:

We don't have all the information yet. We don't know much beyond what has already been reported by The Register and by Intel.

What we assume as of now:

  • There is a bug in Intel CPUs that allows user space software to read kernel's memory;
  • There is a patch that fixes the issue (presumably) committed in mainline kernel;
  • The patch results in 5% to 30% performance penalty;
  • The patch is complex and needs to be reworked to be applied by KernelCare.

What we don't know:

  • We don't know just how pervasive the problem is, and how it can be exploited;
  • We don't know if the patch accepted by mainline kernel fixes the vulnerability completely;
  • We don't know if the patch will crash servers under some workloads;
  • We don't know what information about vulnerability becomes public and which details will be revealed.

What we are doing now:

  • We are adopting existing mainline kernel for CloudLinux 7 and CentOS 7;
  • We are working on adopting the patch for patching by KernelCare. Yet, due to the complexity of the patch, we don't have an ETA yet.

What we plan to do:

  • Prepare KernelCare patches in order: CentOS/RHEL/CloudLinux 7, CentOS/RHEL/CloudLinux 6/Virtuozzo, Ubuntu, Debian, other…;
  • Wait until upstream provides info on their patches, to make sure that our adopted patches work;
  • Potentially -- release our KernelCare patches as 'experimental' once they are ready;
  • Deliver patched CloudLinux 7, and then CloudLinux 6 kernel into beta, as soon as it is ready.

We are also thinking about the best way to deliver the patches, as they can have major adverse performance effect on your servers (due to 5%-30% performance penalty). We want to make sure that you have a way to control it.

We cannot provide you with an ETA yet, but we know that we will not be delivering anything today, January 3rd. We will provide you with more information tomorrow, or sooner if we have more information.
 

Topic: KernelCare Blog CloudLinux OS Blog

82532 people viewed this

Comments (148)

 
by Guest - Richard Hordern / Wednesday, 03 January 2018 21:02

A bit more info has been recently published here:
https://newsroom.intel.com/news/intel-responds-to-security-research-findings/

Intel say that AMD and ARM also have this issue and that the impact is workload related and won’t be noticeable for everyone.

A bit more info has been recently published here: https://newsroom.intel.com/news/intel-responds-to-security-research-findings/ Intel say that AMD and ARM also have this issue and that the impact is workload related and won’t be noticeable for everyone.
by Guest - Miguel / Thursday, 04 January 2018 19:10

When do you expect to have a solution?still waiting

When do you expect to have a solution?still waiting
by Guest - somebody / Thursday, 04 January 2018 22:26

I just want to say thanks for your outstanding work. I hope there will be soon patches for kernelcare, specially for OpenVZ and CentOS 6.
I also hope you can live patch the microcode update.

@Miguel - 'still waiting', really?
There are, probably, missing informations about the whole case and as i see, there are not even patches for all operating systems. Still, Cloudlinux is working - probably 24/7 - on a quick solution. Appreciate that.

I just want to say thanks for your outstanding work. I hope there will be soon patches for kernelcare, specially for OpenVZ and CentOS 6. I also hope you can live patch the microcode update. @Miguel - 'still waiting', really? There are, probably, missing informations about the whole case and as i see, there are not even patches for all operating systems. Still, Cloudlinux is working - probably 24/7 - on a quick solution. Appreciate that.
by Guest - AlexP / Thursday, 04 January 2018 23:30

Have problem with CloudLinux Server release 6.9. Please, check info into this article.

1) yum error

Command line error: no such option: --cloudlinux-updates-testing,cloudlinux-hybrid-testing

2) CL6
No package linux-firmware available.
Package(s) microcode_ctl available, but not installed.

3) CL6
cloudlinux-updates-testing/primary_db | 2.0 MB 00:02
No package kernel-3.10.0-714.10.2.lve1.4.79.el6h available.

Have problem with CloudLinux Server release 6.9. Please, check info into this article. 1) yum error Command line error: no such option: --cloudlinux-updates-testing,cloudlinux-hybrid-testing 2) CL6 No package linux-firmware available. Package(s) microcode_ctl available, but not installed. 3) CL6 cloudlinux-updates-testing/primary_db | 2.0 MB 00:02 No package kernel-3.10.0-714.10.2.lve1.4.79.el6h available.
by Igor Seletskiy / Thursday, 04 January 2018 23:38

sorry, corrected the upgrade instructions.

sorry, corrected the upgrade instructions.
by Igor Seletskiy / Thursday, 04 January 2018 23:33

Do you have it configured for CloudLinux hybrid kernel?
https://docs.cloudlinux.com/index.html?hybrid_kernel.html

Do you have it configured for CloudLinux hybrid kernel? https://docs.cloudlinux.com/index.html?hybrid_kernel.html
by Guest - AlexP / Friday, 05 January 2018 00:27

Thx Igor. Now all okay.
Thanks all your team for that updates.

Thx Igor. Now all okay. Thanks all your team for that updates.
by Guest - Mr. Anonymous / Friday, 05 January 2018 00:27

Thank you, first CloudLinux 7 server rebooted into new kernel without an issue so far. No noticeable change in CPU utilisation but this is a very low load server and we're during off-peak times. The important thing is the server booted and is running without an issue.

Thank you, first CloudLinux 7 server rebooted into new kernel without an issue so far. No noticeable change in CPU utilisation but this is a very low load server and we're during off-peak times. The important thing is the server booted and is running without an issue.
by Guest - Mr. Anonymous / Friday, 05 January 2018 00:45

I am now seeing the following:

No package kernel-3.10.0-714.10.2.lve1.4.79.el7 available.

On mirrors de-proxy.cloudlinux.com and de-proxy.cl-mirror.net

There was no problem with de-proxy.cloudlinux.com on the previous server.

Is this just a matter of waiting for mirrors to update or has the package been pulled for some reason?

Thank you!
Mr. Anonymous

I am now seeing the following: No package kernel-3.10.0-714.10.2.lve1.4.79.el7 available. On mirrors de-proxy.cloudlinux.com and de-proxy.cl-mirror.net There was no problem with de-proxy.cloudlinux.com on the previous server. Is this just a matter of waiting for mirrors to update or has the package been pulled for some reason? Thank you! Mr. Anonymous
by Guest - Mr. Anonymous / Friday, 05 January 2018 01:31

Whoops - this was my mistake due to using the protectbase plugin. CloudLinux support responded in an amazing 2 minutes pointing out the issue and recommending to use "--disableplugin=protectbase --disableexcludes=all". Awesome work CloudLinux! :)

Whoops - this was my mistake due to using the protectbase plugin. CloudLinux support responded in an amazing 2 minutes pointing out the issue and recommending to use "--disableplugin=protectbase --disableexcludes=all". Awesome work CloudLinux! :)
by Guest - David Majchrzak / Friday, 05 January 2018 01:20

Hi!

CL6 failed to boot, I've sent a screen of the console to your support email.
CL7 seems fine so far, good work!

Hi! CL6 failed to boot, I've sent a screen of the console to your support email. CL7 seems fine so far, good work!
by Guest - Lucas / Friday, 05 January 2018 01:21

i got this error:

Could not retrieve mirrorlist http://httpupdate.cpanel.net/cPAddons-c6-x86_64-mirrorlist error was
14: PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error"
Error: Cannot find a valid baseurl for repo: cpanel-addons-production-feed

when i ran
yum clean all --enablerepo=cloudlinux-updates-testing && yum update microcode_ctl && yum install kernel-2.6.32-896.16.1.lve1.4.48.el6 --enablerepo=cloudlinux-updates-testing

cat /etc/*release*
CloudLinux Server release 6.9 (Igor Volk)
CloudLinux Server release 6.9 (Igor Volk)
cpe:/o:redhat:enterprise_linux:6:ga:server

i got this error: Could not retrieve mirrorlist http://httpupdate.cpanel.net/cPAddons-c6-x86_64-mirrorlist error was 14: PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error" Error: Cannot find a valid baseurl for repo: cpanel-addons-production-feed when i ran yum clean all --enablerepo=cloudlinux-updates-testing && yum update microcode_ctl && yum install kernel-2.6.32-896.16.1.lve1.4.48.el6 --enablerepo=cloudlinux-updates-testing cat /etc/*release* CloudLinux Server release 6.9 (Igor Volk) CloudLinux Server release 6.9 (Igor Volk) cpe:/o:redhat:enterprise_linux:6:ga:server
by Guest - Lucas / Friday, 05 January 2018 01:24

already started to work, forget about it

already started to work, forget about it
by Guest - Richard / Friday, 05 January 2018 01:35

My machine seems to boot to Enabling system quotas and then throws an error after a minute saying that the CPU has hung for 67 seconds.

Works when I select the previous kenel

My machine seems to boot to Enabling system quotas and then throws an error after a minute saying that the CPU has hung for 67 seconds. Works when I select the previous kenel
by Guest - Lucas / Friday, 05 January 2018 01:44

Me too :( in CL6

Me too :( in CL6
by Guest - Jonathan / Friday, 05 January 2018 01:49

Oddly; installing a fresh install of CL6 and upgrading the kernel works fine (CL6 vm on hyper-v). Upgrading my existing CL6 webservers on hyper-v is giving the same results as you ; getting stuck at enabling quotas.

Oddly; installing a fresh install of CL6 and upgrading the kernel works fine (CL6 vm on hyper-v). Upgrading my existing CL6 webservers on hyper-v is giving the same results as you ; getting stuck at enabling quotas.
by Guest - Richard / Friday, 05 January 2018 09:06

I should have given a little more information, this machine is CL6 on Proxmox cluster.

I should have given a little more information, this machine is CL6 on Proxmox cluster.
by Guest - Znuff / Friday, 05 January 2018 04:50

Hello,

Having issues with CL6h and Flashcache. The Flashcache module is not compiled for this kernel anymore.

I wasn't able to compile it myself (I don't seem to have the correct gcc flags available).

Please provide flashcache module for this kernel. Any other alternatives for speeding up spinning disks is not functional in CL6h (bcache is not working, for example).

Hello, Having issues with CL6h and Flashcache. The Flashcache module is not compiled for this kernel anymore. I wasn't able to compile it myself (I don't seem to have the correct gcc flags available). Please provide flashcache module for this kernel. Any other alternatives for speeding up spinning disks is not functional in CL6h (bcache is not working, for example).
by Guest - John / Friday, 05 January 2018 07:38

Noticed one error:

grep: /boot/grub2/grub.cfg: No such file or directory
grep: /boot/grub2/grub.cfg: No such file or directory
grep: /boot/grub2/grub.cfg: No such file or directory

[WARNING] Kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 GRUB entry is missing
[ERROR] Kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 initramfs GRUB entry is missing and cannot be automatically added due to unknown reasons, please fix GRUB config manually
Verifying : 1:kernel-3.10.0-714.10.2.lve1.4.79.el7.x86_64 1/2
Verifying : kernel-3.10.0-693.el7.x86_64 2/2

Removed:
kernel.x86_64 0:3.10.0-693.el7

Installed:
kernel.x86_64 1:3.10.0-714.10.2.lve1.4.79.el7


This sytem has efi so grub is at /boot/efi/EFI/centos/grub.cfg

Noticed one error: grep: /boot/grub2/grub.cfg: No such file or directory grep: /boot/grub2/grub.cfg: No such file or directory grep: /boot/grub2/grub.cfg: No such file or directory [WARNING] Kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 GRUB entry is missing [ERROR] Kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 initramfs GRUB entry is missing and cannot be automatically added due to unknown reasons, please fix GRUB config manually Verifying : 1:kernel-3.10.0-714.10.2.lve1.4.79.el7.x86_64 1/2 Verifying : kernel-3.10.0-693.el7.x86_64 2/2 Removed: kernel.x86_64 0:3.10.0-693.el7 Installed: kernel.x86_64 1:3.10.0-714.10.2.lve1.4.79.el7 This sytem has efi so grub is at /boot/efi/EFI/centos/grub.cfg
by Guest - Urgh / Friday, 05 January 2018 16:18

Same problem. Server rebooted OK though.

Same problem. Server rebooted OK though.
1 2 3 4 5 6 7 8

Leave your comment

Guest, Thursday, 13 December 2018

Captcha Image