CloudLinux - CloudLinux Blog - Intel CPU Bug - Meltdown and Spectre - KernelCare and CloudLinux
CloudLinux OS Blog

By accepting you will be accessing a service provided by a third-party external to https://www.cloudlinux.com/

Intel CPU Bug - Meltdown and Spectre - KernelCare and CloudLinux

Intel CPU Bug - Meltdown and Spectre - KernelCare and CloudLinux
Update [May 29, 2018 12:25am PT]

Meltdown fixes for Ubuntu 16.04 are now on test.

To deploy them:
edit /etc/sysconfig/kcare/kcare.conf

Add:
PREFIX=test

Run:
kcarectl --update

TAGNAME: update-2018-05-29-test-1

ubuntu-xenial:
  CVE-2017-5754: Systems with microprocessors utilizing speculative execution and
    indirect branch prediction may allow unauthorized disclosure of information to
    an attacker with local user access via a side-channel analysis of the data cache.
  cvelist: [CVE-2017-5754]
  latest-version: 4.4.0-122.146

Update [Mar 30, 2018 12:25am PT]

Spectre fixes.
We are glad to announce that we've released Specre fixes to the TEST feed for the following distributions:

RHEL 6
Cent OS 6
CentOS 6 Plus
CloudLinux 6
OpenVZ
Proxmox VE 3.10
Proxmox VE 2.6

RHEL 7
CentOS 7

CentOS7 Plus
CloudLinux 7
CloudLinux 6 Hybrid

To deploy them:
edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

Note: It is a must to apply microcode changes. For the instructions follow the link: Microcode update form Intel

Update [Mar 20, 2018 05:25am PT]

Spectre fixes.
We've delayed our fixes for Specre vulnerability, as it is includes  Microcode update form Intel

That fixes was released by Intel last week (20180312 Release). Now we in process of carefull testing our patches with Specre fixes together with that Microcode change. 

ETA for public test release for el7 kernels - next week


Update [Mar 12, 2018 02:21am PT]

We are glad to announce that patches with Meltdown fixes are ready and released for: 

- CentOS 6 (latest-version: 2.6.32-696.20.1.el6)
- Cent OS 6 Plus (latest-version: 2.6.32-696.20.1.el6)
- RHEL 6 (latest-version: 2.6.32-696.20.1.el6)
- CloudLinux 6 (latest-version: 2.6.32-896.16.1.lve1.4.51.el6)
- OpenVZ (latest-version: 2.6.32-042stab127.2)
- Proxmox VE 3.10 ( latest-version: 3.10.0-22-pve_3.10.0-52)
- Cent OS 7  ( latest-version: 3.10.0-693.21.1.el7)
- Cent OS 7 Plus ( latest-version: 3.10.0-693.17.1.el7)
- RHEL 7 (latest-version: 3.10.0-693.21.1.el7)
- CloudLinux 7 (latest-version: 3.10.0-714.10.2.lve1.5.12.el7)
- CloudLinux 6 Hybrid (latest-version: 3.10.0-714.10.2.lve1.5.12.el6h)


Update [Feb 26, 2018 05:59am PT]

Fixed patches with Meltdown fixes for CentOS6 are going to be released till the end of that week. 
For CentOS 6 users: If you facing any problems with current patches, please, don't hesitate to contact our support team. We will get back to you shortly.

Patches with Spectre fixes (part 2) for el7 are ready, so we are in phase of testing that patches. Will be on Test server till the end of that week as well.

Next Spectre fixes (el6, Ubuntu, Debian) will be held on March.

Meltdown fixes for Ubuntu 16.04 and 14.04, Debian 7 are going to be released within next 2 weeks. Patches for Debian 8 and 9 will be held on March.

 

Update [Feb 20, 2018 08:20am PT]

For CentOS 6 users:


We still working on a proper fix for the crashed patch. Finally, we figured out what was the real source of the problem, so fix is coming.

ETA - Feb 22

Update [Feb 16, 2018 10:20am PT]

For CentOS 6 users:


Unfortunately, few clients running CentOS 6 distribution faced crashes after our last patch was applied. That is why we unroll that from a patch server until we do a proper fix. ETA - Feb 19

Update [Feb 16, 2018 00:20am PT]

We have released patches for Meltdown vulnerability for Xen PV hosts for CentOS7, CentOS7 Plus, CloudLinux 6 Hybrid, Proxmox VE 3.10, RHEL 7. Our internal tests, as well as tests with live customers over the past three days, didn't reveal any crashes.


Update [Feb 13, 2018 06:00am PT]

We are actively working on the fixes for others supported distros. Patches for servers that running Xen PV are going to be deployed to TEST today.
Patches with a fix for Meltdown for Ubuntu 16.04 and Ubuntu 14.04 to be added next week.


Update [Feb 10, 2018 11:00am PT]

We are happy to announce that we have released patches for Meltdown vulnerabilit for RedHat, CentOS, CloudLinux 6 as well as Virtuozzo/OpenVZ 2.6.32 kernels. Our internal tests, as well as tests with live customers, didn't reveal any crashes.

* Note: If you are running Xen PV -- they will not work



Update [Feb 6, 2018 1:57pm PT]

We have pushed Meltdown patches for RedHat, CentOS, CloudLinux 6 as well as Virtuozzo/OpenVZ 2.6.32 kernels to TEST channel

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

Update [Feb 2, 2018 11:27am PT]

Patches for CentOS 7 were released.


Update [Feb 2, 2018 8:27am PT]

Patches for CloudLinux 6 Hybrid and CloudLinux 7 were released.  Patches for Cent OS 7 will be released in 2 hours.

Update [Feb 1, 2018 8:25am PT]

Patches for RHEL 7, Cent-OS 7 Plus and Proxmox VE 3.10 were released. Other RHEL 7 related distributions will be released tomorrow.

Update [Jan 30, 2018 7:55am PT]

So far we had no negative reports and several positives. This is your last chance to test it before tomorrow's rollout of patches.

Please, try testing it on at least one machine.

Update [Jan 29, 2018 7:50am PT]

New patchset released to test for the same distributions as bellow with three more issues fixed. Hopefully this will be the last 'test' patchset. Please, help us with testing it. It seems like we removed all issues that might cause crash, and the two out of three issues were minor/didn't result in the crash.

Update [Jan 23, 2018 5:00am PT]

CloudLinux 7, CloudLinux 6h, CentOS/RHEL 7 and Proxmox VE 3.10 patches are ready/waiting for you in test channel. The issue from last round of tests had been resolved.

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

 

 

Update [Jan 22, 2018 5:40am PT]

We have received one crash report, that is related to NMI. We will not be releasing patches today, as we have to fix this issue first.

Thank you, everyone, who helped us test this patch. Please, don't install test patches for now, as we now know they have an issue.

Update [Jan 21, 2018 9:50am PT]

So far we have a few reports that patches are working as expected/well, no negative reports/crashes. We would like few more people to try/test, before we push it out.

Update [Jan 20, 2018 1:30pm PT]

CloudLinux 7, CloudLinux 6h, CentOS/RHEL 7 patches are ready/waiting for you in test channel.

* Note: If you are running Xen PV -- they will not work (see post bellow).

To deploy them:

edit /etc/sysconfig/kcare/kcare.conf

Add:

PREFIX=test

Run:

kcarectl --update

If you see any issues with the server -- contact our support. 

We really need your feedback. This patch is more complex than any we have ever done, and it is impossible to fully test it. If you have a large park of servers, consider having at least one of them tested with test channel, BEFORE we push it to production.

 

Update [Jan 18, 2018 9:20pm PT]

CPU Hotplugging has been fixed. Xen PV doesn't work for now, everything else seems to work just fine. Our plan is to release patches to testing tomorrow morning for those who wants to test, followed by general release on Monday.

Note, this is for CL/CentOS/RHEL 7 only. CL/CentOS/RHEL/Virtuozzo 6 should follow shortly after. The rest of the distributions will be covered by the end of the next week.

* If you run on Xen PV -- you might want to disable KernelCare updates https://docs.kernelcare.com/index.html?settings.htm , as new version will not apply, but will remove old version.

PS: We do plan to add support for Xen PV as well, but only after we resolve issues with the rest of distributions.

 

Update [Jan 18, 2018 9:20pm PT]

We have been running extensive tests for CL/CentOS/RHEL 7 patchset, and so far the only issue we found was with CPU hotplugging. This issue is fixed, and we have started next set of tests. 

Update [Jan 17, 2018 1:10pm PT]

We are testing patches CloudLinux 7, CloudLinux 6h, CentOS/RHEL7 on hardware with NMI, and planning to make them available tomorrow on our 'test' channel. We will provide exact instructions on how you can test them.

Once we are done with this platforms, we will start adding all other distributions.

Update [Jan 16, 2018 8:10am PT]

We are getting very close with Meltdown patches. Our KernelCare team got the patchset working & stable. It is going through extensive code review now, and soon we will send it for ~12 hours of testing. If everything goes through well -- we might have patches available for those willing to test them either later today or tomorrow. Stay tunned.

Update [Jan 14, 2018 7:10am PT]

In the past few days we have made a bit more progress with KernelCare patch for Meltdown. We also hit three distinct issues that we are now working to overcome. We hope that once we done with them within next 1-2 days, we will not hit any other major issues.

Update [Jan 11, 2018 7:35am PT]

We are starting to roll out protection against CVE-2017-5753 via KernelCare. This is one of CVEs affecting Spectre. It is significantly smaller than KAISER changes that  are needed to protect against Meltdown.
The defence against Meltdown is still work in progress.

Update [Jan 11, 2018 6:35am PT]

We have resolved one particularly elusive bug yesterday, and had a lot of progress since than. Yet, we still don't have fully working solution. There few small issues that preventing us from moving forward, and we are working on them now. So far the patch size is 79kb, and it includes more than 1000 lines of changes, some of which is asm, but most of it is C. This is a huge sized patch, as typically our patches within 100 lines. The sheer size of the patch, and the number of subsistems it touches is what makes it so hard & causes such a long delay. Yet, we are getting really close.

Update [Jan 10, 2018 5:21am PT]

KernelCare Update -- Sometimes optimizm is a good thing. Sometimes it drives you into attempting something "nearly" imposible, and you spend much more time than you expect, and still don't see the end of the road.
We are still working on the patches, still fighting bugs. A lot of them are quite elusive. It seems that it might be one of those projects where last 10% of the job takes 90% of the work. We made virtually no progress since yesterday. Only few bugs were fixed, and several new bugs were detected. Given that we have one shot at this, and if we get it wrong, the server will crash -- it might take us long time.

I was hopeful that we will be able to pass this stage much quicker - but now it is obvious it is not going to happen. We will continue working, and trying to solve these problems asap. Yet, realistically -- as of now Friday is the best case ETA, and it can easily spill over to the next week :(

I appologize for wrong ETA, major delays with this patch and any inconvenience it might have caused.

Update [Jan 9, 2018 5:40pm PT]

KernelCare update -- We are very close, and now we are hitting small(er) bugs. Yet, realistically -- it will take us at least another day before we release. There is just too much code/to many places where bugs appear.

CloudLinux update -- We will release new CL6 & CL7 kernels either today or tomorrow that should fix some of the issues people are having

 

Update [Jan 8, 2018 6:20pm PT]

KernelCare update - we still don't have a build that doesn't crash right away, but we believe we are getting close. Yet, we have to shift most likely release to tomorrow, with possibility it will shift even further to Wednesday. Sorry about all the delays. We are doing the best we can.

Update [Jan 7, 2018 4:45pm PT]

Last update for the day. We still don't have a KernelCare patchset build, but most components needed are there. Re-addressing of existing processes seems to work (that was our main worry). We are hopeful to have KC patches tomorrow, but given that we still don't have a build, tells me that it might be pushed to Tue before we will start releasing them.

I am very sorry about the delays. We are working full force, and my team who I am very proud off, is doing excellent job. Yet, the complexity of the problem is so high that it is very hard to predict exact timeline.

Update [Jan 7, 2018 9:35am PT]

We received a number of complains about CloudLinux 6 kernels. Most of them are related to Xen, but some are not. We traced some of them to upstream patches (CentOS kernels having similar issues).
We don't have solution for it right now. I would recommend people experiencing this issues to move to hybrid kernel: https://docs.cloudlinux.com/index.html?hybrid_kernel.html

It allows to run CloudLinux 7 kernel on CloudLinux 6 servers, and we haven't had any reports regarding it.

We don't have any KernelCare updates yet. We are still trying to do first build of patches today, but yet to be ready for that so far.

Update [Jan 6, 2018 7:55am PT]

KernelCare updates:

As we are starting to piece the code together, we are are starting to hit first snags. So far nothing major, but today is no longer probable. We are still shooting for Sunday/Monday for first releases for EL7 (RedHat/CentOS/CloudLinux 7) releases. 
Note: This is still maybe :( Few major things are yet to be merged (like pgd_realloc responsible for moving exisitng processes to new addressing scheme)

Update [Jan 6, 2018 7:10am PT]

CloudLinux 6 kernels are moved to production.

# yum clean all && yum update kernel-firmware && yum install kernel-2.6.32-896.16.1.lve1.4.49.el6

It is the same kernel as was in beta yesterday.

Update [Jan 5, 2018 10:10am PT]

CloudLinux 7 Beta kernels with Resellers limits has been patched now as well. We are planning to release this kernels (together with reseller limits) to production on 15th. We will not be bringing in reseller limits update via KC, so you might want to take this time and make a jump into resellers limits anyway if you are planning to reboot.

Update instructions:

CL7:
yum clean all --enablerepo=cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.5.8.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
yum clean all --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.5.8.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing
 
Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
- KMODLVE-142: resync stats before returning IO usage
- KMODLVE-140: fix panic with module loading
- KMODLVE-139: add ability to set debug level in load time
- KMODLVE-138: properly check lve_cgroup_kernel_open return value
- KMODLVE-134: code cleanup for better test coverage
- KMODLVE-131: improve failure IDs handling
- KMODLVE-127: lvp_lve_move implementation
 
 

Update [Jan 5, 2018 7:45am PT]

KernelCare update

I just had update on where we are on individual tasks that people are working on for KernelCare. Here is the summary of what we know, what we expect, and suggestions:

1. Fixes available should prevent 2 out of 3 possible attack vectors. Intel promises microcode updates next week to close all known attack vectors. Applying microcode typically requires reboot (but we will try to allow doing it via KC)
2. Exploit for meltdown is very simple, as long as attacker can execute arbitrary code, they can read everything in RAM
3. The patchset is big & complex. We also need to augment it with a lot of our own code to make it patchable with KernelCare, as we need to change the way existing / running processes access kernel space.

Even though we didn't hit any roadblocks today, I think our initial projection of Saturday was very optimistic. We need to continue not hitting any roadblocks to make it.

More realistic projection is Monday, due to complexity & number of components we need to bring together & make sure they all working.

My recommendation as of now would be:
* If you are a service provider where it is easy for an attacker to run arbitrary code, don't wait
* If you are in full control of your servers / enterprise / corp customer with no imediate access for hackers to run any code on your servers, and your risk tolerance is high enough, you might be able to hold off until we have a patch.

Note: We are still going to continue working through weekend/full force to get this patch out. This is just my suggestions best of the information available to me at this moment

Update [Jan 5, 2018 7:00am PT]

CloudLinux 7 and CloudLinux 6hybrid kernels are in production now. It is the same kernels that were pushed to beta, so if you upgraded earlier, you don't need to upgrade again.

Upgrade Instructions:
CL7:

# yum clean all && yum install linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7
 
CL6h:
# yum clean all && yum install linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h
 
What to expect next:
We will move CloudLinux 6 kernels to production tomorrow morning if no new issues will be found.
 
KernelCare update:

So far we are progressing as expected for tomorrow release. We didn't hit any lucky breaks, nor we hit any walls. Yet, we are still far away to be confident on Tomorrow's release.
There is still a posibility that it will be delayed until Sunday, Monday, or we hit something that we didn't realize to be a blocker, and we might have to delay even further :(
I will try to make another (single) update on KC today, but there is really not much information at this stage beyond: We are writting code/testing what we have.

Update [Jan 5, 2018 5:23am PT]

CL6 kernel released to beta. Update command:

# yum clean all --enablerepo=cloudlinux-updates-testing && yum update kernel-firmware --enablerepo=cloudlinux-updates-testing && yum install kernel-2.6.32-896.16.1.lve1.4.49.el6 --enablerepo=cloudlinux-updates-testing
 
Changelog:
CKSIX-143: fixed deadlock when disk quota is enabled

Update [Jan 4, 2018 5:50pm PT]

Some people are experiencing issues iwth CloudLinux 6 kernel. We will push a new kernel early tomorrow morning. Please, hold off install CloudLinux 6 kernel for now. CloudLinux 7 and CloudLinux 6hybrid kernels should be fine.

Update [Jan 4, 2018 3:50pm PT]

CloudLinux 6 kernel is now available. Please, note that unlike CloudLinux 7 kernel, we didn't have time to fully tested. We will complete our tests early tomorrow morning (probably before 7am PT). So, try testing it first, and post if your tests were successful here.

Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
 
Upgrade instructions

yum clean all --enablerepo=cloudlinux-updates-testing && yum update microcode_ctl && yum install kernel-2.6.32-896.16.1.lve1.4.48.el6 --enablerepo=cloudlinux-updates-testing

Don't forget to reboot.

 

Update [Jan 4, 2018 3:37pm PT]

Corrected upgrade instructions

Upgrade instructions
CL7:
# yum clean all --enablerepo=cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
# yum clean all --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing

 

Update [Jan 4, 2018 1:45pm PT]

CloudLinux 7 and CloudLinux 7 hybrid kernels are out. Please, try them, post how it works for you/what overhead you are seeing. Our tests show single digits overhead on with our syntetic / hosting related workloads.

Changelog:
- Added patches for Meltdown and Spectre attacks (CVE-2017-5753, CVE-2017-5715, CVE-2017-5754)
 
Upgrade instructions
CL7:
# yum clean all --cloudlinux-updates-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el7 --enablerepo=cloudlinux-updates-testing
 
CL6h:
# yum clean all --cloudlinux-updates-testing,cloudlinux-hybrid-testing && yum update linux-firmware microcode_ctl && yum install kernel-3.10.0-714.10.2.lve1.4.79.el6h --enablerepo=cloudlinux-updates-testing,cloudlinux-hybrid-testing

 

Don't forget to reboot.
CloudLinux 6 kernel should follow within an hour, stand by for more info.

There will be no more KC status updates until tomorrow.

Update [Jan 4, 2018 10:25am PT]

We need to make one more iteration for CloudLinux 7 kernels - this should be the last one. Another ~3 hours.

We also got patches for CloudLinux 6 (thanks to the great team at our upstream kernel at virtuozzo.com). Hopefully it will be ready in another 4 hours.

Update [Jan 4, 2018 7:37am PT]

Some setback on CloudLinux 7 kernels. We need to make some changes and restart the build/test cycles. Another ~3 hours.

Update [Jan 4, 2018 4:50am PT]

Good News: We have CloudLinux 7 and CloudLinux 7 hybrid kernels being built right now. We hope to have them for you in the next 3 hours. CloudLinux 6 kernels will follow shortly after that.

Bad News: We will not have KernelCare patches until Saturday, and even that is considered optimistic at best. We will also have to release three sets of patches, and it might take us a week to cover it all.

The first set of patches should cover KPTI patchset that fixes a meltdown attack (that is the one we optimistically plan for Saturday). There are multiple complexities with the patch, one of which is to change how addressing works for existing processes, and the other one is how to deal with unpatch (and changing addressing again). This code would have to be written from scratch, as this condition doesn't happen.

The second patch will be focused on speculation control that wasn't part of mainstream, but part of RHEL kernel, and tries to address one of the Spektre attack.

The third patchset will try to load microcode used to protect against second Spectre attack on the fly. Typically microcode is loaded on OS boot, but we will try to safely apply it using KernelCare.

Why is it so bad: There is a chance that we don't understand "everything" yet, and there is something that will prevent us from delivering this patches altogether, or will delay them even more.  We are trying to hot patch six months of work of a fairly big group of kernel developers in a very short time and will be working non stop as long as we can. Yet, this is really complex problems and we foresee that we will hit a few brick walls that we would have to smash before we will have patches. 

Update [Jan 3, 2018 8:03pm PT]

There will be no more updates until Jan 4th. Please, expect more updates around 5 am PT. We are trying to solve it as fast as we can, but the changes are big, intrusive, and we had little prior warning.

Update [Jan 3, 2018 6:54pm PT]

PoC for Spectre attack is publicly available. This is the attack that has no patches yet from any vendor, and might not be even possible to  protect against.

https://pastebin.com/A58h8N7y

Update [Jan 3, 2018 6:10pm PT]

We are mostly done with CloudLinux 7 patches (full kernel, not KernelCare), but there are still several hours of work left before we can start testing them.

We also identified the areas that need to be modified for patches to be applicable by KernelCare and started to work on those areas.

We hope to release some CloudLinux kernels into beta tomorrow. We are yet to have ETA for KernelCare patches due to the large size of the needed patch, and some critical areas that the patch affects.

Update [Jan 3, 2018 3:40pm PT]

We have succesfully downloaded the sources for the patch for RHEL7, and started to work on them. Yet, please, don't expect any KC patches today. The patchset is complex, and it will take time to adopt it.

Update [Jan 3, 2018 3:40pm PT]

RedHat code cannot be downloaded for now (for whatever reason, we are dealing with it)

Xen released Xen Security Advisory - Intel and AMD CPUs affected, no mitigation/solution for now for two out of three ways of attacking the system.

Update [Jan 3, 2018 3:25pm PT]

RedHat released security advisory/fixes for RHEL 6.4. This gives us something to work with.

Update [Jan 3, 2018 3:15pm PT]

The vulnerabilities were finally disclosed: https://spectreattack.com/

Our initial attempt to port mainline patches to CloudLinux 7 & EL 7 kernels is not succesful due to lack of PCID support. Without PCID, the performance drop will be 30%+. We are working on workarounds, and waiting for an update from the upstream.

 

Original Story:

We don't have all the information yet. We don't know much beyond what has already been reported by The Register and by Intel.

What we assume as of now:

  • There is a bug in Intel CPUs that allows user space software to read kernel's memory;
  • There is a patch that fixes the issue (presumably) committed in mainline kernel;
  • The patch results in 5% to 30% performance penalty;
  • The patch is complex and needs to be reworked to be applied by KernelCare.

What we don't know:

  • We don't know just how pervasive the problem is, and how it can be exploited;
  • We don't know if the patch accepted by mainline kernel fixes the vulnerability completely;
  • We don't know if the patch will crash servers under some workloads;
  • We don't know what information about vulnerability becomes public and which details will be revealed.

What we are doing now:

  • We are adopting existing mainline kernel for CloudLinux 7 and CentOS 7;
  • We are working on adopting the patch for patching by KernelCare. Yet, due to the complexity of the patch, we don't have an ETA yet.

What we plan to do:

  • Prepare KernelCare patches in order: CentOS/RHEL/CloudLinux 7, CentOS/RHEL/CloudLinux 6/Virtuozzo, Ubuntu, Debian, other…;
  • Wait until upstream provides info on their patches, to make sure that our adopted patches work;
  • Potentially -- release our KernelCare patches as 'experimental' once they are ready;
  • Deliver patched CloudLinux 7, and then CloudLinux 6 kernel into beta, as soon as it is ready.

We are also thinking about the best way to deliver the patches, as they can have major adverse performance effect on your servers (due to 5%-30% performance penalty). We want to make sure that you have a way to control it.

We cannot provide you with an ETA yet, but we know that we will not be delivering anything today, January 3rd. We will provide you with more information tomorrow, or sooner if we have more information.
 

Deploying Reseller Limits is easy
Beta: CageFS and liblve updated
 

Комментарии 147

Thanks! :-) I'll sit tight then and wait for RHEL

Thanks! :-) I'll sit tight then and wait for RHEL

Hello,

Can you please tell me why do you suggest yum update microcode_ctl only for CL7, and not for CL6...

Is there a specific reason you are not suggesting microcode_ctl to be installed on CL6?

Hello, Can you please tell me why do you suggest yum update microcode_ctl only for CL7, and not for CL6... Is there a specific reason you are not suggesting microcode_ctl to be installed on CL6?

Hello Deyan, it is required to update the microcode for CL6 as for CL7 also. Sorry if it was written unclearly.

Hello Deyan, it is required to update the microcode for CL6 as for CL7 also. Sorry if it was written unclearly.

It appears microcode_ctl is updated for fedoria
https://pagure.io/microcode_ctl/blob/03eb7989a3ffe264a72389d81e9b976bdac0b0ad/f/Changelog
2.1-15 09 January 2018, Anton Arapov
Intel CPU microcode update. 20180108

Do any fellow customers have redhat subscription access to read updates such as https://access.redhat.com/solutions/3315431 and https://access.redhat.com/solutions/1147173 to get an idea of the ETA for redhat to provide the updated package?

It appears microcode_ctl is updated for fedoria https://pagure.io/microcode_ctl/blob/03eb7989a3ffe264a72389d81e9b976bdac0b0ad/f/Changelog 2.1-15 09 January 2018, Anton Arapov Intel CPU microcode update. 20180108 Do any fellow customers have redhat subscription access to read updates such as https://access.redhat.com/solutions/3315431 and https://access.redhat.com/solutions/1147173 to get an idea of the ETA for redhat to provide the updated package?

Hello,
there is no microcode_ctl package for RHEL / CentOS yet. We will release it as soon as it comes out.

Hello, there is no microcode_ctl package for RHEL / CentOS yet. We will release it as soon as it comes out.

Given the complexity of these patches and that they're still evolving (like Intel pulling microcode changes), can you add a way to disable the meltdown/spectre patching like how you did for CVE-2015-5157 via kcare_modify_ldt = 1 ? Some dedicated server customers may need to do testing first since this can impact high IO loads before going live with the patches (or since they run a single app on the server, its not really relevant either ). Thanks!

Given the complexity of these patches and that they're still evolving (like Intel pulling microcode changes), can you add a way to disable the meltdown/spectre patching like how you did for CVE-2015-5157 via kcare_modify_ldt = 1 ? Some dedicated server customers may need to do testing first since this can impact high IO loads before going live with the patches (or since they run a single app on the server, its not really relevant either ). Thanks!

Yes, we will add something like this for meltdown/spectre patches

Yes, we will add something like this for meltdown/spectre patches

I have just realised that CentOS 7 with latest kernel 3.10.0-693.11.6.el7.x86_64 is not vulerable to Spectre and Meltdown attacks, but latest CloudLinux 7 kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 is vulnarable to Spectre variant 2 (CVE-2017-5715) because SPEC_CTRL and IBRS features are not available in CL kernel.

Am I missing something here, or you just did not implement all changes from RHEL/CentOS kernel?

Can you please explain this?

Thank you.

I have just realised that CentOS 7 with latest kernel 3.10.0-693.11.6.el7.x86_64 is not vulerable to Spectre and Meltdown attacks, but latest CloudLinux 7 kernel 3.10.0-714.10.2.lve1.4.79.el7.x86_64 is vulnarable to Spectre variant 2 (CVE-2017-5715) because SPEC_CTRL and IBRS features are not available in CL kernel. Am I missing something here, or you just did not implement all changes from RHEL/CentOS kernel? Can you please explain this? Thank you.

For me I have:

* Hardware / CPU microcode support for mitigation (SPEC_CTRL MSR): YES
* Hardware / CPU microcode support for mitigation (CPUID bit): YES
* Kernel support for IBRS: YES
* IBRS enabled for Kernel space: NO
* IBRS enabled for User space: NO

So both kernel and CPU supports it.. still not enabled though in kernel and user space, and no way to enable, such as echoing to `/sys/kernel/debug/x86/ibrs_enabled`

For me I have: * Hardware / CPU microcode support for mitigation (SPEC_CTRL MSR): YES * Hardware / CPU microcode support for mitigation (CPUID bit): YES * Kernel support for IBRS: YES * IBRS enabled for Kernel space: NO * IBRS enabled for User space: NO So both kernel and CPU supports it.. still not enabled though in kernel and user space, and no way to enable, such as echoing to `/sys/kernel/debug/x86/ibrs_enabled`

Yes, that is the same behavior. It is also important to mention that IBRS even enabled just for Kernel space have a significant performance impact, so maybe there is a reason why that is not implemented in CL kernel for now. I hope we will get some official explanation about this...

Yes, that is the same behavior. It is also important to mention that IBRS even enabled just for Kernel space have a significant performance impact, so maybe there is a reason why that is not implemented in CL kernel for now. I hope we will get some official explanation about this...

Would it be possible to update the intel cpu microcode directly with the intel mictocode update dat file from https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File on a CL6 (non hybrid) system?

These instructions seem to say yes on some versions of centos and redhat:
http://news.softpedia.com/news/intel-releases-processor-microcode-patch-for-linux-oses-here-s-how-to-update-519316.shtml

Any thoughts on this for a CL6 system?

Would it be possible to update the intel cpu microcode directly with the intel mictocode update dat file from https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File on a CL6 (non hybrid) system? These instructions seem to say yes on some versions of centos and redhat: http://news.softpedia.com/news/intel-releases-processor-microcode-patch-for-linux-oses-here-s-how-to-update-519316.shtml Any thoughts on this for a CL6 system?

IBRS support requires microcode update.
We'd recommend you to wait for a microcode update in our repository. It will be available soon after it appears in upstream's repo.

IBRS support requires microcode update. We'd recommend you to wait for a microcode update in our repository. It will be available soon after it appears in upstream's repo.

IBRS is already available in CentOS 7, with microcode_ctl-2.1-22.2, that is available since 4th january.

It just does not work in CloudLinux, so it looks like it's not related to microcode update, since that same package is also available in CloudLinux repo, but IBRS is not working and SPEC_CTRL is not available. This is from CloudLinux kernel...

kernel: FEATURE SPEC_CTRL Not Present


This is "Present" on CentOS and IBRS is working.

IBRS is already available in CentOS 7, with microcode_ctl-2.1-22.2, that is available since 4th january. It just does not work in CloudLinux, so it looks like it's not related to microcode update, since that same package is also available in CloudLinux repo, but IBRS is not working and SPEC_CTRL is not available. This is from CloudLinux kernel... [quote]kernel: FEATURE SPEC_CTRL Not Present[/quote] This is "Present" on CentOS and IBRS is working.

I've got a confirmation from CL kernel dev team that they are aware of this problem and already started working on it. We will update you on the progress.

I've got a confirmation from CL kernel dev team that they are aware of this problem and already started working on it. We will update you on the progress.

It seems that the January 4 redhat microcode update may not have contained all the november 2017 intel microcode updates but only a subset. Would like more clarification on this if anyone has solid info.

Also the https://access.redhat.com/labsinfo/speculativeexecution says "You must contact your hardware OEM to receive the appropriate microcode/firmware for your processor " whereas Intel has released microcode updates at https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File on January 8 for many cpus more quickly than server bios updates are available.

If Redhat will be a long while out with microcode updates, or not release them, could cloudlinux consider making a method to apply the Intel microcode updates to a CL6 system?

It seems that the January 4 redhat microcode update may not have contained all the november 2017 intel microcode updates but only a subset. Would like more clarification on this if anyone has solid info. Also the https://access.redhat.com/labsinfo/speculativeexecution says "You must contact your hardware OEM to receive the appropriate microcode/firmware for your processor " whereas Intel has released microcode updates at https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File on January 8 for many cpus more quickly than server bios updates are available. If Redhat will be a long while out with microcode updates, or not release them, could cloudlinux consider making a method to apply the Intel microcode updates to a CL6 system?

In your last post you said you patched CVE-2017-5753, but Ubuntu says that the patch for such CVE is not yet available: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
I’m confused.

In your last post you said you patched CVE-2017-5753, but Ubuntu says that the patch for such CVE is not yet available: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown I’m confused.

It's Ubuntu that haven't patched their kernels for CVE-2017-5753

It's Ubuntu that haven't patched their kernels for CVE-2017-5753

KernelCare patches with the CVE-2017-5753 fix that are currently being deployed for distributions that already have this problem fixed in their latest kernel updates (mostly RHEL and derivatives). When Ubuntu will release their version of this fix, KernelCare patches will try to add it to older kernels.

KernelCare patches with the CVE-2017-5753 fix that are currently being deployed for distributions that already have this problem fixed in their latest kernel updates (mostly RHEL and derivatives). When Ubuntu will release their version of this fix, KernelCare patches will try to add it to older kernels.

Can someone from CloudLinux please provide some official info why we still don't have patched kernel for CVE-2017-5715 with working IBRS, when RedHat and CentOS were patched for all three attack variants since 4th january?

When can we expect new kernel with working IBRS for CloudLinux?

Can someone from CloudLinux please provide some official info why we still don't have patched kernel for CVE-2017-5715 with working IBRS, when RedHat and CentOS were patched for all three attack variants since 4th january? When can we expect new kernel with working IBRS for CloudLinux?

IBRS support requires microcode update.
Meltdown and 1 type Spectre attack are already covered. We have a problem with applying new microcode while loading kernel, so 2 type Spectre attack might not be covered. We are investigating this problem.

IBRS support requires microcode update. Meltdown and 1 type Spectre attack are already covered. We have a problem with applying new microcode while loading kernel, so 2 type Spectre attack might not be covered. We are investigating this problem.
Уже зарегистрированны? Войти на сайт
Guest
12.07.2020

Изображение капчи