CloudLinux OS Blog - Let us subject MDS vulnerability to the glare of truth
CloudLinux OS Blog

Featured 

Let us subject MDS vulnerability to the glare of truth

mds3

Ok, let me throw a little light on the last release of the CloudLinux 7 and CloudLinux 6 kernel with the MDS vulnerability patch.

MDS vulnerability explanation

In the last three days, we’ve received a whole bunch of questions like Should I disable Hyper-Threading or not? and How Hyper-Threading disabling can impact performance? So, here we are with some important information about the point.

But what is the problem? CPU has two execution threads per physical core. Both threads share the same resources inside the CPU. It means sibling cores can see the same data as the primary core can.

So what?

The problem involves different attacks:

  • Kernel — user space attack.
  • Userspace — userspace attack for threads running on the same physical core.
  • Virtual Machine — Virtual Machine attack.

Different attack faces need different ways to mitigate it.

  1. If you have a trusted user space, CPU buffers need to be flushed at the exit from the kernel, so the application isn’t able to see kernel data. That's what a microcode update does–it provides the ability to make it flush. If the CPU is supported by the microcode update, you can see a message:
    "Mitigation: Clear CPU buffers"
    OR
    "Mitigation: Clear CPU buffers; SMT vulnerable" in the /sys/devices/system/cpu/vulnerabilities/mds file or by running dmesg |grep MDS.
  2. Virtual machine to virtual machine (VM) attack is different to the previous one. Two VMs can share the same CPU core so they can share CPU data. The kernel adds a CPU buffer flush in this case, in the same way that point one does.
  3. But, resources aren’t completely isolated between cores in a package. The primary core and siblings share some resources at run time. So different applications on the Host or different applications in different VMs can access the same data. Intel CPUs have different protections against this type of attack. Some CPUs have little protection and others none at all. If you want to guarantee this attack will never happen, you can add “,nosmt” string to the end of the mds parameter.
    For example: mds=”full,nosmt”. This will provide an additional check and enable Hyper-Threading only if this is safe.
  4. You will see a message:
    "Mitigation: Clear CPU buffers"
    in the system file or in the dmesg output. Currently, only ATOM series CPUs have this protection.

The problem can be mitigated by a CPU scheduler change. The scheduler must avoid balance loading between vCPUs, but this is a very large change and it is not available for the Linux kernel yet.

What CPUs can have their microcode updated?

Intel doesn’t provide a microcode update for all CPU’s. Only some new ones can be updated at the moment.

Product NamesCPUIDCPUID Intel formatPlatform ID
Xeon Scalable Gen206-55-750657bf
Core Gen206-2a-7206a712
Core Gen306-3a-9306a912
Core Gen406-3c-3306c332
Core Gen506-3d-4306d4c0
Core Gen3 X Series; Xeon E5 v206-3e-4306e4ed
Xeon E7 v206-3e-7306e7ed
Core Gen4 X series; Xeon E5 v306-3f-2306f26f
Xeon E7 v306-3f-4306f480
Core Gen406-45-14065172
Core Gen406-46-14066132
Core Gen506-47-14067122
Core Gen606-4e-3406e3c0
Xeon Scalable06-55-450654b7
Xeon D-21xx06-55-450654b7
Xeon D-1520/4006-56-25066210
Xeon D-1518/19/21/27/28/31/33/37/41/48, Pentium D1507/08/09/17/1906-56-35066310
Xeon D-1557/59/67/71/77/81/8706-56-45066410
Xeon D-1513N/23/33/43/5306-56-55066510
Pentium N/J4xxx, Celeron N/J3xxx, Atom x5/7-E39xx06-5c-9506c93
Core Gen6; Xeon E3 v506-5e-3506e336
Atom Processor C Series06-5f-1506f101
Pentium Silver N/J5xxx, Celeron N/J4xxx06-7a-1706a101
Core Gen8 Mobile06-8e-9806e910
Core Gen7 Mobile06-8e-9806e9c0
Core Gen8 Mobile06-8e-a806eac0
Core Gen8 Mobile06-8e-b806ebd0
Core Gen8 Mobile06-8e-d806ed94
Core Gen7; Xeon E3 v606-9e-9906e92a
Core Gen8 Desktop, Mobile, Xeon E06-9e-a906ea22
Core Gen806-9e-b906eb02
Core Gen906-9e-c906ec22
Core Gen9 Mobile06-9e-d906ed22

Some of them are planned for the future.

Product NamesCPUIDPlatform ID
Intel® Atom® Processor C2750, C2730, C2550, C2530, C2350406D81
Intel® CoreTM Processor Extreme Edition i7-3960X, i7-3970X Intel® CoreTM Processor i7-3820, 3930K206D76D
Intel® Xeon® Processor E5-2620, E5-2630, E5-2630L, E5- 2640, E5-2650, E5-2650L, E5-2660, E5-2667, E5-2670, E5- 2680, E5-2690206D66D
Intel® Xeon® Processor E5-1428L, E5-1620, E5-1650, E5- 1660, E5-2403, E5-2407, E5-2418L, E5-2420, E5-2428L, E5-2430, E5-2430L, E5-2440, E5-2448L, E5-2450, E5- 2450L, E5-2470, E5-2603, E5-2609, E5-2620, E5-2630, E5- 2630L, E5-2637, E5-2640, E5-2643, E5-2648L, E5-2650, E5-2650L, E5-2658, E5-2660, E5-2665, E5-2667, E5-2670, E5-2680, E5-2687W, E5-2690, E5-4603, E5-4607, E5-4610, E5-4617, E5-4620, E5-4640, E5-4650, E5-4650L Intel® Pentium® Processor 1405206D76D
Intel® Atom® Processor Z3770, Z3740, Z3770D, Z3740D, Z3770, Z3740, Z3680, Z3770D, Z3740D306732
Intel® Pentium® Processor J2900, J2850 Intel® Pentium® Processor N3520, N3510 Intel® Celeron® Processor J1900, J1850, J1800, J1750 Intel® Celeron® Processor N2920, N2910, N2820, N2815, N2810, N2806, N2805306730C
Intel® Pentium® Processor J2900, J2850 Intel® Pentium® Processor N3520, N3510 Intel® Celeron® Processor J1900, J1850, J1800, J1750 Intel® Celeron® Processor N2920, N2910, N2820, N2815, N2810, N2806, N2805306730C

To determine the CPUID, use the command:

# a=$(head -3 /proc/cpuinfo | tail -1 | awk '{print $4}'); b=$(head -4 /proc/cpuinfo| tail -1 | awk '{print $3}'); c=$(head -6 /proc/cpuinfo| tail -1 | awk '{print $3}'); printf "%02x-%02x-%02x\n" $a $b $c

To determine the CPUID in Intel format, use the command:

# a=$(head -3 /proc/cpuinfo | tail -1 | awk '{print $4}'); b=$(head -4 /proc/cpuinfo| tail -1 | awk '{print $3}'); c=$(head -6 /proc/cpuinfo| tail -1 | awk '{print $3}'); cpuid=$(printf "%02x-%02x-%x" $a $b $c); printf ${cpuid:3:1}${cpuid:0:2}${cpuid:4:1}${cpuid:6:2}"\n"

There are several CPUs which won’t have updated microcode. See the detailed list here: https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf.

For these processors, it is recommended to disable Hyper-Threading.

How Hyper-Threading disabling can impact performance

We have not received any reports about the performance impact from these MDS mitigations. However, people from RedHat report that there is, and that the “impact will be felt more in applications with high rates of user-kernel-user space transitions. For example, system calls, NMIs, and interrupts.”

They have conducted several tests to evaluate the impact on the following workloads:

  • Applications that spend a lot of time in user mode tended to show the smallest slowdown, usually in the 0-5% range.
  • Applications that did a lot of small block or small packet network I/O showed slowdowns in the 10-25% range.
  • Some microbenchmarks that did nothing other than enter and return from user space to kernel space showed higher slowdowns.

As RedHat specialists have said, “MDS mitigation can be fully enabled, with SMT also disabled by adding the “mds=full,nosmt” flag to the kernel boot command line.
MDS mitigation can be fully disabled by adding the “mds=off” flag to the kernel boot command line.
There is no way to disable it at runtime.”

More reading

Beta: LVE Manager, LVE-Utils, CageFS, LibLVE, and ...
CloudLinux OS Feature Survey - CLOSED
 

Comments 5

Guest - Jeff on Friday, 24 May 2019 20:52

I'm confused by the microcode availability as the Intel chart doesn't match the list above.

Intel's Guidance PDF shows many CPUs in green that are not listed above.
https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf

Are the green "production" microcode updates in the Intel guidance available, or are those the ones being worked on now? It's confusing because intel uses the words "available or planned" at the start and then in the chart uses "production or planned". I thought production status meant available.

I'm confused by the microcode availability as the Intel chart doesn't match the list above. Intel's Guidance PDF shows many CPUs in green that are not listed above. https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf Are the green "production" microcode updates in the Intel guidance available, or are those the ones being worked on now? It's confusing because intel uses the words "available or planned" at the start and then in the chart uses "production or planned". I thought production status meant available.
Ivan Zhmud on Friday, 24 May 2019 21:35

Hello, Jeff!
The table above should the same as the Intel's table in pdf. Just in our table is used classic cpuid and in PDF is used another its own format of cpuid. Also Intel's table content the same cpuid in several rows. Anyway, you can be guided by the Intel table. How to define cpuid in Intel format you can find in our article.
Cpuid in Intel's table with production status "production" means that Intel has completed all validation and is authorizing customers to use this MCU in a production environment i.e. it is already available for update.
Planned – Intel is planning on releasing a MCU at a future date.

Please let us know if you will have any questions. Thank you.

Hello, Jeff! The table above should the same as the Intel's table in pdf. Just in our table is used classic cpuid and in PDF is used another its own format of cpuid. Also Intel's table content the same cpuid in several rows. Anyway, you can be guided by the Intel table. How to define cpuid in Intel format you can find in our article. Cpuid in Intel's table with production status "production" means that Intel has completed all validation and is authorizing customers to use this MCU in a production environment i.e. it is already available for update. Planned – Intel is planning on releasing a MCU at a future date. Please let us know if you will have any questions. Thank you.
Guest - Jeff on Saturday, 25 May 2019 19:54

Thanks. However, on multiple E3 and E5 cpus:

Yum log shows
May 16 02:23:26 Updated: 2:microcode_ctl-1.17-33.11.el6_10.x86_64

Server rebooted after microcode update and new kernel installed

DMESG still reports
MDS: Vulnerable: Clear CPU buffers attempted, no microcode

https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf says "Production" for cpus in question

Why is DMESG showing "no microcode" for the E3 and E5 cpus even though they are green and labeled "production" in the intel guidance pdf?

Thanks. However, on multiple E3 and E5 cpus: Yum log shows May 16 02:23:26 Updated: 2:microcode_ctl-1.17-33.11.el6_10.x86_64 Server rebooted after microcode update and new kernel installed DMESG still reports MDS: Vulnerable: Clear CPU buffers attempted, no microcode https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf says "Production" for cpus in question Why is DMESG showing "no microcode" for the E3 and E5 cpus even though they are green and labeled "production" in the intel guidance pdf?
Inessa Atmachian on Monday, 27 May 2019 15:35

Hi Jeff,
We are sorry, but we are still investigating the problem.
We will keep you with updates in this blog post.

Hi Jeff, We are sorry, but we are still investigating the problem. We will keep you with updates in this blog post.
Ivan Zhmud on Thursday, 30 May 2019 14:01

Hello Jeff, we've released the new kernel with fixed checker which should show the correct status.
You can find details here https://www.cloudlinux.com/cloudlinux-os-blog/entry/beta-cloudlinux-7-and-cloudlinux-6-hybrid-kernel-updated-1-20
After updating to the new kernel and reboot you should see the correct status.
We strongly recommend you to upgrade your kernel to the latest version. This allows to switch on anti-MDS protection even if your current microcode is new enough.

Hello Jeff, we've released the new kernel with fixed checker which should show the correct status. You can find details here https://www.cloudlinux.com/cloudlinux-os-blog/entry/beta-cloudlinux-7-and-cloudlinux-6-hybrid-kernel-updated-1-20 After updating to the new kernel and reboot you should see the correct status. We strongly recommend you to upgrade your kernel to the latest version. This allows to switch on anti-MDS protection even if your current microcode is new enough.
Already Registered? Login Here
Guest
Monday, 26 August 2019

Captcha Image