Let us subject MDS vulnerability to the glare of truth

Ok, let me throw a little light on the last release of the CloudLinux 7 and CloudLinux 6 kernel with the MDS vulnerability patch.

MDS vulnerability explanation

In the last three days, we’ve received a whole bunch of questions like Should I disable Hyper-Threading or not? and How Hyper-Threading disabling can impact performance? So, here we are with some important information about the point.

But what is the problem? CPU has two execution threads per physical core. Both threads share the same resources inside the CPU. It means sibling cores can see the same data as the primary core can.

So what?

The problem involves different attacks:

• Kernel — user space attack.
• Userspace — userspace attack for threads running on the same physical core.
• Virtual Machine — Virtual Machine attack.

Different attack faces need different ways to mitigate it.

1. If you have a trusted user space, CPU buffers need to be flushed at the exit from the kernel, so the application isn’t able to see kernel data. That's what a microcode update does–it provides the ability to make it flush. If the CPU is supported by the microcode update, you can see a message:
   "Mitigation: Clear CPU buffers"
   OR
   "Mitigation: Clear CPU buffers; SMT vulnerable" in the /sys/devices/system/cpu/vulnerabilities/mds file or by running dmesg |grep MDS.
2. Virtual machine to virtual machine (VM) attack is different to the previous one. Two VMs can share the same CPU core so they can share CPU data. The kernel adds a CPU buffer flush in this case, in the same way that point one does.
3. But, resources aren’t completely isolated between cores in a package. The primary core and siblings share some resources at run time. So different applications on the Host or different applications in different VMs can access the same data. Intel CPUs have different protections against this type of attack. Some CPUs have little protection and others none at all. If you want to guarantee this attack will never happen, you can add “,nosmt” string to the end of the mds parameter.
   For example: mds=”full,nosmt”. This will provide an additional check and enable Hyper-Threading only if this is safe.
4. You will see a message:
   "Mitigation: Clear CPU buffers"
   in the system file or in the dmesg output. Currently, only ATOM series CPUs have this protection.

The problem can be mitigated by a CPU scheduler change. The scheduler must avoid balance loading between vCPUs, but this is a very large change and it is not available for the Linux kernel yet.

What CPUs can have their microcode updated?

Intel doesn’t provide a microcode update for all CPU’s. Only some new ones can be updated at the moment.

Some of them are planned for the future.

To determine the CPUID, use the command:

# a=$(head -3 /proc/cpuinfo | tail -1 | awk '{print $4}'); b=$(head -4 /proc/cpuinfo| tail -1 | awk '{print $3}'); c=$(head -6 /proc/cpuinfo| tail -1 | awk '{print $3}'); printf "%02x-%02x-%02x\n" $a $b $c

To determine the CPUID in Intel format, use the command:

# a=$(head -3 /proc/cpuinfo | tail -1 | awk '{print $4}'); b=$(head -4 /proc/cpuinfo| tail -1 | awk '{print $3}'); c=$(head -6 /proc/cpuinfo| tail -1 | awk '{print $3}'); cpuid=$(printf "%02x-%02x-%x" $a $b $c); printf ${cpuid:3:1}${cpuid:0:2}${cpuid:4:1}${cpuid:6:2}"\n"

There are several CPUs which won’t have updated microcode. See the detailed list here: https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf.

For these processors, it is recommended to disable Hyper-Threading.

How Hyper-Threading disabling can impact performance

We have not received any reports about the performance impact from these MDS mitigations. However, people from RedHat report that there is, and that the “impact will be felt more in applications with high rates of user-kernel-user space transitions. For example, system calls, NMIs, and interrupts.”

They have conducted several tests to evaluate the impact on the following workloads:

• Applications that spend a lot of time in user mode tended to show the smallest slowdown, usually in the 0-5% range.
• Applications that did a lot of small block or small packet network I/O showed slowdowns in the 10-25% range.
• Some microbenchmarks that did nothing other than enter and return from user space to kernel space showed higher slowdowns.

As RedHat specialists have said, “MDS mitigation can be fully enabled, with SMT also disabled by adding the “mds=full,nosmt” flag to the kernel boot command line.
MDS mitigation can be fully disabled by adding the “mds=off” flag to the kernel boot command line.
There is no way to disable it at runtime.”

More reading

• You can find a complete review of the MDS vulnerability from RedHat here: https://access.redhat.com/security/vulnerabilities/mds.
• For the performance impact of disabling Hyper-Threading, see the “Disabling Hyper-Threading” section at https://access.redhat.com/security/vulnerabilities/L1TF-perf.
• If you are using KernelCare, please visit https://blog.kernelcare.com/zombieload where you can find the MDS vulnerability patch release schedule, instructions on how to mitigate the MDS vulnerability, and also watch the video with insights regarding MDS from our CEO, Igor Seletskiy.