CloudLinux - CloudLinux Blog - Issues with CloudLinux OS 7.7 update on systems with kernel v3.10.0-862 or lower
CloudLinux OS Blog

By accepting you will be accessing a service provided by a third-party external to https://www.cloudlinux.com/

Featured 

Issues with CloudLinux OS 7.7 update on systems with kernel v3.10.0-862 or lower

CL_OS_kernel_warning

Issue description

The recent CloudLinux OS 7.7 update includes the iproute-4.11.0-25.el7[.cloudlinux] package that is not working correctly on systems with a 3.10.0-862 or lower kernel version.
If you apply the update and reboot the server, the network will probably go down.

Issue solution

The solution is to upgrade to the iproute-4.11.0-25.el7.cloudlinux.1 package by running the command:

yum upgrade iproute*

Additional notes

Kernels newer than 3.20.0-862 do not face the issue with the iproute-4.11.0-25.el7[.cloudlinux] package.

Explanations and apologies

We are very sorry about this issue.

We knew that the new iproute-4.11.0-25.el7 package is not compatible with kernels lower than 3.10.0-862 and mentioned it in our release announcement and took the appropriate steps: we added conflicts to the new packages and tested the update on the typical configurations.
But, unfortunately, we did not take into account all cases and some of our clients faced issues after the update.
Now, we are investigating the issue and releasing the hot-fix which is actually a downgrade to the previous stable package.

Once again, please accept our sincerest apologies for this problem and be assured that we have a plan for preventing this from happening again.

Beta: kernel module for CloudLinux 7 hybrid update...
Beta: Alt-PHP updated
 

Comments 19

Guest - Andrew Brown on Wednesday, 04 September 2019 11:39

So how do we run a patch on a virtual machine that now has no network connection??

Is there any file level amendement we can make as I can mount the drive on another machine to access, but I cant run it as the OS> as it has no network

So how do we run a patch on a virtual machine that now has no network connection?? Is there any file level amendement we can make as I can mount the drive on another machine to access, but I cant run it as the OS> as it has no network
Inessa Atmachian on Wednesday, 04 September 2019 12:18

Hi Andrew!
Here is a quick instruction:
1. Boot on another machine (recovery, LiveCD, etc).
2. Mount affected system into the directory including /proc, /dev, etc.
3. Download https://repo.cloudlinux.com/cloudlinux/7/os/x86_64/Packages/iproute-4.11.0-25.el7.cloudlinux.1.x86_64.rpm package into that directory.
4. Chroot into the directory from step 2.
5. Install the package and reboot

Hi Andrew! Here is a quick instruction: 1. Boot on another machine (recovery, LiveCD, etc). 2. Mount affected system into the directory including /proc, /dev, etc. 3. Download https://repo.cloudlinux.com/cloudlinux/7/os/x86_64/Packages/iproute-4.11.0-25.el7.cloudlinux.1.x86_64.rpm package into that directory. 4. Chroot into the directory from step 2. 5. Install the package and reboot
Inessa Atmachian on Wednesday, 04 September 2019 12:30

If you have any doubts, please contact our support team at https://cloudlinux.zendesk.com/hc/requests/new

If you have any doubts, please contact our support team at https://cloudlinux.zendesk.com/hc/requests/new
Guest - Andrew Brown on Wednesday, 04 September 2019 12:36

OK, tried this.

1. Booted another machine and attached the bad OS drive and mounted it to /odin
2. Can see the content of the back OS disk with /proc, /dev/ boot, /root etc
3. Got the file and CHMOD 755 ok
4. CHroot /odin. working ok
5. RPM -i Throwing "error: Failed to initialize NSS library"

Advice please, still have multiple servers offline due to this mistake.

OK, tried this. 1. Booted another machine and attached the bad OS drive and mounted it to /odin 2. Can see the content of the back OS disk with /proc, /dev/ boot, /root etc 3. Got the file and CHMOD 755 ok 4. CHroot /odin. working ok 5. RPM -i Throwing "error: Failed to initialize NSS library" Advice please, still have multiple servers offline due to this mistake.
Guest - Georgi Vasilev on Wednesday, 04 September 2019 13:00

Hi, you should mount bind the dev directory
mount -o bind /dev /odin/dev/

Hi, you should mount bind the dev directory mount -o bind /dev /odin/dev/
Guest - Andrew Brown on Wednesday, 04 September 2019 13:27

Thank you...

These steps got our servers back online....

Now for the debrief and find out how to disable ALL auto updates!

We all make mistakes, but this was a big one and CloudLinux def dropped the ball!

Thank you... These steps got our servers back online.... Now for the debrief and find out how to disable ALL auto updates! We all make mistakes, but this was a big one and CloudLinux def dropped the ball!
Diogo on Wednesday, 04 September 2019 14:47

Hi all,

The solution we found was download the RPM of the new kernel version ( https://www.repo.cloudlinux.com/cloudlinux/7/os/x86_64/Packages/kernel-3.10.0-962.3.2.lve1.5.26.2.el7.x86_64.rpm ) and move it to a pen drive.

Then we started the server, mounted the pen drive and we've installed the kernel using the rpm -ivh command.

After the reboot the network devices will start, and you can run the yum update.

In case you have VPS servers and you're able to manage it, you can create an ISO from the pen drive you used to copy the rpm, upload that ISO to your datastore/virtualization environment and then you can mount it on the VPS server to install the kernel manually.

Hi all, The solution we found was download the RPM of the new kernel version ( https://www.repo.cloudlinux.com/cloudlinux/7/os/x86_64/Packages/kernel-3.10.0-962.3.2.lve1.5.26.2.el7.x86_64.rpm ) and move it to a pen drive. Then we started the server, mounted the pen drive and we've installed the kernel using the rpm -ivh command. After the reboot the network devices will start, and you can run the yum update. In case you have VPS servers and you're able to manage it, you can create an ISO from the pen drive you used to copy the rpm, upload that ISO to your datastore/virtualization environment and then you can mount it on the VPS server to install the kernel manually.
Guest - Ryan Smith on Wednesday, 04 September 2019 15:25

What's the long term plan for the iproute package for users of older kernels? Will you make it so older kernels will always be compatible with this package going forward? We have iproute excluded in /etc/yum.conf now and don't want to encounter any expected issues down the road.

What's the long term plan for the iproute package for users of older kernels? Will you make it so older kernels will always be compatible with this package going forward? We have iproute excluded in /etc/yum.conf now and don't want to encounter any expected issues down the road.
Guest - Ryan Smith on Wednesday, 04 September 2019 23:59

Are there any known issues with Imunify360 under CloudLinux 7.7? I'm seeing a huge increase in incidents after upgrading to CL 7.7 and I'm starting to wonder if Imunify360 is actually blocking the traffic now or not.

Are there any known issues with Imunify360 under CloudLinux 7.7? I'm seeing a huge increase in incidents after upgrading to CL 7.7 and I'm starting to wonder if Imunify360 is actually blocking the traffic now or not.
Guest - Elliott Willink on Thursday, 05 September 2019 01:36

Hope this helps someone, but before you try offline copying RPM's, booting a much older kernel (possibly even rescue/CL 7.2 can't remember exactly which one worked) brought the network back up. Then a "yum downgrade iproute" (Did this twice to get back to the previous version, looked like it had updated twice?) and reboot.

Once it booted back into 7.7 I did a yum update kernel, (reboot just to be sure), yum update iproute, reboot and now everything is up to date and happy.

Hope this helps someone, but before you try offline copying RPM's, booting a much older kernel (possibly even rescue/CL 7.2 can't remember exactly which one worked) brought the network back up. Then a "yum downgrade iproute" (Did this twice to get back to the previous version, looked like it had updated twice?) and reboot. Once it booted back into 7.7 I did a yum update kernel, (reboot just to be sure), yum update iproute, reboot and now everything is up to date and happy.
Guest - Pablo Gómez on Thursday, 05 September 2019 09:51

Well, I had console access to the broken server so I fixed it by doing a manual config of the interface like:

ifconfig {interface} {IP/Mask}

and adding a default gateway:

route add default gw {GW_IP}

Now I have at less some basic connectivity to work with "yum":

yum update iproute*

And that was it!

Then rebooted and made an update:

yum clean all; yum update

Hope someone will find it useful.

Well, I had console access to the broken server so I fixed it by doing a manual config of the interface like: ifconfig {interface} {IP/Mask} and adding a default gateway: route add default gw {GW_IP} Now I have at less some basic connectivity to work with "yum": yum update iproute* And that was it! Then rebooted and made an update: yum clean all; yum update Hope someone will find it useful.
Guest - Pedro on Thursday, 05 September 2019 12:06

Pablo Gómez - thank you! It helps with Hetzner DC:

1) order rescue - vkvm
2) login in vkvm with root
3) ifconfig {interface} {IP}
4) route add default gw {GW_IP}
5)

Pablo Gómez - thank you! It helps with Hetzner DC: 1) order rescue - vkvm 2) login in vkvm with root 3) ifconfig {interface} {IP} 4) route add default gw {GW_IP} 5)
Inessa Atmachian on Thursday, 05 September 2019 12:54

Hi Ryan,
We are investigating the possibility to add support for old kernels to the iproute package. But, for now, we have nothing specific to say. In any case, we will find the solution and will prevent this situation from happening again.

Hi Ryan, We are investigating the possibility to add support for old kernels to the iproute package. But, for now, we have nothing specific to say. In any case, we will find the solution and will prevent this situation from happening again.
Guest - Adam on Monday, 09 September 2019 02:44

@Guest - Andrew Brown
When you installed rpm and rebooted, you meant to disconnect mounted drive and reboot it as primary drive. Yeah?

@Guest - Andrew Brown When you installed rpm and rebooted, you meant to disconnect mounted drive and reboot it as primary drive. Yeah?
Inessa Atmachian on Monday, 09 September 2019 15:45

Adam, yes, you should unmount a drive before reboot.

Adam, yes, you should unmount a drive before reboot.
Guest - John on Friday, 13 September 2019 07:13

Issue not yet resolved!!!

Issue not yet resolved!!!
Dmitriy Serafin on Friday, 13 September 2019 07:48

Hi John,

If you still experience some issues with the CloudLinux OS 7.7 update, could you please submit a ticket to us here?
https://cloudlinux.zendesk.com/hc/en-us/requests/new
We'll be glad to help.

Thank you.

Hi John, If you still experience some issues with the CloudLinux OS 7.7 update, could you please submit a ticket to us here? https://cloudlinux.zendesk.com/hc/en-us/requests/new We'll be glad to help. Thank you.
Guest - vladim on Tuesday, 17 September 2019 08:18

Hello,
I have
3.10.0-962.3.2.lve1.5.25.8.el7.x86_64 kernel.

It's newer than .862. Is it safe to reboot?

Hello, I have 3.10.0-962.3.2.lve1.5.25.8.el7.x86_64 kernel. It's newer than .862. Is it safe to reboot?
Inessa Atmachian on Tuesday, 17 September 2019 10:19

Hello Vladim,

Yes, you can reboot.

Hello Vladim, Yes, you can reboot.
Already Registered? Login Here
Guest
Sunday, 09 August 2020

Captcha Image