Blog

New beta version of CageFS 3 available. New version has following fixes and improvements following issues:

• Fixes race condition that would cause serving wrong /etc/passwd or /etc/group file
  
• Ability for sendmail to send mail when eximmailtrap is enabled
  
• White listed user/groups that can see additional users in password and group files
  
• Fixed permissions for cagefs-update.log
  

To update to the new version, please run:
# yum update cagefs --enablerepo=cloudlinux-updates-testing
# cagefsctl --remount-all

New beta version of CageFS is available. The new version fixes multiple crashes in CageFS FUSE, that were causing spontaneous error 500. In addition it adds full support for LiteSpeed/cPanel and for InterWorx control panel. 

Other multiple bug fixes and improvements, including:

• Issues when sending mail via exim were resolved 
  
• Issues related to cPanel CageFS plugin were resolved
  
• Improved detection of LiteSpeed, with automatic configuration of CageFS settings in LiteSpeed
  
• CageFS jail API was improved for better backward compatibility.
  

To update:
# yum update cagefs cagefs-fuse lve lve-wrappers pam_lve --enablerepo=cloudlinux-updates-testing

If you have LiteSpeed installed:
# cagefsctl --create-mp
# cagefsctl --remount-all

“top” is one of the most useful command in any sys admin arsenal. It is amazing tool that lets you find an issue with a server. It is by far the best “birds eye” view to the server that you can get. Anytime someone has a problem with server performance or stability, the first thing I do, I run top. While the same information can be gathered by running multitude of other commands – having it all in one place, refreshing constantly, is very helpful. This guide will walk you through top in a typical shared hosting settings.
Lets take a look:

top - 02:12:43 up 1:12, 5 users, load average: 3.93, 4.29, 12.40
Tasks: 269 total, 6 running, 262 sleeping, 0 stopped, 1 zombie
Cpu(s): 19.4%us, 11.4%sy, 0.0%ni, 49.0%id, 19.6%wa, 0.2%hi, 0.3%si, 0.0%st
Mem: 8028612k total, 4529728k used, 3498884k free, 389300k buffers
Swap: 4192924k total, 181872k used, 4011052k free, 2663176k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
47660 mysql 6 -10 1052m 233m 4784 S 2.8 3.0 1:11.31 mysqld
47374 nobody 15 0 305m 141m 2836 S 0.0 1.8 0:03.71 httpd
49478 nobody 15 0 304m 141m 2864 S 0.0 1.8 0:00.34 httpd
49781 nobody 15 0 305m 141m 2320 S 0.0 1.8 0:03.09 httpd
39462 nobody 15 0 305m 140m 2408 S 0.0 1.8 0:07.94 httpd
50142 nobody 18 0 305m 140m 2320 S 1.4 1.8 0:02.84 httpd 


Right away we can see that the server was recently rebooted - just an hour ago due to this info (top - 02:12:43 up 1:12), and that load is subsiding (load average: 3.93, 4.29, 12.40). Now, everyone knows that “high” load average is bad, and low load average is good. This three values represent load averages for 1, 5 and 15 minutes correspondingly. The general rule of thumb is that if load average is higher then number of cores on a server – it is a bad thing. If it is less – it is ok. The things are bit more complex on CloudLinux servers, due to CPU throttling – but lets continue discussing 'top' instead.

So, once we take a look at load averages, if they are high – we know that something might be going on with the server. More often then not – top can help you pinpoint the issue.

Here is another example:
top - 01:49:59 up 49 min, 5 users, load average: 18.02, 32.55, 34.91
Tasks: 269 total, 1 running, 268 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.4%us, 0.6%sy, 0.0%ni, 16.7%id, 82.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8028612k total, 2268460k used, 5760152k free, 88348k buffers
Swap: 4192924k total, 731688k used, 3461236k free, 309828k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8842 mysql 11 -10 1274m 178m 4076 S 0.0 2.3 0:46.40 mysqld
27379 root 18 0 232m 123m 3768 D 0.0 1.6 0:00.93 httpd
22731 a19a5a1 16 0 184m 53m 3692 D 0.0 0.7 0:01.13 php
23384 a19a5a1 16 0 184m 53m 3900 D 0.0 0.7 0:01.05 php
19443 a1a16ala 18 0 209m 51m 2772 D 0.0 0.7 0:01.52 php
21254 a19a5a1 16 0 184m 51m 3692 D 0.0 0.7 0:01.26 php
24659 a19a5a1 18 0 182m 48m 3612 D 0.0 0.6 0:00.70 php
20674 a1p16ple 16 0 168m 48m 3604 D 0.3 0.6 0:01.57 php
20796 user 16 0 179m 48m 3540 D 0.7 0.6 0:00.80 php
21179 user 18 0 179m 48m 3540 D 0.0 0.6 0:00.70 php

Load averages are high, and you can see 82.2%wa. “wa” -- stands for IO Wait – the % of cpu time spent by CPU waiting for IO (usually disks) to respond. Lets look at the topmost process in the “top” results: it is mysql. This is pretty easy give away, that something is going on with MySQL. The processes are sorted by CPU usage, and MySQL is IO intensive. So, if MySQL is actively using CPU, and IO usage is high – there is a high chance that the issue is with mysql

Running mysqladmin processlist can help you pinpoint what might be exact problem. Sometimes it might be corrupted database, sometimes – bad query... If you train your eye – you can find out the cause of the issue within seconds of looking at the results.

Here is another one:

01:15:52 up 15 min, 4 users, load average: 28.37, 20.02, 10.44
Tasks: 762 total, 1 running, 760 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.5%us, 1.8%sy, 0.0%ni, 0.0%id, 97.3%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 8028612k total, 7981536k used, 47076k free, 34512k buffers
Swap: 4192924k total, 2894460k used, 1298464k free, 272708k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23058 root 16 0 10476 1044 752 S 3.3 0.0 0:40.76 /usr/sbin/lveps -t
25203 root 18 0 39136 6000 1568 D 2.0 0.1 0:00.18 /usr/local/cpanel/bin/dcpumon
23048 root 15 0 13412 1716 748 S 1.0 0.0 0:12.96 top -c

io wait is at 97.3% , and at that level system would be hardly responsive. Pretty much all the memory taken and ~2.5GB of Swap is used. With a trained eye, you would see right away that “something” using up all the memory. Pressing SHIFT-M would sort processes by memory usage, and you will see which processes are eating up all the RAM, often providing you with enough info on what to do next.
Here is a bit more complex one:

top - 01:27:59 up 31 days, 22:31, 3 users, load average: 2.36, 2.33, 4.96
Tasks: 697 total, 1 running, 692 sleeping, 0 stopped, 4 zombie
Cpu(s): 2.1%us, 1.2%sy, 0.0%ni, 50.9%id, 45.5%wa, 0.1%hi, 0.1%si, 0.0%st
Mem: 4023964k total, 3917884k used, 106080k free, 298548k buffers
Swap: 8385920k total, 504136k used, 7881784k free, 1557260k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
950397 user123 16 0 183m 19m 7276 S 7.0 0.5 0:00.08 php
950396 user321 16 0 0 0 0 Z 4.4 0.0 0:00.05 php <defunct>
950312 nobody 15 0 216m 113m 2020 S 2.6 2.9 0:00.25 httpd
946406 nobody 15 0 217m 114m 2032 S 1.8 2.9 0:01.03 httpd
950314 nobody 18 0 217m 114m 1992 S 1.8 2.9 0:00.14 httpd
7166 root 10 -5 0 0 0 S 0.9 0.0 38:39.15 kondemand/2
8194 root 10 -5 0 0 0 S 0.9 0.0 67:35.63 kondemand/6
8195 root 10 -5 0 0 0 S 0.9 0.0 93:49.75 kondemand/7 



IO wait is pretty high (45.5%), swap is used – but not that much, just short of 1/2GB. It might appear that all the RAM is used up as well (3917884k used out of 4023964k total), but that is not the case.
The are about 1.5GB memory “cached”, and almost 300MB used for buffers. The “cached” memory, is memory used to speed up disk IO. The data is cached by linux kernel, so that instead of re-reading same file from disk, it would be taken directly from RAM. If kernel sees that there are not enough RAM – it would free up caches and buffers. In this case, kernel decided it is more efficient to save some of the processes to Swap (most likely those that were sleeping for a long time), than it is to purge cache. As long as swap usage doesn't change much within few minutes of time – the most likely reason for high IO wait is not Swap.

Looking at topmost processes in the list doesn't bring up anything that is easy to qualify as an issue. While IO bound processes are often CPU bound as well, it wasn't the case here. This is where you can use another good tool, iotop (can be installed via yum install iotop), that will show you processes that use up IO. In this particular case it was rsync causing high IO wait

Total DISK READ: 2.89 M/s | Total DISK WRITE: 212.68 K/s
951134 be/3 root 6.93 M/s 0.00 B/s 0.00 % 99.99 % rsync -rlptD --exclude=*/proc/* --delete /v~ /backup/cpbackup/daily/dirs/_var_lib_mysql_

Here are two more examples, where the issue at hand is not that clear cut, yet top still give enough info to go into right direction. First one is from my previous blog post: 

top – 2:15:42 up 48 min, 3 users, load average: 8.62, 4.78, 3.26
Tasks: 176 total, 15 running, 161 sleeping, 0 stopped, 0 zombie
Cpu(s) : 8.0%us, 4.5%sy, 0.1%ni, 3.2%id, 8.2%wa, 0.0%hi, 76.0%si, 0.0%st
Mem: 8031276k total, 7363928k used, 667348k free, 365104k buffers
Swap: 4194296k total, 91396k used, 4102900k free, 5077444k cached 


si or software interrupts where the issue. Once again, this helps rule out quite a few scenarios, and shows what to “google” for. In this case USB module http://www.cloudlinux.com/blog/clnews/109.php?sphrase_id=18641 was an issue

This issue I have seen first on AWS “micro” server (the top header is taken from one of forum post at amazon, I couldn't find my own), but I have also seen it recently at one of the customer's server running Xen.

top - 22:42:48 up 1 day, 1:56, 1 user, load average: 2.63, 2.21, 1.85
Tasks: 46 total, 5 running, 41 sleeping, 0 stopped, 0 zombie
Cpu(s): 27.3%us, 9.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.0%si, 62.0%st
Mem: 1747660k total, 1677908k used, 69752k free, 6780k buffers
Swap: 917496k total, 36k used, 917460k free, 1269348k cached

The key here is CPU 'stolen'
st - Steal Time. The amount of CPU ’stolen’ from this virtual machine by the hypervisor for other tasks (such as running another virtual machine) – a fairly recent addition to the top command, introduced with the increased virtualization focus in modern operating systems (man top)
In the last case, moving VM to another server with more resources solved the issue.
As you can see top is quite useful, if you cross reference all the data that it presents.

Since about 15 years ago, in addition to traditional shared and dedicated hosting, service providers started offering VPS hosting. VPS stands for Virtual Private Servers, the other acronym used for this is VDS (Virtual Dedicated Server). A VPS is something in between shared and dedicated hosting, closer to the 'dedicated' end of range.



A VPS is functionally equivalent to a separate physical server, is tuned to the individual customer's needs, has the privacy of a separate physical server, and so on. The trick is, multiple VPSes (tens of even hundreds of them) reside on a single real server, sharing its hardware resources, reducing the needs for hardware, rack space, electricity bills, thus lowering the price level for a customer.

Technically, a VPS functionality is implemented as an additional layer between the hardware and the software. Usually, a VPS is either a Virtual Machine (as in VMware, Xen, KVM) or a container (as in OpenVZ, Solaris Zones or LXC). Virtual Machines use hypervisor technology, and every VM-based VPS runs a full software stack, including the OS kernel and set of drivers for (virtual) hardware, the very same way your usual server runs.



Containers are a bit different, more light-weighted, since there is only one single OS kernel is running, and VPSes only run userspace software (i.e. no own kernels, drivers etc.). Now, if you have read up to this point and are not asleep, you deserve a small and nice reward: go get yourself a cup of coffee or whatever your favourite drink is. Got it? Let's continue.



Both containers and virtual machines have some overhead: this is the price one has to pay for splitting the piece of real hardware into multiple smaller pieces, getting some virtual hardware. This partitioning is done via isolation, and the isolation overhead is much less for containers than for VMs, but it's still there.

The other thing worth mentioning besides the isolation overhead is the importance of fair resource management. Indeed, since multiple VPSes share the same set of hardware resources (CPU, RAM, disk, networking), care has to be taken of how to divide this resources in a fair manner, so that no single VPS abuse a resource, say, by eating up all disk space or saturating the disk I/O channel.

Now, let's consider the ideal VPS solution, for which the isolation is bullet-proof (which almost true for VMs) and don't cost us more than a penny (which is almost true for containers), plus the resource management is just perfect, all ruled by no one less than king Solomon in all his great wisdom. Even with that ideal VPS solution in place, there are troubles around the corner.



First of all, every VPS runs (at least) the complete set of userspace software, which needs to be maintained. That is, unlike in shared hosting, you have to take care about software updates (if you care of security, for instance). You need to configure some auxiliary components like system logger or cron daemon. Basically, you need to be a full-scale sysadmin for that, or hire one, which is not that cheap at all. It could be useful if you are a sysadmin, so you can recompile PHP with your favorite options and patches (which is great fun!), but  if you're not...  than the need to manage your OS yourself is more of a burden, if not a nightmare.

Don't forget that every VPS needs a distinct public IP address. I hear you say "It's not my problem!", and I tend to agree, your HSP should take care about this. Still, considering a recent trend in IPv4 address space shortage, and less-than-warm acceptance of IPv6, this might be your problem, too.

Overall, most of the problems that you have with dedicated hosting also apply to VPS hosting. Dedicated hosting, though, is used by bigger web sites, which have proper sysadmin and other resources, and the knowledge needed to manage operating system instances. A VPS owner is usually not so rich and huge, but she faces the same set of problems as a dedicated guy.

So, which way to go if you want VPS benefits (like proper isolation and resource controlled by a Solomon) but don't like its maintenance burdens? Still don't know the answer? Oh, come on, it's pretty obvious! Go CLOUDLINUX! Amen.

It is finally here! CageFS 3.0 is ready for adventurous souls that want to put latest and greatest on their servers.Right now it is only available released it only for CloudLinux 5.x. CloudLinux 6.x support will be  coming soon. Here are some of the highlights of a new version:

• Better namespace handing, requiring only fraction of mount points comparing to CageFS 2.x
  
• CageFS-FUSE provides virtualized /etc & /var/log -- increasing security, and decreasing complexity of maintaining CageFS
  
• Caged directories are no longer visible in /proc/mounts, solving all related issues with cPanel
  
• Management plugins for cPanel and Plesk will be installed automatically (other control panels coming soon)
  
• Automatic detection of cPanel and Plesk (other control panels coming soon), with automatic configuration to adjust for the running system
  
• Improved command line tools
  

The current version should work with LiteSpeed, and or with custom control panels with some additional configuraiton.


More information on CageFS, how to install it or update from previous version can be found here:
http://www.cloudlinux.com/docs/cagefs/index.php

Just in time for a new year we have a new version of lve-stats. That version brings few enhancements and bug fixes. We have added admin & user styles for charting & lveinfo output. The new style will make sure that user will see CPU chart & output normalized relative to 100% of usage.

We also improved our integration with MySQL & PostgreSQL. You can now create /etc/sysconfig/lvestats.readonly and provide user that has read only (SELECT) access to history table in lvestats database. That file will be used by lveinfo & lvecharts to generate end user statistics. 

This way you can make /etc/sysconfig/lvestats readable only by root, preventing ability of user to insert or delete any info fromlvestats database. 
Please, note that this issue doesn't exist with SQLite that is being used by default.

Additionally, we fixed CPU reporting in CloudLinux 6. Before lve-stats would record CPU usage higher then it would be possible due to NCPU limit.

To install latest update, please run:
# yum update lve-stats --enablerepo=cloudlinux-updates-testing

New kernel 2.6.32-231.21.1.lve0.9.18 was moved to production. It includes following bug fixes:

• Rebase to 2.6.32-131.21.1 RHEL 6 update (RHSA-2011:1465)
  
• Multitude of fixes brought in by OpenVZ 042stab044.11 kernel update
  

To update

It was a regular Monday morning – busy as usual. Then one client popped up with a very interesting problem. His server became unresponsive lately, had very high load, and he was wondering why CloudLinux wasn't stopping the issue.

I quickly logged into the server (and the unresponsiveness become obvious right away) and ran top. I start with top pretty much every time someone has “overload” issue with the server. Running top was the right thing to do this time over as well as it gave me an idea where to look next. The si was at 70% -- something was wrong, really wrong

si stands for % of CPU used to handle software interrupts. On most servers you would rarely see si using more then 2 to 4% of CPU. Software interrupt is an asynchronous signal that needs to be handled by some code. They are normal, and happen all the time. For example, software interrupts happen on each timer tick or when network card receives a packet of data, and it needs software to process that data.

My next step was to see which software interrupts are the most frequent on this system, and might be causing the issue.
# cat /proc/interrupts 
           CPU0       CPU1       
  0: 1566845520      60143    IO-APIC-edge  timer
  1:          1          2    IO-APIC-edge  i8042
  8:          0          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:          1          3    IO-APIC-edge  i8042
 50:        226          0         PCI-MSI  hda_intel
169:          0          0   IO-APIC-level  uhci_hcd:usb5
209:          0          0   IO-APIC-level  uhci_hcd:usb4
217:          0          0   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2
225:      13475  111221010   IO-APIC-level  uhci_hcd:usb3, ata_piix
233:         62  327366496         PCI-MSI  eth0
NMI:     493362     580033 
LOC: 1566919097 1566920751 
RES:   27519611   15339092 
ERR:          0
MIS:          0

Now, the first column stands of IRQ (interrupt request) number,  CPU0, and CPU1 stand for number of times interrupt was handled by particular CPU.  The next column is type of the interrupt – which is not important in this case, and the last column are modules that are listening for the interrupt.

I knew that timer could be ignored, it increments on each clock tick.
The LOC stands for local timer – and can be ignored as well.
RES stands for Rescheduling interrupts – and it looked fine.

The other two IRQ numbers that were very active were 225 and 233.
225 was used by uhci_hcd:usb3, ata_piix while 233 was used by eth0.
This is a web server, so I expected lots of network traffic, and high IRQ activity for eth0 was normal.
IRQ 225 didn't look as good. uhci_hcd is used for USB, and ata_piix is your standard ATA hard disk. The disk activity (based on iotop and iostat output) wasn't that high, but the counter was increasing very fast. Could it be interrupt storm caused by some conflict between two devices?

Well, USB is not needed on a web server, so it was easy to test.
# rmmod uhci_hcd ohci_hcd ehci_hcd 
unloaded USB related modules, and now
# cat /proc/interrupts
 CPU0       CPU1       
  0: 1567154926      60143    IO-APIC-edge  timer
  1:          1          2    IO-APIC-edge  i8042
  8:          0          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:          1          3    IO-APIC-edge  i8042
 50:        226          0         PCI-MSI  hda_intel
225:      13475  111239531   IO-APIC-level  ata_piix
233:         62  327404778         PCI-MSI  eth0
NMI:     493550     580292 
LOC: 1567228505 1567230167 
RES:   27526034   15340183 
ERR:          0
MIS:          0

USB was no more. System became responsive, and si dropped to 3%, load average dropped as well. It was a conflict between USB and ATA. I added USB modules to blacklist so they wouldn't be loaded after reboot. That was done by adding following lines to /etc/modprobe.d/blacklist.conf
# disable usb
blacklist uhci_hcd
blacklist ohci_hcd
blacklist ehci_hcd

While the situation is highly unusual, and probably sign of faulty hardware or bios, it raised an interesting question if USB should be disabled on the server. For most web servers USB is not in use* anyway. Is there harm in having those modules loaded? First of all they take up a little memory. Yet, on some motherboards (like in this example)  they might share same IRQ number with another device, and that is bad. It means each time such interrupt happens, both interrupt handlers will wake up and try to decide which will handle it. That wastes CPU cycles. It might not be as bad as in this case (as this one for caused by some hardware issue), but it is still a waste. 

It makes a lot of sense to disable any hardware not in use – it might give some extra breathing space for the server.


* If you use KVM - it might use USB for console access, disabling USB is not recommended in this case.

There's not a big secret to anyone nowadays, that every server has at least one IP address (which is also tied to a domain name). As simply as that, if you want to access the site, you need to know either it's domain name or an IP address.

A web server is identified by an IP address and a port name (which defaults to 80 for http and 443 for https). But what if we want to host more than one web site? Ages ago it would mean having multiple IP addresses on a server, one per web site, or using non-standard port numbers (like http://yoursite.com:88). Both solutions are no bargain: an IP address space is limited, and non-standard port number means a user should memorize it.

The glorious solution was described in version 1.1 of http protocol standard, and it is pretty simple. A web browser must add a “Host:” header to any http request, with its value set to a hostname part of an address hat user has typed in a location field. Now, if you want to browse www.mysite.com the request containing “Host: www.mysite.com” header  is sent to a specific IP address. In the end you use the same IP for www.othersite.com, too, or maybe hundreds of such sites. Yes, this is how shared web hosting works. There is usually one web server listening to one IP/port combination, which reads the "Host:" header and supply the content based on its value.



A small talk nerdy to me part for those who are eager to know all the small details. In shared hosting there’s one copy of the Apache web server running plus, let's say, PHP installed either as mod_php, or as a CGI executable. That one copy of Apache handles all the incoming HTTP requests for all the websites that are sharing the server: receiving, processing, sending back a static file (say, an image), running scripts, logging and so on. When Apache is running in this way it runs as a single user, usually www or apache. At the moment Apache + mod_php is the fastest way of runnin PHP scripts, as the PHP interpretor is persistent and in-process, leaving two others (CGI + SuEXEC and  FastCGI per user) far behind.

All the above might seem to be a brilliant scheme from the first look, but there are a number of drawbacks. Since it's just one http server instance serving multiple sites, a load on one of such sites will cause a slowdown of all the others, and that's just a mild scenario,  more real-world one would be a DDoS attack on a site, rendering not only this one, but all the others dead and non-responsive. This is because there is no way to control resources (CPU time, disk and network bandwidth, used RAM etc.) on a per web site basis.

Another problem with shared hosting is security, or rather a lack of it. In a shared environment, different web sites are just different directories, usually there's little or no separation between those. All the web sites accounts and Apache user are members of the same group, so the thing is that Apache user has read (and sometimes even write) access to all the files of all the websites residing on the same server. If one of the sites is hacked, most probably others are vulnerable, too. More than that, evil hacker can just easily become a customer, get a legitimate account on the box, and then just upload PHP scripts to access the other websites hosted on the box (including all the PHP scripts that contain usernames and passwords for all the MySQL databases. Already having a panic attack? Clutching your head and wondering, what to do?



The simple answer lately was 'go for dedicated hosting' (which means having a distinct physical server for every given web site). Thanks to modern technologies and dark powers of electricity, these days those servers can be virtual dedicated ones, just multiple virtual machines running on top of one physical server (yeah, remember those VPSes you heard of?). This or that way, this looks very much like a trip back to 1990s (along with NAFTA, Hubble telescope, MMORPGs and Red Hot Chili Peppers): single IP per web server, administrating/management nightmares, and so on. Do we REALLY have to choose between good old one server per web site , or a shared hosting?

In CloudLinux, we can have the best of both worlds. It is shared hosting, but with decent resource controls applied to make those web sites more independent of each other. In a nutshell, the CloudLinux technology makes web server processes that handle the request of a specific web site be bound by resource constraints specified for this very site. Remember that “Host:” header which tells a web server what site a client wants? Whenever Apache in CloudLinux sees that, it jumps into so-called LVE (lightweight virtual environment) dedicated to this specific site. Now, with the help of CloudLinux, you can host multiple sites in a same shared manner, but have things about as safe an in case of dedicated hosting, specifying priorities and resource constraints for each of those sites. Now, if one site has a huge increase in traffic, it won't affect the others and make them slow and unresponsive (if a system is configured properly). You need to give it a try. You will be amazed. I guarantee it.

New version will log all processes currently in memory, when there is not enough memory. This should provide better information on what is causing server to overload.
To install:
To update:

More info on memmonitor: http://www.cloudlinux.com/docs/memmonitor.php

Beta version of LVE Manager for ISPmanager is available. It provides web UI to manage, monitor and adjust LVE settings for ISPmanager control panel.
More info can be found here: http://www.cloudlinux.com/docs/ispmanager_lvemanager.php

New kernel 2.6.18-374.3.1.el5.lve0.8.44 is available. It includes following bug fixes and new features:

• Rebased to upstream kernel 028stab094.3 including security and bug fixes RHSA-2011:1212
• Kernel panic fix for lve_list_next race condition
• Rebase to upstream 28stab093.2 kernel (security and bug fixes RHSA-2011:1065)
• Ability to change NCPU on the fly
• Ability to destroy LVE on the fly (via API)
• Two modes to calculate load averages with ability to switch between them
• Fix for OOM/hanged task issue
• IO Priorities

Code

# yum install kernel

If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

To change NCPU on the fly, use:

Code

# lvectl set LVE_ID --ncpu N --force

--force will cause NCPU change on the fly. Please, don't use that option with kernels prior to lve0.8.42, as it can crash the system

Load Averages
CloudLinux has a  modified way to calculated load averages, as processes can wait on CPU because of LVE limits, and not because of lack of CPU resources. Previously, our LA algorithm was ignoring uninterruptable processes. You can now switch to another LA algorithm that accounts for uninterruptable processes by running:

Code

sysctl -w kernel.full_loadavg=1

Switching to that mode will cause higher load averages during high IO activity intervals. This should be useful on cPanel servers that have high IO Wait without high load average

You can always switch back by running:

Code

sysctl -w kernel.full_loadavg=0

IO Priorities

While we still planning to release IO limits by the end of this year, this release introduces IO priorities. Each LVE is set to have IO priority of 100 by default (highest possible).
You can lower that priority, causing particular LVE to be de-prioritizes IO wise. This means that if there is lack of resources on the server, that LVE will get less IO operations then other LVEs.
IO Priorities work only with CFQ IO scheduler.

LVE Manager plugin for InterWorx is available. It fixes the issue when resellers were able to see LVE Manager.
To update:
Special thanks to InterWorx team for providing the patch.

New kernel  2.6.18-374.el5.lve0.8.43 is available from our cloudlinux-updates-testing repository. It includes everything from lve0.8.42 kernel as well as a bug fix for a bug that was present since lve0.8.42 kernel

• Fix for lve_list_next race condition

# yum install kernel-2.6.18-374.el5.lve0.8.43 --enablerepo=cloudlinux-updates-testing
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

I am happy to announce the release of CloudLinux OS 6.1. The new version comes with 2.6.32 kernel, updated apache & php packages and brings us in line with EL distribution. All features available in CloudLinux OS 5.x should be available in CloudLinux OS 6.1
New ISO images with CloudLinux OS 6.1 are available, and conversion scripts are compatible with CentOS 6.0 and RHEL 6.1. You can find images and conversion scripts here:

http://www.cloudlinux.com/downloads/

New kernel  2.6.32-231.6.1.lve0.9.8 is available. The kernel fixes the issue with low default shared memory limits inside LVE that was causing issues for eAccelerator.

To update:

We are happy to announce CloudLinux 6.1 Beta. This release brings us in line with RHEL 6.x, and CentOS 6.x. The standard conversion scripts (centos2cl or cpanel2cl) were upgraded to handle conversion of RHEL/CentOS 6.x servers and ISO images for CloudLinux 6.1 are available for download http://www.cloudlinux.com/downloads

CloudLinux 6.1 comes with a new kernel based on 2.6.32. The new kernel is more efficient than the one available in CloudLinux 5.x. The new version will no longer create migration threads, and in most cases, should outperform CloudLinux 5.x

There are few known issues with CloudLinux 6:

• lve-stats don't work - should be fixed within days
• CageFS and MySQL governor are not yet available on CloudLinux 6.1 - we plan to introduce compatible versions in a few weeks

Everything else should be fully functional. We are waiting for our upstream kernel (OpenVZ) to be moved to 'stable' from 'beta', and we should announce a stable release of CloudLinux 6.1 soon after. We expect this within the next 30 days.

Please try CloudLinux 6.x in your environments. A license is required, so please feel free to register another account/request another trial license if needed.

The memory limits in CloudLinux are confusing at best. First of all they count virtual memory allocated by processes, instead of physical memory. And virtual memory use can be much higher, as Linux is very efficient in using same physical memory for multiple processes. We plan to add physical memory limits in the future – yet, this is not the only issue with memory limits. 

No matter if we limit physical or virtual memory, there will always be some guess work in detecting if the script error was due to memory limit, or if it was due to permissions, configuration errors or errors in the script itself. Such errors is the primary reason for us to ship CloudLinux with memory limits disabled by default.  Memory limits are useful, and can often save server from overloading, swapping & going down. Yet, they can also add errors, that most sys admins don't connect to memory limits right away

When software (such as php interpretor or mod_fcgid daemon) tries to allocate memory from a system, LVE can prevent that from happening. It would do it same way OS would do it in the case when there not enough memory. Most applications when they try to allocate memory, and fail, they will fail as well. It would look pretty much as if failed due to bug, or some other error. The distinction is very small, and usually comes as part of cryptic error message and strange exit code. When it comes to website, such errors usually pop up as error 500 – which means that script used to serve the request failed due to some error. In this case it usually means that PHP interpretor failed (same way it would fail on bad php script). Basically – PHP or some other components fails, for whatever reason, and error 500 served. Not much for CL to do here.

Sometimes it gets even worth. Recently we got a customer who complained about mail() not working in php script. It was working before, but it stopped working after CloudLinux was installed. We knew that CloudLinux 'never' does something like that, and were totally baffled. It was verifiable error. Running php script that was trying to send email would come back with:

Switching back to CentOS kernel would solve the problem (that would disable LVE). It took us some time to stumble upon the fact that it might be memory limits. Once we did, it took a minute to verify it. There was enough memory to run php interpretor, but not enough for sendmail to run on top of it. Hence sendmail would fail, and php would deliver such message. Increasing memory limit removed the issue. There is an easy way to figure out if the issue relates to memory limits. All you need to do is to run:


If you see user for which script failed in the list, it means that some script for that user hit memory limit within the past 10 minutes. Run the script again, re-check lveinfo (note, it takes 1 minute for it to update) – and you know for sure. Same information can be taken out of /proc/lve/list

Of course this is not enough, and we plan to do more. We want to create sophisticated notification system, so that both admin & user would be notified in case memory limits are reached. Additionally, we are researching the possibility to detect run time, on webserver level, when one of the processes that was used to serve up the request hit memory limit – and if we can intervene & serve our own error message in such cases. We are still at researching it – and if that would be possible, it would create a nice way to take out the confusion.