Post of site blogs "CloudLinux Inc." (www.cloudlinux.com) Post of site blogs "CloudLinux Inc." (www.cloudlinux.com) http://www.cloudlinux.com en http://backend.userland.com/rss2 Sat, 04 Feb 2012 01:47:56 -0500 Igor Seletskiy: CageFS 3 Beta 5
  • Additional configuration options for exim are added
  • PAM lve settings are preserved during upgrade
  • Failures to compile new suPHP are detected
  • Prevent incorrect directories in cagefs skeleton
  • Disable checks that skeleton is already mounted
To update to the new version, please run:
# yum update cagefs pam_lve --enablerepo=cloudlinux-updates-testing

More...]]> http://www.cloudlinux.com/blog/clnews/124.php http://www.cloudlinux.com/blog/clnews/124.php Tue, 31 Jan 2012 09:44:31 -0500 Igor Seletskiy: CageFS 3 Beta 4
  • Fixes race condition that would cause serving wrong /etc/passwd or /etc/group file
  • Ability for sendmail to send mail when eximmailtrap is enabled
  • White listed user/groups that can see additional users in password and group files
  • Fixed permissions for cagefs-update.log
    To update to the new version, please run:
    # yum update cagefs --enablerepo=cloudlinux-updates-testing
    # cagefsctl --remount-all
    More...]]> http://www.cloudlinux.com/blog/clnews/123.php http://www.cloudlinux.com/blog/clnews/123.php Tue, 24 Jan 2012 13:11:33 -0500 Igor Seletskiy: Local Privileged Escalation CVE-2012-0056 -- Not Vulnerable http://blog.zx2c4.com/749 .I am happy to say that NONE of our kernels are vulnerable. We have tested, verified and made sure that they are all safe and are not exploitable via that vulnerability
    More...]]>     http://www.cloudlinux.com/blog/clnews/122.php http://www.cloudlinux.com/blog/clnews/122.php Tue, 24 Jan 2012 02:14:42 -0500     Igor Seletskiy: CageFS 3 beta 3
    Other multiple bug fixes and improvements, including:
    • Issues when sending mail via exim were resolved 
    • Issues related to cPanel CageFS plugin were resolved
    • Improved detection of LiteSpeed, with automatic configuration of CageFS settings in LiteSpeed
    • CageFS jail API was improved for better backward compatibility.


    To update:
    # yum update cagefs cagefs-fuse lve lve-wrappers pam_lve --enablerepo=cloudlinux-updates-testing

    If you have LiteSpeed installed:
    # cagefsctl --create-mp
    # cagefsctl --remount-all
    More...]]>     http://www.cloudlinux.com/blog/clnews/121.php http://www.cloudlinux.com/blog/clnews/121.php Fri, 20 Jan 2012 11:24:13 -0500     Igor Seletskiy: Kernel: 2.6.18-374.12.1.el5.lve0.8.54 is in production
    • rebase to  274.12.1.el5 kernel (RHSA-2011:1479)
    • migration processes are now hidden from /proc file system
    • Load averages calculation optimization
    • LVE destroy fix to prevent kernel crash
    • UBC related optimization for vcpu structures
    • xtable kernel panic fix
    • namespaces reworked for CageFS 3.0
    To install:

    ====code==== 
    # yum install kernel-2.6.18-374.12.1.el5.lve0.8.54
    =============
    If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel
    More...]]>     http://www.cloudlinux.com/blog/clnews/115.php http://www.cloudlinux.com/blog/clnews/115.php Wed, 18 Jan 2012 00:00:00 -0500     Igor Seletskiy: Using top to pinpoint issues in seconds. Lets take a look:

    top - 02:12:43 up 1:12, 5 users, load average: 3.93, 4.29, 12.40
    Tasks: 269 total, 6 running, 262 sleeping, 0 stopped, 1 zombie
    Cpu(s): 19.4%us, 11.4%sy, 0.0%ni, 49.0%id, 19.6%wa, 0.2%hi, 0.3%si, 0.0%st
    Mem: 8028612k total, 4529728k used, 3498884k free, 389300k buffers
    Swap: 4192924k total, 181872k used, 4011052k free, 2663176k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    47660 mysql 6 -10 1052m 233m 4784 S 2.8 3.0 1:11.31 mysqld
    47374 nobody 15 0 305m 141m 2836 S 0.0 1.8 0:03.71 httpd
    49478 nobody 15 0 304m 141m 2864 S 0.0 1.8 0:00.34 httpd
    49781 nobody 15 0 305m 141m 2320 S 0.0 1.8 0:03.09 httpd
    39462 nobody 15 0 305m 140m 2408 S 0.0 1.8 0:07.94 httpd
    50142 nobody 18 0 305m 140m 2320 S 1.4 1.8 0:02.84 httpd 


    Right away we can see that the server was recently rebooted - just an hour ago due to this info (top - 02:12:43 up 1:12), and that load is subsiding (load average: 3.93, 4.29, 12.40). Now, everyone knows that “high” load average is bad, and low load average is good. This three values represent load averages for 1, 5 and 15 minutes correspondingly. The general rule of thumb is that if load average is higher then number of cores on a server – it is a bad thing. If it is less – it is ok. The things are bit more complex on CloudLinux servers, due to CPU throttling – but lets continue discussing 'top' instead.

    So, once we take a look at load averages, if they are high – we know that something might be going on with the server. More often then not – top can help you pinpoint the issue.

    Here is another example:
    top - 01:49:59 up 49 min, 5 users, load average: 18.02, 32.55, 34.91
    Tasks: 269 total, 1 running, 268 sleeping, 0 stopped, 0 zombie
    Cpu(s): 0.4%us, 0.6%sy, 0.0%ni, 16.7%id, 82.2%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 8028612k total, 2268460k used, 5760152k free, 88348k buffers
    Swap: 4192924k total, 731688k used, 3461236k free, 309828k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    8842 mysql 11 -10 1274m 178m 4076 S 0.0 2.3 0:46.40 mysqld
    27379 root 18 0 232m 123m 3768 D 0.0 1.6 0:00.93 httpd
    22731 a19a5a1 16 0 184m 53m 3692 D 0.0 0.7 0:01.13 php
    23384 a19a5a1 16 0 184m 53m 3900 D 0.0 0.7 0:01.05 php
    19443 a1a16ala 18 0 209m 51m 2772 D 0.0 0.7 0:01.52 php
    21254 a19a5a1 16 0 184m 51m 3692 D 0.0 0.7 0:01.26 php
    24659 a19a5a1 18 0 182m 48m 3612 D 0.0 0.6 0:00.70 php
    20674 a1p16ple 16 0 168m 48m 3604 D 0.3 0.6 0:01.57 php
    20796 user 16 0 179m 48m 3540 D 0.7 0.6 0:00.80 php
    21179 user 18 0 179m 48m 3540 D 0.0 0.6 0:00.70 php

    Load averages are high, and you can see 82.2%wa. “wa” -- stands for IO Wait – the % of cpu time spent by CPU waiting for IO (usually disks) to respond. Lets look at the topmost process in the “top” results: it is mysql. This is pretty easy give away, that something is going on with MySQL. The processes are sorted by CPU usage, and MySQL is IO intensive. So, if MySQL is actively using CPU, and IO usage is high – there is a high chance that the issue is with mysql

    Running mysqladmin processlist can help you pinpoint what might be exact problem. Sometimes it might be corrupted database, sometimes – bad query... If you train your eye – you can find out the cause of the issue within seconds of looking at the results.

    Here is another one:

    01:15:52 up 15 min, 4 users, load average: 28.37, 20.02, 10.44
    Tasks: 762 total, 1 running, 760 sleeping, 0 stopped, 1 zombie
    Cpu(s): 0.5%us, 1.8%sy, 0.0%ni, 0.0%id, 97.3%wa, 0.0%hi, 0.4%si, 0.0%st
    Mem: 8028612k total, 7981536k used, 47076k free, 34512k buffers
    Swap: 4192924k total, 2894460k used, 1298464k free, 272708k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    23058 root 16 0 10476 1044 752 S 3.3 0.0 0:40.76 /usr/sbin/lveps -t
    25203 root 18 0 39136 6000 1568 D 2.0 0.1 0:00.18 /usr/local/cpanel/bin/dcpumon
    23048 root 15 0 13412 1716 748 S 1.0 0.0 0:12.96 top -c

    io wait is at 97.3% , and at that level system would be hardly responsive. Pretty much all the memory taken and ~2.5GB of Swap is used. With a trained eye, you would see right away that “something” using up all the memory. Pressing SHIFT-M would sort processes by memory usage, and you will see which processes are eating up all the RAM, often providing you with enough info on what to do next.
    Here is a bit more complex one:

    top - 01:27:59 up 31 days, 22:31, 3 users, load average: 2.36, 2.33, 4.96
    Tasks: 697 total, 1 running, 692 sleeping, 0 stopped, 4 zombie
    Cpu(s): 2.1%us, 1.2%sy, 0.0%ni, 50.9%id, 45.5%wa, 0.1%hi, 0.1%si, 0.0%st
    Mem: 4023964k total, 3917884k used, 106080k free, 298548k buffers
    Swap: 8385920k total, 504136k used, 7881784k free, 1557260k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    950397 user123 16 0 183m 19m 7276 S 7.0 0.5 0:00.08 php
    950396 user321 16 0 0 0 0 Z 4.4 0.0 0:00.05 php <defunct>
    950312 nobody 15 0 216m 113m 2020 S 2.6 2.9 0:00.25 httpd
    946406 nobody 15 0 217m 114m 2032 S 1.8 2.9 0:01.03 httpd
    950314 nobody 18 0 217m 114m 1992 S 1.8 2.9 0:00.14 httpd
    7166 root 10 -5 0 0 0 S 0.9 0.0 38:39.15 kondemand/2
    8194 root 10 -5 0 0 0 S 0.9 0.0 67:35.63 kondemand/6
    8195 root 10 -5 0 0 0 S 0.9 0.0 93:49.75 kondemand/7 



    IO wait is pretty high (45.5%), swap is used – but not that much, just short of 1/2GB. It might appear that all the RAM is used up as well (3917884k used out of 4023964k total), but that is not the case.
    The are about 1.5GB memory “cached”, and almost 300MB used for buffers. The “cached” memory, is memory used to speed up disk IO. The data is cached by linux kernel, so that instead of re-reading same file from disk, it would be taken directly from RAM. If kernel sees that there are not enough RAM – it would free up caches and buffers. In this case, kernel decided it is more efficient to save some of the processes to Swap (most likely those that were sleeping for a long time), than it is to purge cache. As long as swap usage doesn't change much within few minutes of time – the most likely reason for high IO wait is not Swap.

    Looking at topmost processes in the list doesn't bring up anything that is easy to qualify as an issue. While IO bound processes are often CPU bound as well, it wasn't the case here. This is where you can use another good tool, iotop (can be installed via yum install iotop), that will show you processes that use up IO. In this particular case it was rsync causing high IO wait

    Total DISK READ: 2.89 M/s | Total DISK WRITE: 212.68 K/s
    951134 be/3 root 6.93 M/s 0.00 B/s 0.00 % 99.99 % rsync -rlptD --exclude=*/proc/* --delete /v~ /backup/cpbackup/daily/dirs/_var_lib_mysql_

    Here are two more examples, where the issue at hand is not that clear cut, yet top still give enough info to go into right direction. First one is from my previous blog post: 

    top – 2:15:42 up 48 min, 3 users, load average: 8.62, 4.78, 3.26
    Tasks: 176 total, 15 running, 161 sleeping, 0 stopped, 0 zombie
    Cpu(s) : 8.0%us, 4.5%sy, 0.1%ni, 3.2%id, 8.2%wa, 0.0%hi, 76.0%si, 0.0%st
    Mem: 8031276k total, 7363928k used, 667348k free, 365104k buffers
    Swap: 4194296k total, 91396k used, 4102900k free, 5077444k cached 


    si or software interrupts where the issue. Once again, this helps rule out quite a few scenarios, and shows what to “google” for. In this case USB module http://www.cloudlinux.com/blog/clnews/109.php?sphrase_id=18641 was an issue

    This issue I have seen first on AWS “micro” server (the top header is taken from one of forum post at amazon, I couldn't find my own), but I have also seen it recently at one of the customer's server running Xen.

    top - 22:42:48 up 1 day, 1:56, 1 user, load average: 2.63, 2.21, 1.85
    Tasks: 46 total, 5 running, 41 sleeping, 0 stopped, 0 zombie
    Cpu(s): 27.3%us, 9.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.0%si, 62.0%st
    Mem: 1747660k total, 1677908k used, 69752k free, 6780k buffers
    Swap: 917496k total, 36k used, 917460k free, 1269348k cached

    The key here is CPU 'stolen'
    st - Steal Time. The amount of CPU ’stolen’ from this virtual machine by the hypervisor for other tasks (such as running another virtual machine) – a fairly recent addition to the top command, introduced with the increased virtualization focus in modern operating systems (man top)
    In the last case, moving VM to another server with more resources solved the issue.
    As you can see top is quite useful, if you cross reference all the data that it presents.
    More...]]>     http://www.cloudlinux.com/blog/clnews/120.php http://www.cloudlinux.com/blog/clnews/120.php Tue, 17 Jan 2012 22:08:42 -0500     Igor Seletskiy: Hybrid kernel for CloudLinux 5 -- First beta The 2.6.32 kernel is a major improvement representing more then 5 years of development. It is generally faster & more stable then 2.6.18 kernel shipped with CloudLinux 5. 

    To install the kernel on your CloudLinux 5 server, please run:
    # yum update rhn-setup --enablerepo=cloudlinux-updates-testing 
    # /usr/sbin/normal-to-hybrid
    # reboot

    To convert back to original CloudLinux 5 server:
    # /usr/sbin/hybrid-to-normal
    # reboot

    Please, note following limitations & changes regarding CloudLinux 6 kernel/Hybrid kernel:
    • Hybrid kernel doesn't correctly recognize xvda devices (using sda instead) on Xen PV installations, causing kernel panik on boot
    • There is an issue with systems with CageFS 2 installed, please refrain from installing on such servers until second beta.
    • CageFS 3 beta is not available yet with this kernel (should become available next week)
    • Load averages are calculated the way they are calculated int CentOS. This means that if one site is getting limited, it will cause limits to increase, even if there is no increase in actual CPU usage, and server is not overloaded
    • Memory limits are enabled by default, there is no ubc setting, and you cannot do lvectl ubc disable. To disable memory limits, set them to 0
    • There is no separate kernel-xen, instead regular kernel package will be installed. That package will satisfy xen requirements, and supports para-virtualization

    More...]]>     http://www.cloudlinux.com/blog/clnews/118.php http://www.cloudlinux.com/blog/clnews/118.php Thu, 12 Jan 2012 11:23:37 -0500     Lesya Novaselskaya: VPS: hot or not?


    A VPS is functionally equivalent to a separate physical server, is tuned to the individual customer's needs, has the privacy of a separate physical server, and so on. The trick is, multiple VPSes (tens of even hundreds of them) reside on a single real server, sharing its hardware resources, reducing the needs for hardware, rack space, electricity bills, thus lowering the price level for a customer.

    Technically, a VPS functionality is implemented as an additional layer between the hardware and the software. Usually, a VPS is either a Virtual Machine (as in VMware, Xen, KVM) or a container (as in OpenVZ, Solaris Zones or LXC). Virtual Machines use hypervisor technology, and every VM-based VPS runs a full software stack, including the OS kernel and set of drivers for (virtual) hardware, the very same way your usual server runs.



    Containers are a bit different, more light-weighted, since there is only one single OS kernel is running, and VPSes only run userspace software (i.e. no own kernels, drivers etc.). Now, if you have read up to this point and are not asleep, you deserve a small and nice reward: go get yourself a cup of coffee or whatever your favourite drink is. Got it? Let's continue.



    Both containers and virtual machines have some overhead: this is the price one has to pay for splitting the piece of real hardware into multiple smaller pieces, getting some virtual hardware. This partitioning is done via isolation, and the isolation overhead is much less for containers than for VMs, but it's still there.

    The other thing worth mentioning besides the isolation overhead is the importance of fair resource management. Indeed, since multiple VPSes share the same set of hardware resources (CPU, RAM, disk, networking), care has to be taken of how to divide this resources in a fair manner, so that no single VPS abuse a resource, say, by eating up all disk space or saturating the disk I/O channel.

    Now, let's consider the ideal VPS solution, for which the isolation is bullet-proof (which almost true for VMs) and don't cost us more than a penny (which is almost true for containers), plus the resource management is just perfect, all ruled by no one less than king Solomon in all his great wisdom. Even with that ideal VPS solution in place, there are troubles around the corner.



    First of all, every VPS runs (at least) the complete set of userspace software, which needs to be maintained. That is, unlike in shared hosting, you have to take care about software updates (if you care of security, for instance). You need to configure some auxiliary components like system logger or cron daemon. Basically, you need to be a full-scale sysadmin for that, or hire one, which is not that cheap at all. It could be useful if you are a sysadmin, so you can recompile PHP with your favorite options and patches (which is great fun!), but  if you're not...  than the need to manage your OS yourself is more of a burden, if not a nightmare.

    Don't forget that every VPS needs a distinct public IP address. I hear you say "It's not my problem!", and I tend to agree, your HSP should take care about this. Still, considering a recent trend in IPv4 address space shortage, and less-than-warm acceptance of IPv6, this might be your problem, too.

    Overall, most of the problems that you have with dedicated hosting also apply to VPS hosting. Dedicated hosting, though, is used by bigger web sites, which have proper sysadmin and other resources, and the knowledge needed to manage operating system instances. A VPS owner is usually not so rich and huge, but she faces the same set of problems as a dedicated guy.

    So, which way to go if you want VPS benefits (like proper isolation and resource controlled by a Solomon) but don't like its maintenance burdens? Still don't know the answer? Oh, come on, it's pretty obvious! Go CLOUDLINUX! Amen.
    More...]]>     http://www.cloudlinux.com/blog/clnews/vps-hot-or-not.php http://www.cloudlinux.com/blog/clnews/vps-hot-or-not.php Thu, 12 Jan 2012 11:04:15 -0500     Igor Seletskiy: CageFS 3.0 Beta 2
  • Adds automatic detection for LiteSpeed webserver (stand-alone ony)
  • Fixes caching bug in CageFS-FUSE etcfs
  • Improves stability and performance of CageFS-FUSE
  • Fixes various bugs related to mounting/unmounting of filesystems
  • Correct handing of umask

    To  update:
    # yum update cagefs cagefs-fuse lve liblve liblve-devel lve-wrappers --enablerepo=cloudlinux-updates-testing
    * LiteSpeed + cPanel & LiteSpeed using Apache's httpd config file are currently not supported. We are working to add support for it.
    More...]]> http://www.cloudlinux.com/blog/clnews/117.php http://www.cloudlinux.com/blog/clnews/117.php Wed, 11 Jan 2012 09:07:10 -0500 Igor Seletskiy: CageFS 3.0 is ready for beta testing
    • Better namespace handing, requiring only fraction of mount points comparing to CageFS 2.x
    • CageFS-FUSE provides virtualized /etc & /var/log -- increasing security, and decreasing complexity of maintaining CageFS
    • Caged directories are no longer visible in /proc/mounts, solving all related issues with cPanel
    • Management plugins for cPanel and Plesk will be installed automatically (other control panels coming soon)
    • Automatic detection of cPanel and Plesk (other control panels coming soon), with automatic configuration to adjust for the running system
    • Improved command line tools
    The current version should work with LiteSpeed, and or with custom control panels with some additional configuraiton.


    More information on CageFS, how to install it or update from previous version can be found here:
    http://www.cloudlinux.com/docs/cagefs/index.php
    More...]]>     http://www.cloudlinux.com/blog/clnews/116.php http://www.cloudlinux.com/blog/clnews/116.php Thu, 05 Jan 2012 17:05:23 -0500