Working with LVE (version 0.7 and higher)

About LVE

LVE is a kernel level technology developed by the CloudLinux team. The technology has common roots with container based virtualization. Yet, it is lightweight, and transparent. The goal of LVE is to make sure that no single web site can bring down your web server.
Today, a single site can consume all CPU resources, IO resources or apache processes -- and bring the server to the halt.
LVE prevents that.
It is done via collaboration of apache modules and kernel.
mod_hostinglimits is apache module that:

Detects VirtualHost from which the request came. Detects if it was meant for cgi or PHP script. Switches apache process used to serve that request into LVE for the user determined via SuexecUserGroup for that virtual host. Lets apache to serve the request. Removes apache process from user's LVE.

The kernel makes sure that all LVEs get fair share of the server's resources. This means, for example, that 20 apache processes serving a heavy site will use the same amount of CPU as one apache process serving a smaller site.
We plan to expand this technology to include IO (coming soon)
Each LVE limits amount of entry processes (Apache processes entering into LVE) to prevent single site exhausting all apache processes. If the limit is reached -- mod_hostinglimits will not be able to place apache process into LVE, and will return error code 503 (server busy). This way very heavy site would slow down and start returning 503 errors, without affecting other users.

Checking if LVE is installed

To use LVE you have to have CloudLinux kernel installed, and LVE module loaded
you can check the kernel by running the following command:

uname -r

You should see something like 2.6.18-294.8.1.el5.lve0.7.30 The kernel should have lve in its name. To see if lve kernel module is loaded do

# lsmod|grep lve
lve                    46496  0

Enabling / Disabling LVE

You can toggle LVE on/ff by editing
/etc/sysconfig/lve and setting LVE_ENABLE variable to yes or no.
Setting it to yes will enable LVE, setting it to no will disable LVE.
You need to reboot the server, after you set this option to make the changes live

Adjusting LVE default values

Values set in /etc/container/ve.cfg are used by default when processes are running within LVE. After editing /etc/container/ve.cfg run the following command:

# /etc/init.d/lvectl reload

Alternatively you can use lvectl command

# lvectl set default --cpu=25 --save

Adjusting LVE default values in config file

Example:

<lveconfig>
  <defaults>
    <cpu limit="100"/>
    <io limit="100"/>
    <other maxEntryProcs="10"/>
  </defaults>
  <lve id="532"> 
    <cpu limit="30"/> 
  </lve>
</lveconfig>

Sets CPU limit to 100%, IO limit to 100% and maximum entry processes to 10 for all LVEs. It also sets the limit of 30% for LVE with ID 532

Adjusting LVE settings on the fly

You can use lvectl command to adjust individual settings on the fly

Example:

# lvectl set 600 --cpu=60

Adjust CPU limit for LVE id 600 to 70%

Monitoring LVE

You can see a list of LVEs with their CPU usage and limits by doing cat /proc/lve/list
The number of entry processes inside LVE will be the value under *refs" minus two. If refs = 4, it means LVE has 2 entry processes. cpu_usage field shows the total amount CPU (in jiffies) all threads within LVE use up to this moment (since last restart).
One of the best way to monitor current usage is lvetop

# lvetop
    
         LVE    REF    PNO    TNO    USR    GRP    CPU    MEM      I/O
         106      3      3      6    N/A    N/A     7%   413412    N/A
         109      1      1      1    N/A    N/A     3%      156    N/A
         122      2      2      2    N/A    N/A     2%   321344    N/A

[root@localhost tests]# cat /proc/lve/list 
veid    refs    cpu    io    cpu_usage
900    22    100    100    229662988927

Additionally you can use tool lveps to see CPU usage, and processes within LVE

usage: lveps [-p] [-n] [-h] [-d]
-p to print per-process/per-thread statistics
-n to print uid/gid rather than user/group
-h to print this brief help message -d displays CPU usage as a percentage over last 30 seconds. Similar to lvetop, but doesn't refresh

[root@localhost tests]# lveps
     LVE     UID     GID     CPU     MEM     I/O
     900    root    root       9    1440     N/A

The CPU field shows amount of CPU used by all the threads within LVE in jiffies, since last restart of the system.

[root@localhost tests]# lveps -p
     LVE     PID     TID     UID     GID     CPU     MEM     I/O
     900     ---     ---    root    root      39    1440     N/A
     ---   14959   14959    root    root       1      72     N/A
     ---   14958   14958    root    root       1      72     N/A
     ---   14957   14957    root    root       1      72     N/A
     ---   14956   14956    root    root       1      72     N/A
     ---   14955   14955    root    root       1      72     N/A
     ---   14954   14954    root    root       1      72     N/A
     ---   14953   14953    root    root       1      72     N/A
     ---   14952   14952    root    root       1      72     N/A
     ---   14951   14951    root    root       1      72     N/A
     ---   14950   14950    root    root       1      72     N/A
     ---   14949   14949    root    root       1      72     N/A
     ---   14948   14948    root    root       1      72     N/A
     ---   14947   14947    root    root       1      72     N/A
     ---   14946   14946    root    root       1      72     N/A
     ---   14945   14945    root    root       1      72     N/A
     ---   14944   14944    root    root       1      72     N/A
     ---   14943   14943    root    root       1      72     N/A
     ---   14942   14942    root    root       1      72     N/A
     ---   14941   14941    root    root       1      72     N/A
     ---   14940   14940    root    root       1      72     N/A
     ---   14918   14918    root    root       0       0     N/A

The CPU value shown is number of seconds all the threads used.

Collecting LVE usage statistics

It is now much easier to collect LVE statistics, by using /proc/lve/usage

The content of the file would look like that:

LAST RESET: 1282217145 
LVE    MEM_FAULT    MEP_FAULT   CPU_USAGE 
333    0            15          80594711 
423    3            0           113980465

LVE -- LVE id (will match user id in many cases)

MEM_FAULT -- number of times process within LVE had memory allocation fault due to memory limit (reserved for future use)

MEP_FAULT -- number of times there was maxEntryProc fault in LVE

CPU_USAGE -- as in /proc/lve/list, since last reset. CPU_USAGE is 8 bytes long

LAST RESET -- time of the last reset. At reset, all counters are reset to 0. cpu_usage is reset to match cpu usage in /proc/lve/list at the time of reset.

To reset usage statistics, do:

# echo clear > /proc/lve/usage

This should be an easy way to find out which users are often affected by maxEntryProcs and get 503 errors just by looking at this counters.

Apache and LVE

CloudLinux comes standard with the module mod_hostinglimits that puts apache processes into LVEs. For more information on this module see: mod_hostinglimits config info Disabling that module should disable LVE on for apache.

Apache and LVE -- mod_fcgid

CloudLinux packages modified version of mod_fcgid, that supports LVE. More info here: FastCGI

Documentation