Working with LVE (version 0.8 and higher)

About LVE

LVE is a kernel level technology developed by the CloudLinux team. The technology has common roots with container based virtualization. Yet, it is lightweight, and transparent. The goal of LVE is to make sure that no single web site can bring down your web server.

Today, a single site can consume all CPU, IO, Memory resources or apache processes -- and bring the server to the halt. LVE prevents that. It is done via collaboration of apache modules and kernel.

mod_hostinglimits is apache module that:
Detects VirtualHost from which the request came. Detects if it was meant for cgi or PHP script. Switches apache process used to serve that request into LVE for the user determined via SuexecUserGroup for that virtual host. Lets apache to serve the request. Removes apache process from user's LVE.

The kernel makes sure that all LVEs get fair share of the server's resources. This means, for example, that 20 apache processes serving a heavy site will use the same amount of CPU as one apache process serving a smaller site.

Today we can limit CPU, Memory and number of entry processes. We plan to expand this technology to include IO (coming soon)

Each LVE limits amount of entry processes (Apache processes entering into LVE) to prevent single site exhausting all apache processes. If the limit is reached -- mod_hostinglimits will not be able to place apache process into LVE, and will return error code 503 (error code we selected, so that it wouldn't interfere with any other error codes). This way very heavy site would slow down and start returning 503 errors, without affecting other users.

Checking if LVE is installed

To use LVE you have to have CloudLinux kernel installed, and LVE module loaded
you can check the kernel by running the following command:

uname -r

You should see something like 2.6.18-294.8.1.el5.lve0.8.60 The kernel should have lve in its name. To see if lve kernel module is loaded do

# lsmod|grep lve
lve                    46496  0

Enabling / Disabling LVE

You can toggle LVE on/ff by editing
/etc/sysconfig/lve and setting LVE_ENABLE variable to yes or no.
Setting it to yes will enable LVE, setting it to no will disable LVE.
You need to reboot the server, after you set this option to make the changes live

Adjusting LVE default values

Values set in /etc/container/ve.cfg are used by default when processes are running within LVE. After editing /etc/container/ve.cfg run the following command:

# /etc/init.d/lvectl reload

Alternatively you can use lvectl command

# lvectl set default --cpu=25 --save

Adjusting LVE default values in config file

Example:

<lveconfig>
  <defaults>
    <cpu limit="25"/>
    <io limit="100"/>
    <mem limit="262144"/>
    <ncpu limit="1"/>
    <other maxEntryProcs="10"/>
  </defaults>
  <lve id="532"> 
    <cpu limit="30"/> 
  </lve>
</lveconfig>

Sets CPU limit to 25%, IO limit to 100% (place holder), memory limit to 1GB (memory limit is set as a number of 4096 bytes pages), CPU cores per LVE to 1, and maximum entry processes to 10 for all LVEs. It also sets the limit of 30% for LVE with ID 532

CPU Limits explained

CloudLinux allows to set % of CPU being used relative to total number of CPU cores available on the server. This means that 5% setting on 8 core server will be 40% of single core allocated for LVE. On 4 core server 5% would mean 20% of single core.

In addition to CPU limits each LVE is limited by number of cores that it can use. The default is set to 1. There is a small overhead related to number of sites * number of cores per site. So, a server with 32 cores and 5,000 sites will have a lot of overhead. Yet, setting just one core per LVE reduces that overhead by 32. In most cases, one core per LVE should be enough.

The number of cores settings takes precedence. For example if you have 25% CPU limit, 1 core per LVE, and 8 core server -- each LVE will be limited by 100/8 =~ 12%, instead of 25% CPU, as at most one core will be used by each LVE (1 core on 8 core server ~= 12% of total CPU capacity)

Note: Changing the number of cores allowed for existing LVE requires reboot.

Adjusting LVE settings on the fly

You can use lvectl command to adjust individual settings on the fly

Example:

# lvectl set 600 --cpu=60

Adjust CPU limit for LVE id 600 to 70%

Monitoring LVE

You can see a list of LVEs with their usage and limits by doing cat /proc/lve/list

One of the best way to monitor current usage is lvetop

# lvetop
           ID     EP    PNO    TNO    CPU    MEM    I/O
           test    1      2	 2     2%    728      0

[root@localhost tests]# cat /proc/lve/list 
4:LVE	EP	lCPU	lIO	CPU	MEM	IO	lMEM	lEP	nCPU	fMEM	fEP
0	0	75	25	0	0	0	262144	20	2	0	0
500	0	75	25	0	0	0	4294967 20	3	2	1	
700	1	75	25	1403247	202	0	262144	20	2	0	0

Additionally you can use tool lveps to see CPU usage, and processes within LVE

usage: lveps [OPTIONS]

-p to print per-process/per-thread statistics
-n to print LVE ID istead of username
-o to use formatted output (fmt=id,ep,pid,tid,cpu,mem,io)
-d to show dynamic cpu usage instead of total cpu usage
-c to calculate average cpu usage for <time> seconds (used with -d)
-r to run under realtime priority for more accuracy (needs privileges)
-s to sort LVEs in output (cpu, process, thread)
-t to run in the top-mode
-h to print this brief help message

[root@localhost tests]# lveps
             ID     EP    PNO    TNO    CPU    MEM    I/O
           test      1      2      2      0    792      0

The CPU field shows amount of CPU used by all the threads within LVE in jiffies, since last restart of the system.

[root@localhost tests]# lveps -p
             ID     EP    PNO    PID    TNO    TID    CPU    MEM    I/O
           test      1      2    ---      2    ---    444    792      0
                   ---    ---   8906    ---   8906    440    216    N/A
                   ---    ---   8905    ---   8905      4     68    N/A

Collecting LVE usage statistics

All the LVE statistics info can be collected by monitoring /proc/lve/list

LVE -- LVE id (will match user id in many cases)

EP -- number of entry processes

CPU -- CPU usage, in ticks

MEM -- memory usage, in 4096 bytes pages

IO -- IO usage, place holder

nCPU -- number of cores allocated for this LVE

fMEM -- number of failures, when program within LVE tried to allocate more memory then LVE was allowed to have

fEP -- number of failures when process couldn't enter LVE due to too many entry processes already in LVE

This should be an easy way to find out which users are often affected by maxEntryProcs and get 503 errors just by looking at this counters.

Apache and LVE

CloudLinux comes standard with the module mod_hostinglimits that puts apache processes into LVEs. For more information on this module see: mod_hostinglimits config info Disabling that module should disable LVE on for apache.

Apache and LVE -- mod_fcgid

CloudLinux packages modified version of mod_fcgid, that supports LVE. More info here: FastCGI

Documentation