Clarify new CPU limits
Forum
  1. Forums
  2. CloudLinux and Control Panels
  3. CloudLinux and cPanel
  1. Jules Robinson
  2. Sunday, 20 July 2014
  3.  Subscribe via email
With the recent changes to CloudLinux and LVE manager, could you please clarify the new limits?

I notice the documentation says that \"speed\" is relative to number of cores, so \"100%\" means 1 full core, \"200%\" is 2 full cores and so on. Does this now make the \"nCPU\" parameter redundant if \"speed\" is set?

If \"speed\" is set to \"200%\" would \"nCPU\" be ignored?

Finally, can you also please clarify that changes to these settings no longer requires a reboot, or is this still the case?

Thanks in advance.
Rate this post:
  1. 25.07.2014 08:07:39
  2. # 1
Igor Seletskiy Accepted Answer
Posts: 1194
Joined: 09.02.2010
0
Votes
Undo
The issue is related to a bug in mod_security output filter.  The way it works -- it collects all the output (from script), and then checks it against a number of regexps to see if output doesn't leak any sensitive info, like credit card numbers.
It is all good, until you try to output 'infinite' amount of info (as in case of infinite loop that outputs something on each iteration). To store all that output, you would need infinite amount of RAM, and that is why your httpd processes balloon in size.
We had somewhere a script that kills big httpd processes, but this is still bad for performance (as big httpd process evicts caches). Right solution would be to disable (or fix) mod_security output filters.
  1. 25.07.2014 00:07:31
  2. # 2
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Hey Igor, yes it was the Apache process that ate all the ram and swap. I am using Atomic\'s (paid) rulesets for cpanel which i *think* do include output filtering
  1. 25.07.2014 00:07:51
  2. # 3
Igor Seletskiy Accepted Answer
Posts: 1194
Joined: 09.02.2010
0
Votes
Undo
Send it to me direct. [email protected]
yeBTW: 
when they crashed it -- was it apache process that became really big/ate up all the RAM? If yes, are you using mod_security with output filtering?
  1. 24.07.2014 23:07:39
  2. # 4
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
OK found them ... 2 scripts from 2 different customers that managed to crash the servers via memory exhausting (endless loops in php). Can I email them to you directly or should I open a ticket at the helpdesk? Cheers D.
  1. 24.07.2014 10:07:11
  2. # 5
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Yeah I have the script somewhere .. but in bed now (1am here) so will get back to you soon with the script.

Basically what happened was from memory it was just a simple i for loop in that script .. I managed to take a screen shot of top before server crashed and it was showing apache as using 10gb ram + swap (out of 12 total ..just shortly before it crashed) and that script was one of the top processes too. But the ram was definately used up by apache for whatever reason.. I'll get back to you tommorow. Cheers

edit: all my servers run suPHP and have so for years now 
  1. 24.07.2014 10:07:43
  2. # 6
Igor Seletskiy Accepted Answer
Posts: 1194
Joined: 09.02.2010
0
Votes
Undo
Do you have that PHP script? Infinite loop in php is fully controled memory limits -- as long as you are using suPHP/mod_fcgid/mod_lsapi/or php via suexec.
If you use mod_php (with ITK, ruid2 or without) -- then yes, you cannot control memory for those PHP processes. Only CPU & IO. 
  1. 24.07.2014 10:07:25
  2. # 7
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Whilst I do appreciate your response and time you\'ve taken to write that Bogdan, my issue was to do with simple endless loops written by customers (by mistake) in PHP scripts, nothing to do with rewrites/redirects.

I guess I\'ll just have to find some other way to stop this from happening since LVE can\'t control apache. If anyone has any tips in which direction I should be looking I\'d very much appreciate it.
  1. 24.07.2014 09:07:30
  2. # 8
Richard Hordern Accepted Answer
Posts: 212
Joined: 19.03.2011
0
Votes
Undo
as we are working on a very big rework, that should add many cool featurs (like snapshoting of processes & queries when limits are hit, burstable limits and reseller limits). 
Great news ! Sounds very exciting :)
  1. 23.07.2014 21:07:59
  2. # 9
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Bogdan

regarding the endless loop thing ... this has happened multiple times now on different servers (all with cloudlinux and all had 25% cpu limit and 1gb memory etc) 

Each time it was the Apache process that was eating all the ram even though there was an associated PHP process (simple php script with endless loop) and because apache is owned by root/nobody I guess it wasn't throttled. I just find it frustrating that a user can circumvent lve limits so easily (unintentionally). I opened a ticket about this some years ago when it happened the first time (it has happened again ..different users..different servers) and fr om memory Igor said LVE couldn't do anything about this because it's Apache and because it's probably due to mod_security or something.

I just find it emberassing when I have to tell a customer to check their php scripts for endless loops because I know how easily it can crash the server and still have no solution for it.

Would it not be possible to put LVE limits on Apache ? ie. atleast a memory lim it so that if it starts to consume more than X Gb of ram apache gets stopped and restarted automatically?
  1. 23.07.2014 20:07:50
  2. # 10
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Hi Bogdan

as it turns out the package "lvemanager" was not installed on any of my cpanel boxes so I must have been using an old WHM frontend for Lve manager ... the new one looks completely different (much nicer).

edit: fixed unrelated problem removed comments regarding it.

In regards to the "104" - it makes no sense. All the servers are 4 core (8 threads). The WHM frontend now shows SPEED as "100" which is what I want if I understood your docs correctly (ie. I want each user to be able to use max 1 core). However "lvectl list" still shows every user's SPEED set to "104"
I've even run

lvectl set default --speed=100% 
lvectl apply all
 

and when I do lvectl list its still showing "104" in the SPEED column. is this just a problem with lvectl ?

Here is my output for rpm -qa | grep -i lve (can you tell me if I have all the packages and if they are correct versions please??)


[~]# rpm -qa | grep -i lve
kernel-2.6.18-408.8.2.el5.lve0.8.61.3
lve-1.2-1.12.el5.cloudlinux
kernel-2.6.18-448.16.1.el5.lve0.8.70
liblve-devel-1.2-1.12.el5.cloudlinux
lve-wrappers-0.6-1.el5.cloudlinux
lve-utils-1.4-18.3.el5.cloudlinux
kernel-headers-2.6.18-471.3.1.el5.lve0.8.72
liblve-1.2-1.12.el5.cloudlinux
pam_lve-0.3-7.el5.cloudlinux
lvemanager-0.8-1.32.el5.cloudlinux
kmod-lve-2.6.18-408.8.2.el5.lve1.1.65.3-1.1-9.9.el5
lve-stats-0.10-31.2.el5

I even did yum reinstall lve-utils but it didn't make any change - "list" still shows that 104 number... so random.
  1. 23.07.2014 17:07:56
  2. # 11
Igor Seletskiy Accepted Answer
Posts: 1194
Joined: 09.02.2010
0
Votes
Undo
I am sorry for being MIA. A lot of things required my attention in the past few weeks (and that will continue for 3-4 more weeks I am afraid), and as the result I let documentation & communication via social media to lag behind. I will try to be more responsive.
Jules -- you were right at being angree with me -- I should have paid it more attention.
Delta - I am sorry to hear about upcp being stuck -- next time you see something like that - do ps -auxw, and save the output for us so we can trace it. I haven't heard other users having that problem. Sounds strange.

SPEED vs CPU limits. Documentation had been updated to explain SPEED works. http://docs.cloudlinux.com/index.html?limits.html
It overrides both CPU & NCPU options -- and the only option that matters.
LVE Manager will be updated to remove NCPU within next few weeks.
lve-stats will be updated in the next 2-3 months -- as we are working on a very big rework, that should add many cool featurs (like snapshoting of processes & queries when limits are hit, burstable limits and reseller limits).

If anyone has any more questions regarding speed & CPU -- I would be happy to answer them.
  1. 23.07.2014 12:07:38
  2. # 12
Delta A Accepted Answer
Posts: 8
Joined: 07.05.2014
0
Votes
Undo
Several years since installing cloudlinux and I still feel like I am a beta tester...

First of all, on ALL of our cpanel servers upcp got hung up after the latest lve-stats upd ate. I had to manuall kill upcp and do a /scripts/upcp --force afterwards.

This is where upcp hung..


[20140723.024207]      [538983] Dependency Installed:
[20140723.024207]      [538983]   alt-sqlite.x86_64 0:3.8.3.1-1.el5                                             
[20140723.024207]      [538983] 
[20140723.024207]      [538983] Updated:
[20140723.024207]      [538983]   lve-stats.noarch 0:0.10-31.1.el5                                              
[20140723.024207]      [538983] 
[20140723.024207]      [538983] Complete!
[20140723.024213]      [538983] checkyum version 21.1


ANYWAY ..now that I finally got that going again. I look at the WHM lve limits and it's just random numbers such as "13" for the cpu limit and I can't change the bloody thing! Trying to edit the default settings says "settings saved" but when I goto the main page it still shows "13" as the cpu limit. WTF? How did it come up with that number to begin with?

OK ANYWAY ... so I thought maybe it's just the WHM frontend that's buggy so I login to root shell and use the "lvectl" command.

lvectl limits all shows this..


[~]# lvectl limits all
      ID   SPEED    NCPU    VMEM      EP      IO
 default     104       1    1.0G      20      25
     514     104       1    1.0G      20      25
     643     104       1    1.0G      20      25
.
.. and so on, same for every user.

104 WHAT ????? 104% of cpu ? 104 Mhz ?? WHAT IS IT DOING!!!!

Trying to rese t this limit with this command:

lvectl apply all --speed=25% 

does NOTHING. "lvectl limits all" stil shows that "104" as 'SPEED' as WHM still shows "13"

What the hell is going on here? I feel like I have completely lost control over the server as cloudlinux is just putting arbitrary and non-consistent limits in the cli/whm and is not accepting the new ones like "lvectl apply all --speed=25% "

THIS IS NOT GOOD ENOUGH!!! We can't run buggy software that actually has such a huge and visible impact on our servers!

Come on Igor this is not good enough and you know it. You need to test this crap thoroughly before releasing it. We are paying customers and right now I am paying for nothing but stress.
PS> The documentation is and always has been a joke. It is written in such a way that only the person who wrote it could ever understand it.

I am really getting fed up with this. We had a customer make a simple mistake writing a PHP script with an endless loop in it and it crashed the server by exhausing all ram and swap in 2 minutes. I am starting to wonder if we're paying for snake oil.

Really really angry right now.
  1. 21.07.2014 20:07:20
  2. # 13
Jules Robinson Accepted Answer
Posts: 10
Joined: 10.04.2014
0
Votes
Undo
@Jose: Thanks for the information :)

I didn\'t want to create a support ticket as I thought the information would be useful for others here in public. Shame they are able to provide the information in the ticket but not in their own documentation or here on the forums. Disappointing.
  1. 21.07.2014 20:07:47
  2. # 14
Jose Dieguez Accepted Answer
Posts: 7
Joined: 20.07.2014
0
Votes
Undo
As i had the same doubts, i contacted Support asking them to explain me a little more how is working now.


Here is the first response: 


Hello.
True, NCPU abolished in the new versions. 
1 core = 100%, 2 = 200% core, core 4 = 400%, etc. 
For example, 1 core = 2000MHz. If I set a value of 150% - that account can consume 3000MHz.

For setting limits in the new versions use the CPU-parameter. 
I understand that the option of SPEED (% or Mhz) it's the mandatory, so the nCPU should be ignored... but, to make sure, i asked that:


Me: 
So if i set 400 at CPU %, it's full 4 Cores.

Even if i set at nCPU 1?, should i just ignore nCPU?

Thanks

Support:
Hello, Jose.

> So if i set 400 at CPU %, it's full 4 Cores.
Yes.

> Even if i set at nCPU 1?, should i just ignore nCPU?
Yes, this option will be ignored.

Hope this helps a little on this topic.

Have a good day
  1. 21.07.2014 19:07:37
  2. # 15
Jules Robinson Accepted Answer
Posts: 10
Joined: 10.04.2014
0
Votes
Undo
Richard, please stop posting.
  1. 21.07.2014 19:07:55
  2. # 16
Richard Hordern Accepted Answer
Posts: 212
Joined: 19.03.2011
0
Votes
Undo
The new version was only released a couple of days ago, give them a bit of time and they will update the documentation :)
  1. 21.07.2014 19:07:35
  2. # 17
Jules Robinson Accepted Answer
Posts: 10
Joined: 10.04.2014
0
Votes
Undo
@Richard: Are you purposefully being obtuse?? I've mentioned LVE manager several times in this thread, and in fact my very first line in the very first post says the following:

With the recent changes to CloudLinux and LVE manager....

To add to this, the "CPU limits" page of the documentation, which is the page you would expect to find detailed documentation on the implementation and settings regardless of command-line or UI interface, does not mention anything about nCPU being discontinued or no longer being used. Your comments about it being mentioned in the command line documentation are therefore irrelevant and unhelpful. For my own amusement I looked at the documentation for lvectl (which you claim confirms it is redundant and not being used) and it, also, does not mention anything about nCPU being redundant.

Although I did not understand Igor's answer the first time I read it (missed the ignored part), it now makes sense, actually says everything and allows us to understand that the ncpu setting will likely be removed in a future release of LVE Manager as Igor said it's currently ignored.

You continue to make assumptions, false interpretations and attempts to "read between the lines" in the documentation and Igor's responses. This may work for you and your business, but for me and ours I would like concrete and definitive responses here. I don't think that's unreasonable in the slightest. If CloudLinux are going to update their software, they need to update the documentation too. How they expect anyone to understand how their software works and to manage things properly without up to date and accurate documentation is beyond me.

I would really appreciate it if you would stop posting your assumptions and interpretations here. You are just going to confuse people looking for genuine and official information on the subject matter.
  1. 21.07.2014 18:07:54
  2. # 18
Richard Hordern Accepted Answer
Posts: 212
Joined: 19.03.2011
0
Votes
Undo
The documentation for the command line version is clear. LVE Manager documentation does need to be updated.

Although I did not understand Igor\'s answer the first time I read it (missed the ignored part), it now makes sense, actually says everything and allows us to understand that the ncpu setting will likely be removed in a future release of LVE Manager as Igor said it\'s currently ignored.

Thanks Igor, keep up the good work :)
  1. 21.07.2014 17:07:56
  2. # 19
Jules Robinson Accepted Answer
Posts: 10
Joined: 10.04.2014
0
Votes
Undo
@Richard: "According to the documentation the ncpu setting is no longer used." unless we're reading completely different documentation, you are wrong. It states nothing of the sort. Please stop making this statement because you will confuse other people as you've done multiple times in this thread already.

@Igor;  "If you use CPU + NCPU --> it would work as before 
If you use --speed --> NCPU ignored, not used.
"

Can you guys please, please, PLEASE just take some time to craft an actually comprehensible response and then update your documentation? With your above response, what are you actually saying? "If you use CPU + NCPU --> it would work as before." is what you are really saying here that if you haven't upgraded, nothing will change? Isn't that already a little obvious?

"If you use --speed --> NCPU ignored, not used." - great. So the next logical question is, why does the nCPU option remain?

Can you please try and work with your clients here and provide some detailed documentation that makes sense of the situation? I'm disappointed I had to come to the forum to seek clarification of something that should've been simple enough to explain in your documentation. I'm also very disappointed that with all the posts we have made, you didn't address them properly and instead just gave short brief responses.
  1. 21.07.2014 08:07:46
  2. # 20
Richard Hordern Accepted Answer
Posts: 212
Joined: 19.03.2011
0
Votes
Undo
In the lve cpanel interface 6 % and 2 cpu\'s was converted to 197% and 2 cpu\'s. Should\'nt the 2 cpu\'s setting be empty ? Does the cpanel interface use speed and ignore the ncpu setting ? Or did it increase all my accounts settings ?
  • Page :
  • 1
  • 2


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.

EU e-Privacy Directive

We use cookies to ensure you get the best experience using our website and services. Read more about it in our Privacy Policy. Please agree to the use of cookies to proceed. Alternatively, you may disable cookies in your browser at any time.

You have declined cookies. This decision can be reversed.

You have allowed cookies to be placed on your computer. This decision can be reversed.