Extremely high load
Forum
  1. Forums
  2. CloudLinux and Control Panels
  3. CloudLinux and cPanel
  1. Chris Gibson
  2. 28.02.2012
  3.  Subscribe via email
High load, over 140
Rate this post:
  1. 28.02.2012 12:02:13
  2. # 1
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
So about 20 minutes ago I noticed WHM not responding. I checked the server status and load was around 40, by the time I got logged in via SSH it was up over 140..... :(

I restarted cpanel and load has came down. I have been getting alot of brute force notifications and I have been blacklisting them.

However, I was under the impression that this is exactly what Cloud Linux is for(stopping high resource utilization)


Any tips or pointers would be great.
  1. 28.02.2012 12:02:54
  2. # 2
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
Yes, it should have stopped high resource utilization

if you could run top and send a copy of the output when something like this happens -- it would help diagnose the issue. As it is really hard to tell what exactly might have happened.
  1. 28.02.2012 12:02:27
  2. # 3
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
I was struggling to get anything done.

I guess I am wondering if I missed anything during setup.


Currently there are 375 of these processes


/usr/local/cpanel/bin/dovecot-auth /usr/libexec/dovecot/checkpassword-reply
  1. 28.02.2012 12:02:38
  2. # 4
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
I see what is going on. We actually don\'t protect against this particular thing. We don\'t limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.

It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don\'t do it out of the box.
  1. 28.02.2012 13:02:35
  2. # 5
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
ok, I will look into that. Thanks!

On another note, how do I verify that I configured everything correctly?
  1. 28.02.2012 13:02:56
  2. # 6
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
Easiest way -- run lvetop
if you see multiple active users there -- it is configured correctly.
  1. 28.02.2012 13:02:53
  2. # 7
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
Hmmm, it is only showing 1 user out of the 2 it should be showing....
  1. 28.02.2012 13:02:47
  2. # 8
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
Is there active traffic for both users?
Try hitting second user with
ab -c 2 -n 10000 URL_TO_SECOND_USER_SITE

and see if that user appears in lvetop
  1. 28.02.2012 13:02:51
  2. # 9
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
Sorry that is 1 out of 3, and that is only showing a ssh only user, not any cpanel users.
  1. 28.02.2012 13:02:03
  2. # 10
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
Cool thanks, works for all but 1 client
  1. 28.02.2012 14:02:44
  2. # 11
John MacKenzie Accepted Answer
Posts: 14
Joined: 02.04.2011
0
Votes
Undo
Igor Seletskiy wrote:
I see what is going on. We actually don\'t protect against this particular thing. We don\'t limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.
It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don\'t do it out of the box.


Igor how do you actually add another service into an LVE wrapper? more specifically i want to add in r1soft CDP since its causing some load issues (sorry if i should have created a new thread)

Thanks
John
  1. 28.02.2012 14:02:49
  2. # 12
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
/proc/lve/list

Is missing one of the clients... How do I add it?
  1. 28.02.2012 14:02:22
  2. # 13
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
You don\'t need to add them. They should be picked up automatically on SSH/php via web access.

If they don\'t show up after you hit them with ab -- check that you have mod_hostinglimits installed
If you are running plesk -- make sure you have SuexecUserGroup directive (cgi enabled) for those customers.
  1. 28.02.2012 14:02:56
  2. # 14
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
John,

Basically if you have a command like:

/usr/bin/runthis

And you want to run it inside LVE, execute it via:
/usr/sbin/lve_wrapper 33 /usr/bin/runthis

Where 33 is LVE id -- you can use any number here.


All children of that process will be in the same lve, and you can control them all.

If you want to continue to execute /usr/bin/runthis (for example because WHM executes it that way), do:
mv /usr/bin/runthis /usr/bin/runthis.orig
echo <<DONE > /usr/bin/runthis
#!/bin/bash
/usr/sbin/lve_wrapper 33 /usr/bin/runthis.orig
DONE

chmod +x /usr/bin/runthis

The only problem with such approach is that if software updates/overwrites runthis, you need to re-create it.
  1. 28.02.2012 14:02:33
  2. # 15
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
Igor Seletskiy wrote:
You don\'t need to add them. They should be picked up automatically on SSH/php via web access.



If they don\'t show up after you hit them with ab -- check that you have mod_hostinglimits installed

If you are running plesk -- make sure you have SuexecUserGroup directive (cgi enabled) for those customers.


What is the correct way to install it?

Doing a search I see these packages:
[email protected] [/scripts]# yum search mod_hostinglimits
Loaded plugins: fastestmirror, rhnplugin
Loading mirror speeds fr om cached hostfile
* cloudlinux-x86_64-server-5: slc-proxy.cl-mirror.net
Excluding Packages in global exclude list
Finished
========================================================================== Matched: mod_hostinglimits ==========================================================================
hsphere_mod_hostinglimits.x86_64 : Apache module
hsphere_mod_hostinglimits1_3.x86_64 : Apache module
mod_hostinglimits.x86_64 : Apache module
mod_hostinglimits-debuginfo.x86_64 : Debug information for package mod_hostinglimits
mod_hostinglim its-hsphere.x86_64 : Apache module
  1. 28.02.2012 14:02:19
  2. # 16
Igor Seletskiy Accepted Answer
Posts: 1201
Joined: 09.02.2010
0
Votes
Undo
You are running cPanel, so mod_hostinglimits is installed via easyapache

You can check if it is here by running:
/etc/init.d/httpd -M|grep hosting

if it is not there, please, run:
yum install lve-utils lve-stats
/scripts/easyapache --build
  1. 28.02.2012 14:02:06
  2. # 17
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
Hmmm, it is there....

Ahh, I wasn\'t testing with a php file for the client. Once I did it works
  1. 28.02.2012 14:02:17
  2. # 18
Chris Gibson Accepted Answer
Posts: 10
Joined: 28.02.2012
0
Votes
Undo
FYI, Awesome help Igor.

I am heading over to cpanel forums to figure out why the limit on dovecot auth isn\'t working, have over 600 process\'s of just that again.
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.