reaching MaxClients
Forum
  1. Forums
  2. General
  3. General Discussion
  1. Pawel Panek
  2. Wednesday, 11 June 2014
  3.  Subscribe via email
Hi there!

Recently I faced an issue with Apache httpd unable to server content due to reaching MaxClients setting. I found it's caused by requests going to one of the sites hosted on this server. Initially I thought - it's DDoS but later I found those are legitimate http request. The problem was the httpd stopped responding to any other request due to reaching MaxClients setting effectively taking down all sites on this server.
The server runs CloudLinux system with mod_hostinglimits and has kinda low MaxEntryProcs setting for this user - EP=200. It seems to be working good. I saw the users being limited within its LVE settings and getting 503 or 508 for most of these enormous requests. Here comes the issue - httpd still needs to serve those 503 or 508 pages and when it's getting more and more requests it will eventually hit MaxClients setting.

Short term solution for this is just to rise MaxClients. That's not a good solution though. What if we could set MaxClients per vhost?

The mod_itk provides such setting. I'm not sure how it really works but if this can prevent a single tenant from taking down whole server I would say it's good solution.

I wish to see mod_hostinglimits can have similar feature. Please consider adding such feature on CloudLinux roadmap.

How to recreate:
1. Setup CMS site (Wordpress, Joomla)
2. Set MaxClients for mpm you use to 2048
3. On some other, decent server run: siege -c4000 this.site.url/index.php
4. Watch server-status or error log
Rate this post:
  1. 12.06.2014 07:06:28
  2. # 1
Igor Seletskiy Accepted Answer
Posts: 1200
Joined: 09.02.2010
0
Votes
Undo
maxentryprocesses are basically maxclients per user. Just change to * in mod_hostinglimits for allowed handlers. The question is what is your max clients for the server in total.
  1. 17.06.2014 12:06:26
  2. # 2
Pawel Panek Accepted Answer
Posts: 12
Joined: 03.11.2011
0
Votes
Undo
Hi Igor,

I tried setting AllowedHandlers to * but it didn\'t change this behavior. I thought it\'s set correct since I\'m requesting .php file which has been listed in allowed handlers. Besides, AllowedHandlers * didn\'t help.

When MaxClients is set to 4096 you can \'kill\' http server with following test:

ab -c4000 -n80000 http://domainname.tld/index.php

Test site should be some kind of PHP app, eg.: CMS. In the test I used EP=200 but I think EP value doesn\'t matter much in this case

I know it\'s DoS but what I would expect is mod_hostinglimits could protect other sites when a single site is being DoS\'ed. I\'m not sure if MaxClientsPerVhost from mod_itk can do it but it sounds like it does.
  1. 26.06.2014 13:06:03
  2. # 3
Pawel Panek Accepted Answer
Posts: 12
Joined: 03.11.2011
0
Votes
Undo
Update -
I can even change EP to 10 for test site. The result is the same - Apache server will refuse requests for other sites due to reaching MaxClinets limit. What do you think, is it possible to isolate that big number of requests for single vhost so it doesn\'t affect other sites?
  1. 26.06.2014 13:06:58
  2. # 4
Igor Seletskiy Accepted Answer
Posts: 1200
Joined: 09.02.2010
0
Votes
Undo
Of course you can. EP is part of the apache. It cannot go beyond MaxClients. It is not a firewall option. It is there so that once site is slow, and requests to it start to accumulate -- it would not wait for PHP pages to finish in 20 minutes, but rejected requests to those PHP pages right away - preventing natural accumulation of such things.

Sorry, but we are not trying to solve DDoS attacks where a client can create 100,000 connections to your server, or saturate your uplink with 10gbps of traffic.
  1. 26.06.2014 13:06:47
  2. # 5
Pawel Panek Accepted Answer
Posts: 12
Joined: 03.11.2011
0
Votes
Undo
I\'m intentionally trying to avoid using words - DoS or DDoS because the case I\'m having here is real life traffic. Yes, it\'s DoS-like but requests are fully legit.

Also it\'s not about reaching link capacity - bandwidth nor packet rate. That ab test I posted earlier can stop the server much earlier before reaching (D)DoS volumes.

The perfect solution I see here is to have a setting that will just reset tcp connection after reaching \'MaxClientsPerVhost\'. When this setting is higher than EP connecting clients would get 508 error while EP < current # of connections < MaxClientsPerVhost. All connections above MaxClientsPerVhost should get tcp connection reset without serving error page. Of course initial handshake is required to get Host header from client. Then module can check connections counter and eventually decide to close the connection without further processing. In case MaxClientsPerVhost < MaxClients (server wide) then the server should stay safe.

Anyways, I think feasible solution here is to keep increasing MaxClients so Apache can serve 508\'s or 503\'s for site that is already above EP.
  1. 26.06.2014 13:06:29
  2. # 6
Igor Seletskiy Accepted Answer
Posts: 1200
Joined: 09.02.2010
0
Votes
Undo
If you have fully legit traffic with 4000 connections to a single vhost -- don\'t use apache, as it is not feasiable to keep increasing maxclients -> you will run out of memory.
Try switching to LiteSpeed or install nginx as a proxy in front of apache.
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!
Guest
Submit Your Response
Upload files or images for this discussion by clicking on the upload button below. Supports gif,jpg,png,zip,rar,pdf
• Insert • Remove Upload Files (Maximum File Size: 2 MB)
Captcha
To protect the site from bots and unauthorized scripts, we require that you enter the captcha codes below before posting your question.