Typically - you should never have more than 2GB of swap. On shared hosting servers - you're better of with 0.5 or no swap at all.
Let us explain the problem.
1. In ancient times, swap was used so that computer can do its job (even if slow) even in the case when a program(s) doesn't fit into memory. We are past that.
2. In more modern time, it is mostly used so some of the programs that are not used right now could be pushed out of RAM, giving some extra RAM for caches/buffers to get extra performance out of the server.
Note that in that case, only programs that are rarely used (at most once for several minutes, more like once in a few hours) are good candidates for that.
On a typical shared hosting server - there might be about 1/2GB of such programs.
Now, having more swap sounds like a non-issue until your server doesn't have problems.
Yet, it becomes a problem if something "happens" that causes server to use up all the RAM.
With no/small swap, OOM killer (a kernel thread that monitors memory consumption) will detect that server doesn't have enough memory - decide which process to kill, and will kill it.
It will do it for as long as it needed, killing processes left and right.
This will result in servers acting badly, giving out 500 errors, or even stop serving HTTP requests at all. Yet, you will still be able to get into the server via SSH, and figure out/fix whatever is happening.
With the large swap, OOM killer (rightfully) thinks that there are plenty of memory left, so it doesn't act. Processes that were getting more ram, are the one that needs to be running (based on OS), so everything else will be pushed out to swap. That includes Apache/PHP processes, SSH, MySQL, etc...
And whatever is in swap - is VERY slow. 1000s of times slower...
HTTP doesn't work. Requests timeout - though OS tries to bring up Apache/PHP from swap, causing something else into swap, trashing disks.
SSH doesn't work... probably timeouts as well. Maybe you can get it to work if you wait for 30 minutes, as it competes with all the HTTP requests/apache processes.
Logging doesn't work - your IO overloaded with all the swap disk IO that are trashing your disk.
As a result, you get an unresponsive server, and often - no idea what happened.
Our advice - shrink swap to at most 0.5G-1GB, or turn it off completely, as by the time you have 2GB in SWAP - your server is hosed.