I havent had Apache configured well, so I upped the maximum connection limit which really boosted performance.
Then I installed mod_security which did not do the trick against the "slowloris attack" So I installed antiloris on Apache and now When I Dos myself my connections get refused!
For people wanting to know if their servers are "slowloris proof" you can google for the utility, then type in the url of your site server.
You can then look up here if your server is fending off the attack:
This worked for me very well and I hope this helps other people with this issue