server security - question

Discussion in 'Server Operation' started by Ovidiu, Nov 18, 2006.

  1. Ovidiu

    Ovidiu Active Member

    hi there,

    until 5 mins ago I thought I had setup my server secure and nice, I use mod_security. dos_evasive and other general settings to prevent attacks, but severall minutes ago I almost broke down my own server checking one of my sites for broken links. I used xenu sleuth, set it to 100 parallel threads and told it to go max. 10 links deep.

    I expected to get a lot of errors because the dos_evasive module should block me after so many events for 10 mins, but no my serevr went crazy: check it out here:

    traffic broke down, my apache processes maxed out at 80 or so (using a lot of swap memory), my load went to 40-60 and the cpu freaked out.

    any ideas what to check? my guess is that my mysql database had too many connections and broke down - just a guess though...

    I forgot to mention I only got a 512/256 line at home, so how could I possibly almost kill my server?
  2. falko

    falko Super Moderator ISPConfig Developer

    Anything in Apache's error log?
  3. Ovidiu

    Ovidiu Active Member

    well, no nothing in there, as much as the system was concerned nothing unusual was happening, still I could get my server down with my 512/256 line from home?

    the system continued serving or at least trying to serve my link test program files,... apache's processes maxed out, it swapped 1GB out, all other traffic went down ... load was around 40-60, what more coudl I tell you?

    BUT: mod_security and dos_evasive still work, if you try to do severall refreshes one after another, very wuick, you'll get a 403 error and are blocked for severall minutes....
    I am just wondering what the bottleneck was apache2 or mysql? and why did my link test tool did not get blocked?

    anyone intersted can run a test with xenu sleuth, set it to 100 threads and check links 10 deep...
  4. falko

    falko Super Moderator ISPConfig Developer

    I think you should install munin to find out why your server had such a high load.
  5. Ovidiu

    Ovidiu Active Member

    ok, I installed monit and muni according to the howto flaoting around here :)

    I'll let them run for a few days, to understand how its running when not stressend, then I guess I have to stop monit, so it doesn't interfere and start my "attack" again to see what happen, right?

    I'll pm you the links after I did so that you can check what happened too.

    thx for the help
  6. Ovidiu

    Ovidiu Active Member

    hello falko

    I have some sever problems right now. I do nothing, didn't change anything inside apache, what happens is this:

    normal traffic flow, then all of a sudden, apache starts children until it maxes out, starts swapping and server is slooowwww due to this... because it is so slow, there is hardly any more traffic flowing right now.

    I'll pm you the link to the serverstats... p.s. where you see sudden changes in apache children its where monit restarted apache. I stopped monit now so I can see what happens.
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Please have a look at your access.log with the comaind "tail -f ..." Which requests where send to your server when the problem starts? This beahaviour happens often when a not well programmed search engine spider hits your server and requests a large number of pages in a very short time.
  8. Ovidiu

    Ovidiu Active Member

    I'll try that next, meanwhile:

    why the hell is bind swapping that much and why does it need 28 processes???

    and besides that I have 46 apache2 processes, each using 24M and 23M swap..
    Last edited: Nov 22, 2006
  9. Ovidiu

    Ovidiu Active Member

    I think I figured it out for the time being: I had redesigned a wpmu site and had it display a sitewide feed on the mainpage, when I used tail -f to see what was going on, I saw that feed was being requested hundreds of times, must be soem errors, there, I deactivated it and now everythings seems calm...
  10. Ovidiu

    Ovidiu Active Member

    another strange happening, have a look at this top-screen:

    if I do a: tail -f /var/log/httpd/ispconfig_access_log I see: one IP being VERY active, after banning him all is good again, but how can I digg deeper, see what the hell was causing this load?
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Does the IP calls always the same file? If yes, inspect the file.
  12. Ovidiu

    Ovidiu Active Member

    well this thing happened again, arouns 2:00 last night, my server started spawning max. children, opened max. mysql connections and strangled itsself to death... :)

    I'll try and attach some statistics from munin and hotsanic. I'll also have a look what was happening when this all started and get back here with that info.

    Attached Files:

  13. falko

    falko Super Moderator ISPConfig Developer

    Maybe it was a bot or something like that. Check Apache's access log and try to find out if the requests came from on IP.
  14. Ovidiu

    Ovidiu Active Member

    it did not seem to be one ip, nevertheless I am back with some nice stats :) check the attached file

    funny, isn't it :) this time there were serverall ips involved too, and as I am checking my log fiels manually, I can't spot a pattern....
    do you know of a nice tool, maybe local installed (windows?) which can analyze the logfile if I downlaod it?

    Attached Files:

  15. falko

    falko Super Moderator ISPConfig Developer

    Unfortunately no. :(

Share This Page