Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 18th November 2006, 22:26
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Question server security - question

hi there,

until 5 mins ago I thought I had setup my server secure and nice, I use mod_security. dos_evasive and other general settings to prevent attacks, but severall minutes ago I almost broke down my own server checking one of my sites for broken links. I used xenu sleuth, set it to 100 parallel threads and told it to go max. 10 links deep.

I expected to get a lot of errors because the dos_evasive module should block me after so many events for 10 mins, but no my serevr went crazy: check it out here: http://www.web-designerz.de/serverstats/

traffic broke down, my apache processes maxed out at 80 or so (using a lot of swap memory), my load went to 40-60 and the cpu freaked out.

any ideas what to check? my guess is that my mysql database had too many connections and broke down - just a guess though...

###edit###
I forgot to mention I only got a 512/256 line at home, so how could I possibly almost kill my server?
Reply With Quote
Sponsored Links
  #2  
Old 19th November 2006, 17:59
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,721 Times in 2,562 Posts
Default

Anything in Apache's error log?
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #3  
Old 19th November 2006, 20:54
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

well, no nothing in there, as much as the system was concerned nothing unusual was happening, still I could get my server down with my 512/256 line from home?

the system continued serving or at least trying to serve my link test program files,... apache's processes maxed out, it swapped 1GB out, all other traffic went down ... load was around 40-60, what more coudl I tell you?

BUT: mod_security and dos_evasive still work, if you try to do severall refreshes one after another, very wuick, you'll get a 403 error and are blocked for severall minutes....
I am just wondering what the bottleneck was apache2 or mysql? and why did my link test tool did not get blocked?

anyone intersted can run a test with xenu sleuth, set it to 100 threads and check links 10 deep...
Reply With Quote
  #4  
Old 20th November 2006, 13:42
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,721 Times in 2,562 Posts
Default

I think you should install munin to find out why your server had such a high load.
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #5  
Old 20th November 2006, 23:45
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

ok, I installed monit and muni according to the howto flaoting around here :-)

I'll let them run for a few days, to understand how its running when not stressend, then I guess I have to stop monit, so it doesn't interfere and start my "attack" again to see what happen, right?

I'll pm you the links after I did so that you can check what happened too.

thx for the help
Reply With Quote
  #6  
Old 22nd November 2006, 11:48
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

hello falko

I have some sever problems right now. I do nothing, didn't change anything inside apache, what happens is this:

normal traffic flow, then all of a sudden, apache starts children until it maxes out, starts swapping and server is slooowwww due to this... because it is so slow, there is hardly any more traffic flowing right now.

I'll pm you the link to the serverstats... p.s. where you see sudden changes in apache children its where monit restarted apache. I stopped monit now so I can see what happens.
Reply With Quote
  #7  
Old 22nd November 2006, 11:57
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,419
Thanks: 812
Thanked 5,205 Times in 4,081 Posts
Default

Quote:
Originally Posted by Tenaka
hello falko

I have some sever problems right now. I do nothing, didn't change anything inside apache, what happens is this:

normal traffic flow, then all of a sudden, apache starts children until it maxes out, starts swapping and server is slooowwww due to this... because it is so slow, there is hardly any more traffic flowing right now.

I'll pm you the link to the serverstats... p.s. where you see sudden changes in apache children its where monit restarted apache. I stopped monit now so I can see what happens.
Please have a look at your access.log with the comaind "tail -f ..." Which requests where send to your server when the problem starts? This beahaviour happens often when a not well programmed search engine spider hits your server and requests a large number of pages in a very short time.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #8  
Old 22nd November 2006, 12:45
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

I'll try that next, meanwhile:

Quote:
Mem: 1035444k total, 1030560k used, 4884k free, 4544k buffers
Swap: 996008k total, 489652k used, 506356k free, 23540k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
5810 mysql 15 0 146m 28m 6556 S 0.3 2.8 0:52.82 118m mysqld
14515 www-data 16 0 46780 12m 24m R 2.1 1.2 0:01.18 33m apache2
847 bind 18 0 30468 792 4248 S 0.0 0.1 0:00.00 28m named
16057 bind 19 0 30416 792 4248 S 0.0 0.1 0:00.00 28m named
2268 bind 17 0 30416 796 4248 S 0.0 0.1 0:00.00 28m named
15516 bind 22 0 30416 804 4248 S 0.0 0.1 0:00.00 28m named
16003 bind 18 0 30416 804 4248 S 0.0 0.1 0:00.00 28m named
16228 bind 18 0 30416 804 4248 S 0.0 0.1 0:00.00 28m named
17191 bind 20 0 30416 804 4248 S 0.0 0.1 0:00.00 28m named
16407 bind 17 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
15762 bind 20 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
16528 bind 22 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
16629 bind 18 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
16857 bind 17 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
16899 bind 25 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
17697 bind 20 0 30416 808 4248 S 0.0 0.1 0:00.00 28m named
why the hell is bind swapping that much and why does it need 28 processes???

and besides that I have 46 apache2 processes, each using 24M and 23M swap..

Last edited by Ovidiu; 22nd November 2006 at 12:49.
Reply With Quote
  #9  
Old 22nd November 2006, 13:33
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

I think I figured it out for the time being: I had redesigned a wpmu site and had it display a sitewide feed on the mainpage, when I used tail -f to see what was going on, I saw that feed was being requested hundreds of times, must be soem errors, there, I deactivated it and now everythings seems calm...
Reply With Quote
  #10  
Old 15th December 2006, 09:21
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
 
Default

another strange happening, have a look at this top-screen:

Quote:
top - 09:16:30 up 80 days, 15:18, 1 user, load average: 5.59, 5.28, 3.91
Tasks: 275 total, 7 running, 268 sleeping, 0 stopped, 0 zombie
Cpu(s): 92.3% us, 7.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1035444k total, 974100k used, 61344k free, 21524k buffers
Swap: 996008k total, 191044k used, 804964k free, 213392k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27985 www-data 17 0 78120 34m 56m R 37.5 3.4 0:21.91 apache2
29406 www-data 25 0 74188 30m 56m R 13.6 3.0 1:02.57 apache2
30647 www-data 25 0 72720 27m 56m R 13.3 2.7 0:24.91 apache2
29434 www-data 25 0 74432 30m 56m R 10.3 3.0 0:55.08 apache2
29504 www-data 21 0 74580 30m 56m R 10.3 3.0 0:28.11 apache2
30609 www-data 25 0 74216 29m 56m R 10.3 2.9 0:30.38 apache2
if I do a: tail -f /var/log/httpd/ispconfig_access_log I see: one IP being VERY active, after banning him all is good again, but how can I digg deeper, see what the hell was causing this load?
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
php Apps email not going through palkat General 8 21st September 2011 05:35
Statistic not working mzo Installation/Configuration 49 20th April 2011 12:19
Setting up a backup mail server setup with two installations of ISPConfig zitch Tips/Tricks/Mods 7 30th December 2006 10:07
Email - Ueb-Miau mazhar Installation/Configuration 5 21st December 2005 10:01
The Perfect Setup Suse 9.3 - Postfix problems new_bee05 HOWTO-Related Questions 20 25th November 2005 02:30


All times are GMT +2. The time now is 19:46.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.