server blocked/stopped by host

[Tenaka]

Hi guys,

my strato server has stopped working on sunday, I have been trying to fix it the whole last night, but then realized that inside my control panel from strato it said: host has been suspended please contact support.

I did so and they stated:

Leider wurde Ihr Server am Wocheende Ziel einer DoS-Attacke. Um unsere Infrastruktur und Ihren Server vor weiteren Folgeschäden zu bewahren, mussten wir daher Ihren Server kurzfristig vom Netz nehmen.

Wir haben den Server heute vormittag wieder entsperrt und innerhalb der nächsten Stunde sollte der Server wieder wie gewohnt zur Verfügung stehen.

this means my server was hit by a Dos attack and my host has cut it of from the internet to prevent damage.

I then asked if I could see evidence of a DOS attack and what they plan to do against the initiators.

They responded:

Hiermit möchten wir Sie informieren, dass uns unsere Techniker vom STRATO Rechenzentrum darüber in Kenntnis gesetzt haben, dass auf Ihrem Server mit dem Hostnamen: hxxxx.serverkompetenz.net mit der Auftragsnummer: xxxxxx eine fehlerhafte Anwendung ausgeführt wird.

Dies fiel durch eine nicht normal hohe Anzahl von Paketen (21001 eingehende packets/sec) auf. Dieser Vorgang beeinträchtigt die technische Infrastruktur der STRATO Medien AG auf nicht akzeptable Weise und verstößt damit gegen die Regelbetriebsbedingungen unserer Allgemeinen Geschäftsbedingungen, die Sie jederzeit im Internet einsehen können: http://www.strato.de/full/STRATO/agb.html

Es ist möglich, dass sich ein Dritter Zugang zu den root-Benutzerrechten auf Ihrem Server verschafft hat.
Falls Sie die Vermutung hegen, dass ein Dritter Ihren Server gehackt hat, empfehlen wir Ihnen eine Sicherung Ihrer persönlichen Dateien auf dem Server und eine Neuinstallation des Servers.

Erzeugen Sie mit dem Befehl "tar cvfz /backup.tgz <verzeichnisname>" Backupdateien Ihrer persönlichen Dateien.
Geben Sie dann den Befehl "/etc/init.d/networking stop" ein, booten Sie danach im RecoveryModus und kopieren Sie Ihre Dateien per FTP oder SSH auf einen anderen Server.
Nach erfolgter Neuinstallation setzen Sie uns in Kenntnis, dass wir den Server wieder freischalten können. Als Termin haben wir uns spätestens den <date>(0,".",1,1,7) vorgemerkt.

here they try to tell me that there is an application on my server which causes too high load 21001 packets/sec and that my server could have been hacked - they suggest that I backup my server and reinstall...

this seems highly contradictory to me as they told me about DOS first then state I have a rogue application that has a high number of incoming packets - ??? AND in their first email they told me my server would be back online in a few hours, after I asked for more details they come up with this story about this application...

Any suggestions? I just mailed them and asked about clarification.

I'd apreciate any help.

[till]

Hi,

I recommend that when your server is back onlien, you do a scan for rootkits.

http://www.howtoforge.com/faq/1_38_en.html

I think that an high volume of incoming packets does not mean automatically that your server is highjacked. It depends on the port. If e.g. a high number of packets where send to port 80, its an DOS attack but it does not mean that your apache has been highjacked.

[Tenaka]

thx for the hint, I will check after work.

I got another mail from them: please excuse our second mail (about the rogue application) it was sent by mistake to you and was not intended for you :-))

I have seen this happen frequently when the guys doing the support know nothing about their job and just skim the support seeking mails for keywords and then select from a database of ready made emails and send them back

Or they really sent me that mail by mistake...

anyway I'l check back later and report what happened

[Tenaka]

ok guys, it happened again :-(

I'll give you a short summary in english:

I got an email from strato saying that my server was the target of another DOS attack. They sent me a link to their fair usage document where they say are rules against which I am acting (by letting myself getting attacked !)

http://www.strato.de/full/STRATO/agb.html

They will cut my server of the net to prevent damage to their networks...
I can still access my server through a remote recovery console and I should resolve my problems.

If I have any explanations as to why I am being attacked I shall notify them and they tell me to do anything possible to prevent myself from further attacks.

They gave me time to resolve the problems until the 17.02.2006 - what problems?

What the heck can I do if I am getting attacked?

I was still able to access the server through ssh for maybe an hour and neither the logs neither netstat was showing me an unusual amount of incoming connections, neither were there syn attacks to be seen at the first glance. Neither my graphical output of statistics showed anything...

Now I think that if there was an attack there had to be at least some traces of this attack in my logfiles.

I am going to write them a similar email asking for details about the type of the attack, the duration and log excerpts - I mean they must base their accusations upon something.


AND I think I have to change providers although I am usually a very steady customer but this is too much.

any suggestions or ideas?

###edit###

sorry I have to give you more info: the mail reached my with a huge severall hours delay, after I talked to them on the phone they gave me the exact time of the attack and I found some traces. I am currently examining the logfiles.

not much inside the usual syslog files, any other place where I can grab some info? If not can anyone explain how to setup some firewall rules to log attacks of any kind ? just some hints or so would be great.

[falko]

I'd check with chkrootkit and rootkithunter first that you didn't get hacked.

Maybe you can monitor with Nagios ( http://www.nagios.org/ ) what's happening on your network interface.

[Tenaka]

I am currently checking out nagios screenshots to find out what it can do and if it got more info than hotsanic I am using curretnly.
besides I checked if I got hacked, did it last time too, and there is no sign of such activity, anyway I am talking about an incoming attack - there was a traffic spike at 23:50 during the night from thursday to friday. you can check it out here: http://www.web-designerz.de/serverstats/traffic/eth0.html

these stats are all I have there is no more logging being done - so what can I do?

[till]

I think there is not much that you can do if there is an incoming attack. Only your provider can try to block the traffic at the routers in front of your server.

[Tenaka]

I think there is not much that you can do if there is an incoming attack. Only your provider can try to block the traffic at the routers in front of your server.

I know and they won't :(
I am talking about strato, they even told me that they usually ask people to sign a statement that one won't do it again (Unterlassungserklärung) or even cancel their contract, so soon, if I don't start logging so I can file an abuse report to those f***ing idiots attacking my server I will have top look for a new server provider.

so if someone could suggest some logging rules or at least give me some hints as to where and what to log...

[till]

Did you have any idea which port / daemon that may have attacked? Is any logfile for that day bigger as usual?

As far as i know there are some config options in the bastille firewall config file that ISPConfig uses to enable logging, but i never tested it.

[falko]

Since these traffic peaks are occurring regularly - is it maybe a cron job that is causing this? (Or is this just your nightly backup to the Strato FTP server?)

[Tenaka]

@till
nothing noticeable in the logfiles neither in their size, thats why I was asking for help with logging...

@falko
I have had another attack these days, the peaks I am talking about are the green ones, thats incoming traffic. The red ones are indeed the nightly backups which indicates outgoing traffic.

these attacks did not do any harm, the server is still available as I found out by chance. The last attack was registered at midnight, but Strato only cut off my server the next day at lunch so I was able to see what happened: the server load went up to 12 and more, still everything was functionable, the DOS did not succeed, to me it looks like the attacking server was not powerfull enough?

yet strato does not like these attacks and threatened to cancel my contract if there will be any more occurances. So I need logging to file abuse complaints about the originating server.

[till]

Did you had a look at bastille for logging?

/etc/Bastille/bastille-firewall.cfg

There are some options that might help you:

# 12) Logging (all systems)
# With this enabled, ipchains will log all blocked packets.
# ** this could generate huge logs **
# This is primarily intended for the port mointoring system;
# also note that you probably do not want to "AUDIT" any services
# that you are not allowing, as doing so would mean duplicate
# logging
LOG_FAILURES="N" # do not log blocked packets

# 13) Block fragmented packets (all systems)
# There's no good reason to allow these
#ALLOW_FRAGMENTS="N" # safest
ALLOW_FRAGMENTS="Y" # old behavior

# 14) Prevent SMB broadcasts from leaking out NAT setup (all systems)
# Windows machines will poll teh net with SMB broadcasts,
# basically advertising their existence. Most folks agree
# that this traffic should be dropped
#DROP_SMB_NAT_BCAST="N" # allow them (are you sure?)
DROP_SMB_NAT_BCAST="Y" # drop those packets

# 15) Log level (iptables/netfilter/Linux 2.4 only)
# Control what level of logging is used when the firewall logs
# information. Default is warning (4). Lowest priority is
# debug (7); highest is emergency (0). To prevent syslog
# from copying iptables error messages to the console, set
# this to 6 (7 would also work, but 6 is recommended)
# You can also stop syslogd/klogd from printing kernel
# messages to the console by issuing the command
# setterm -msg off
#IP_LOG_LEVEL=6 # level used in 2.2/ipchains
IP_LOG_LEVEL=4 # iptables/netfilter default

# 16) Always attempt to use stateful features for inbound connections
# Always using state will allow the firewall to reject invalid
# packets sent to otherwise open TCP services, e.g. XMAS, NULL
# and SIN/FYN scans. The downside to choosing this behavior is that
# services may become unreachable if the packet filter's state
# table becomes full.
IP_ALWAYS_USE_STATE="N" # default, ensures services remain available
#IP_ALWAYS_USE_STATE="Y" # disallow invalid packets