How To Defend slowloris DDoS With mod_qos (Apache2 On Debian [Lenny])

Want to support HowtoForge? Become a subscriber!
Submitted by MarcusSpiegel (Contact Author) (Forums) on Wed, 2009-07-15 19:06. :: Debian | Linux | Linux | Apache | Apache | Security | Security

How To Defend slowloris DDoS With mod_qos (Apache2 On Debian [Lenny])

From the Slowloris description:

Slowloris is designed so that a single machine (probably a Linux/UNIX machine since Windows appears to limit how many sockets you can have open at any given time) can easily tie up a typical web server or proxy server by locking up all of it's threads as they patiently wait for more data. Some servers may have a smaller tolerance for timeouts than others, but Slowloris can compensate for that by customizing the timeouts. There is an added function to help you get started with finding the right sized timeouts as well. As a side note, Slowloris does not consume a lot of resources so modern operating systems don't have a need to start shutting down sockets when they come under attack, which actually in turn makes Slowloris better than a typical flooder in certain circumstances. Think of Slowloris as the HTTP equivalent of a SYN flood.

I recently had to defend a live attack with slowloris-dos from a botnet. The load-impact is very low but http quits serving very fast. A quick approach was to mangle with timeout settings, wich is fine to defend a single attacker but leads into new issues (ie. large NAT on client-side).

mod_qos gives some fine-grained opportunities to scale the number of used connections and to defend an attack according to bandwidth limits. Unfortunately it is only available as source-package and there are many possible settings, wich might be hard to setup for this special case. So I provide the way that helped me.

 

1. Get the source, build & install

mod_qos is available from sourceforge (http://sourceforge.net/projects/mod-qos/). You will find documentation here http://mod-qos.sourceforge.net/.

cd /tmp/
wget http://downloads.sourceforge.net/sourceforge/mod-qos/mod_qos-8.13-src.tar.gz?use_mirror=freefr
tar xvfz mod_qos-8.13-src.tar.gz

You might want to copy & paste the direct link from sourceforge. As we want to compile mod_qos by use of apxs, we need to install the appropriate dev package and gcc of course, ie:

apt-get install apache2-threaded-dev gcc

Now build & install

cd mod_qos-8.13/apache2/
apxs2 -i -c mod_qos.c

If everything worked fine you'll get something like this:

----------------------------------------------------------------------
Libraries have been installed in:
/usr/lib/apache2/modules
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,--rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 644 /usr/lib/apache2/modules/mod_qos.so

 

2. Now configure and activate

Go to /etc/apache2/mods-available and add a qos.load and qos.conf file

cd /etc/apache2/mods-available/
vi qos.load

LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

vi qos.conf

## QoS Settings
<IfModule mod_qos.c>
    # handles connections from up to 100000 different IPs
    QS_ClientEntries 100000
    # will allow only 50 connections per IP
    QS_SrvMaxConnPerIP 50
    # maximum number of active TCP connections is limited to 256
    MaxClients              256 
    # disables keep-alive when 70% of the TCP connections are occupied:
    QS_SrvMaxConnClose      180
    # minimum request/response speed (deny slow clients blocking the server, ie. slowloris keeping connections open without requesting anything):
    QS_SrvMinDataRate       150 1200
    # and limit request header and body (carefull, that limits uploads and post requests too):
    # LimitRequestFields      30
    # QS_LimitRequestBody     102400
</IfModule>

Now you need to enable the module and restart apache:

a2enmod qos
/etc/init.d/apache2 restart

If you are able to get the server-status by https://__yourserver__/server-status, you'll find mod_qos enabled and working by giving a statistical summary, like this:

mod_qos 8.13

viewer settings
client ip connections

<!--0--> <!--0-->
__yourserver__:0 (base)
connections
free ip entries
255
current connections
1
client ip connections
current
aaa.bbb.ccc.ddd 1
connection settings
max connections
-
max connections with keep-alive
180
max connections per client ip
50
min. data rate (bytes/sec) (min/max/current)
150/1200/154
__yourserver1__:443 (virtual)
uses base server settings
__yourserver1__:80 (virtual)
uses base server settings
__yourserver2__:443 (virtual)
uses base server settings
__yourserver2__:80 (virtual)
uses base server settings

 

3. before & after

On top of the server-status page you can see a summary of all apache workers. To see what happens before and whilst the attack, I used a local environment. So the results here show the basic principle not the real world example.

Let's assume a normal server-load like this:

1 requests currently being processed, 5 idle workers
_.._W._...._...................................................
................................................................

While attacking this server with one single machine results to all workers being occupied:

102 requests currently being processed, 0 idle workers
RRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR..........................

As you can see the attack took all available threads of the apache server wich itself stays in state "R" (Keepalive (read)). A normal apache will wait for 300 seconds to close such a connection and as the attacker doesn't send any real request and just keeps all open connections, No other requests will be served anymore.

Now with the QoS-Module enabled, things look different. Although apache starts the maximum number auf child-workers, it is still able to serve "real-people-requests":

1 requests currently being processed, 97 idle workers
________W_______________________________________________________
__________________________________..............................

We configured apache with mod_qos to handle a limited number of connections per IP and to prefer serving "fast" requests. The attack just opens a connection without sending a real http-reqeuest (like GET /index.html HTTP 1.1). So apache now classifies that as a "slow" request as it has to wait very long to get a request/response from the attacker.

A real request sends the "GET" much faster together with additional http-headers. So in case that all workers are occupied by an attack whilst receiving a real request, apache just drops a stale connection and handles the new "serious" request at almost normal speed.

On a real world server you will get pictures like these (reload... reload... reload):

166 requests currently being processed, 0 idle workers
RRRRRCCCCRRRCRCRWRRRCRRCRCCRRCRRCCCRRRKCRRRRRRRRCRRRRRCRCCRC.RRC
C.RCR.RRC.CC.RCRR.RRRCR.C.C.RR.R.RRCCR.R.R...RRRRRRCCRRR.RRKC.CR
.RRR.C..RRRC.CCRRRR.RRC..R.CCRR.CR.RR.R.R.C.RR.C...C.C.RCRR.C.R.
..RRCRC.RCRCC..R.CCCW.............................R.............

97 requests currently being processed, 59 idle workers
___R_C_RR.CC_R_R_RRC_RCR._.R_RR.RRW___RR__RC______R_R_R__.K_._R.
RCRC___.CKRCRRRRRCR__CRR_RRRKR_RR.___.._._...RC..RR.CR.R.CRRR.R.
.R__._..RRRC.__R__R.R_R...._R_R.__.RR..._._..R.R...R.R.RR.R.R._.
.._RR_C.R.R_R.._.RRRR...........................................

51 requests currently being processed, 92 idle workers
___KR_R_K.._._.K__R__._..K.__._.R.W__R___R____.____K_____.__._R.
R_CKRK_.__RRRR_KK__RR.___RKR__R.R.__C..R._...__..R_.__._.K__R._.
.___._.._R__.C._RKR.__C....R__C.R_.__..._.C..K._...K._.._._._...
..___RK.R.___.._.____...........................................

126 requests currently being processed, 0 idle workers
RRKRRRRCRCCRRKRKCRRRKCRRRRRRCKCKRKRRRRRRKCRCCCRRKKKRRRRRC.....R.
.C.R.CR.RRC.RRRRRCCRR.RCCCRRRKR.K.R....R.R....R.....C..R.R.RR.R.
.........C.C....K.K.RRK....R..C.K..KC...C.C....R...C.R..W.C.R...
..KRR.R.R.RRR....RCRR...........................................

 

4. Links, etc.

I provide this howto as one possible way to solve this particular issue. There might be more ways (and hopefully will be), like using a proxy, loadbalancer, level-7-firewall, etc.

In my case, we had to find a quick solution that is up and running in minutes. For further reading you might find these links helpful:


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by alice (not registered) on Sun, 2011-10-30 10:21.
there are some error infos when i install this mod is pcre is needed the following is my step and error infos steps 1.sudo wget http://mirror.bjtu.edu.cn/apache/httpd/httpd-2.2.21.tar.gz sudo tar zxvf httpd-2.2.21.tar.gz cd httpd-2.2.21 sudo ./configure --prefix=/usr/local/apache2 --with-mpm=prefork --enable-rewrite --enable-so --enable-headers --enable-proxy sudo make sudo make install sudo wget http://sourceforge.net/projects/mod-qos/files/mod_qos-9.72.tar.gz/download sudo tar zxvf mod_qos-9.72.tar.gz cd mod_qos-9.72/apache2 sudo /usr/local/apache2/bin/apxs -i -c mod_qos.c and the following is the error infos: /usr/lib64/apr-1/build/libtool --silent --mode=compile gcc -prefer-pic -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -pthread -I/usr/local/apache2/include -I/usr/include/apr-1 -I/usr/include/apr-1 -c -o mod_qos.lo mod_qos.c && touch mod_qos.slo mod_qos.c:72:18: error: pcre.h: No such file or directory mod_qos.c:330: error: expected specifier-qualifier-list before 'pcre' mod_qos.c:344: error: expected specifier-qualifier-list before 'pcre' mod_qos.c:681: error: expected specifier-qualifier-list before 'pcre' mod_qos.c: In function 'qos_load_headerfilter': mod_qos.c:831: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:831: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:831: error: (Each undeclared identifier is reported only once mod_qos.c:831: error: for each function it appears in.) mod_qos.c:832: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:833: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:834: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:841: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:841: error: 'pcre_free' undeclared (first use in this function) mod_qos.c: In function 'qos_per_dir_event_rules': mod_qos.c:2152: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2154: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c:2155: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c:2159: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c:2165: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:2169: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2170: error: 'qos_rfilter_t' has no member named 'id' mod_qos.c:2171: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c:2171: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:2174: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c: In function 'qos_per_dir_rules': mod_qos.c:2611: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2613: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:2614: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2616: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:2617: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2619: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:2620: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2624: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:2625: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:2631: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:2635: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:2636: error: 'qos_rfilter_t' has no member named 'id' mod_qos.c:2637: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c:2637: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:2640: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c: In function 'qos_header_filter': mod_qos.c:2676: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:2680: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:2685: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:2686: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c: In function 'qos_setenvresheader': mod_qos.c:2897: error: 'pcre' undeclared (first use in this function) mod_qos.c:2897: error: 'pr' undeclared (first use in this function) mod_qos.c:2897: error: expected expression before ')' token mod_qos.c: In function 'qos_parp_hp_body': mod_qos.c:3033: error: 'qos_setenvifparpbody_t' has no member named 'preg' mod_qos.c:3035: error: 'qos_setenvifparpbody_t' has no member named 'name' mod_qos.c:3036: error: 'qos_setenvifparpbody_t' has no member named 'value' mod_qos.c:3048: error: 'qos_setenvifparpbody_t' has no member named 'pregx' mod_qos.c: In function 'qos_post_config': mod_qos.c:6939: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:6940: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:6947: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:6948: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c: In function 'qos_event_setenvresheadermatch_cmd': mod_qos.c:8216: error: 'pcre' undeclared (first use in this function) mod_qos.c:8216: error: 'pr' undeclared (first use in this function) mod_qos.c:8216: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:8216: error: 'PCRE_CASELESS' undeclared (first use in this function) mod_qos.c:8223: error: 'pcre_free' undeclared (first use in this function) mod_qos.c: In function 'qos_event_setenvifparpbody_cmd': mod_qos.c:8292: error: 'qos_setenvifparpbody_t' has no member named 'pregx' mod_qos.c:8296: error: 'qos_setenvifparpbody_t' has no member named 'preg' mod_qos.c:8296: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:8296: error: 'PCRE_CASELESS' undeclared (first use in this function) mod_qos.c:8297: error: 'qos_setenvifparpbody_t' has no member named 'preg' mod_qos.c:8303: error: 'qos_setenvifparpbody_t' has no member named 'preg' mod_qos.c:8303: error: 'pcre_free' undeclared (first use in this function) mod_qos.c:8304: error: 'qos_setenvifparpbody_t' has no member named 'pregx' mod_qos.c:8308: error: 'qos_setenvifparpbody_t' has no member named 'name' mod_qos.c:8309: error: 'qos_setenvifparpbody_t' has no member named 'name' mod_qos.c:8311: error: 'qos_setenvifparpbody_t' has no member named 'value' mod_qos.c:8315: error: 'qos_setenvifparpbody_t' has no member named 'value' mod_qos.c: In function 'qos_deny_cmd': mod_qos.c:8752: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:8757: error: 'qos_rfilter_t' has no member named 'id' mod_qos.c:8759: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:8761: error: 'qos_rfilter_t' has no member named 'action' mod_qos.c:8766: error: 'qos_rfilter_t' has no member named 'type' mod_qos.c:8767: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:8767: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:8768: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:8774: error: 'qos_rfilter_t' has no member named 'pr' mod_qos.c:8774: error: 'pcre_free' undeclared (first use in this function) mod_qos.c:8776: error: 'qos_rfilter_t' has no member named 'text' mod_qos.c: In function 'qos_deny_rql_cmd': mod_qos.c:8782: error: 'PCRE_CASELESS' undeclared (first use in this function) mod_qos.c: In function 'qos_deny_path_cmd': mod_qos.c:8786: error: 'PCRE_CASELESS' undeclared (first use in this function) mod_qos.c: In function 'qos_deny_query_cmd': mod_qos.c:8790: error: 'PCRE_CASELESS' undeclared (first use in this function) mod_qos.c: In function 'qos_headerfilter_rule_cmd': mod_qos.c:9008: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:9013: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9013: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:9015: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:9017: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:9022: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9029: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:9034: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9034: error: 'pcre_free' undeclared (first use in this function) mod_qos.c: In function 'qos_resheaderfilter_rule_cmd': mod_qos.c:9051: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:9053: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9053: error: 'PCRE_DOTALL' undeclared (first use in this function) mod_qos.c:9054: error: 'qos_fhlt_r_t' has no member named 'action' mod_qos.c:9055: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9062: error: 'qos_fhlt_r_t' has no member named 'size' mod_qos.c:9067: error: 'qos_fhlt_r_t' has no member named 'pcre' mod_qos.c:9067: error: 'pcre_free' undeclared (first use in this function) apxs:Error: Command failed with rc=65536 hope your reply thanks
Submitted by lanthruster (not registered) on Sat, 2010-02-06 19:48.

I've tested mod_qos-9.8 with Apache 2.2/FastCGI/PHP/event mpm with Linux box with approximately 30K hosts daily, undergoing DOS flooding attack & SYN/ACK attacks from botnet. ip_conntrack is disabled. 

Though mod_qos looked like a solution, after a while it showed some problems.

1. It counts IP connections wrong, I saw 160 connections from IP and rising while netstat -nt showed no connections from this IP at all. 

2. After it erroneously counts IP addresses there is no way to correct it but to restart apache

3. After about an hour of work it would crash apache threads with segmentations faults, including the apache servers, which is not surprising considering how it counts IP addresses. 

At least it my case mod_qos cannot be considered as a stable solution, though it is addressing the problem in the right way, the implementation suffers from the lack of testing. May be sometimes in the future it's going to be stable but so far alas.


Submitted by Amza Marian (not registered) on Thu, 2009-07-30 07:28.

 "Just a quick heads up: Linux has a limit on filehandles/sockets per process as well. 1024 by default I think. Programs like the Octopus attacker (incredibly fast, older, but less polished than SlowLoris - saw good use in the AnonNet attacks against MAQS and the IFPI) get around that by just spawning more processes."

It is very easy to increase the maximum number of allowed socket on linux. Also, you can use some iptables rules together QoS Module. 

You can read more about increasing the amount of memory associated with input and output socket buffers, tcp tuning and ulimit. (kernel parameters.)

Submitted by Anonymous (not registered) on Fri, 2009-07-17 15:37.
Just a quick heads up: Linux has a limit on filehandles/sockets per process as well. 1024 by default I think. Programs like the Octopus attacker (incredibly fast, older, but less polished than SlowLoris - saw good use in the AnonNet attacks against MAQS and the IFPI) get around that by just spawning more processes.

What gets me is that this type of attack, using connections up rather than bandwidth, is about 30 years old, and people are only just thinking of doing anything about it.
Submitted by nima0102 (registered user) on Sat, 2009-08-01 14:30.
Thanks for this article In last week my server is attacked with this method. My question is, Can not mod_security defend this attack ? thanks
Submitted by MarcusSpiegel (registered user) on Thu, 2009-08-06 10:27.
well, as far as I know mod_security, it analyses the content stream of an http request and compares them to configured signatures. But that kind of (slowloris-)attack is based on the number of connections rather than contents (or requests), so I think mod_security won't help mutch