How To Save Traffic With mod_deflate On Lighttpd 1.4 (Debian Etch)

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 08/08/2007

In this tutorial I will describe how to install and configure mod_deflate on a lighttpd 1.4 web server on Debian Etch. mod_deflate is included by default in lighttpd 1.5, but not in 1.4 where mod_compress is used instead. The advantage of mod_deflate over mod_compress is that it can compress static and dynamic files (such as PHP files), whereas mod_compress can compress static files only. The lighttpd version coming with Debian Etch is 1.4.13, so we have to patch it to support mod_deflate. mod_deflate allows lighttpd to compress files and deliver them to clients (e.g. browsers) that can handle compressed content which most modern browsers do. With mod_deflate, you can compress HTML, text or XML files to approx. 20 - 30% of their original sizes, thus saving you server traffic and making your modem users happier.

Compressing files causes a slightly higher load on the server, but in my experience this is compensated by the fact that the clients' connection times to your server decrease a lot. For example, a modem user that needed seven seconds to download an uncompressed HTML file might now only need two seconds for the same, but compressed file.

By using mod_deflate you don't have to be afraid that you exclude users with older browsers that cannot handle compressed content. The browser negotiates with the server before any file is transferred, and if the browser does not have the capability to handle compressed content, the server delivers the files uncompressed.

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

1 Preliminary Note

You can follow this tutorial no matter if lighttpd is already installed or not. We will build a new lighttpd 1.4.13 .deb package which will replace the existing lighttpd (if it is installed).

If lighttpd is already installed, you can check its version and included features like this:

lighttpd -V

The output should look like this (it should show version 1.4.13):

server1:~# lighttpd -V
lighttpd-1.4.13 (ssl) - a light and fast webserver
Build-Date: Jun  1 2007 18:19:33

Event Handlers:

        + select (generic)
        + poll (Unix)
        + rt-signals (Linux 2.4+)
        + epoll (Linux 2.6)
        - /dev/poll (Solaris)
        - kqueue (FreeBSD)

Network handler:

        + sendfile

Features:

        + IPv6 support
        + zlib support
        + bzip2 support
        + crypt support
        + SSL Support
        + PCRE support
        + mySQL support
        + LDAP support
        + memcached support
        - FAM support
        + LUA support
        + xml support
        + SQLite support
        + GDBM support

server1:~#

2 Building A New Lighttpd Package With mod_deflate Support

First we install everything we need to compile lighttpd from the sources:

apt-get install build-essential

Then we go to the /usr/src directory and download the Debian Etch lighttpd source package:

cd /usr/src
apt-get source lighttpd

Now run

ls -l

to see what lighttpd version you've got. It should be 1.4.13:

server1:/usr/src# ls -l
total 804
drwxr-xr-x 8 root root   4096 2007-08-08 19:03 lighttpd-1.4.13
-rw-r--r-- 1 root src   15173 2007-06-01 20:15 lighttpd_1.4.13-4etch1.diff.gz
-rw-r--r-- 1 root src    1098 2007-06-01 20:15 lighttpd_1.4.13-4etch1.dsc
-rw-r--r-- 1 root src  793309 2007-06-01 20:15 lighttpd_1.4.13.orig.tar.gz
server1:/usr/src#

Now we download the lighttpd-1.4.13.mod_deflate.jz.patch file from http://trac.lighttpd.net/trac/wiki/Mod_Deflate and patch the lighttpd 1.4.13 sources with it:

wget http://trac.lighttpd.net/trac/attachment/wiki/Mod_Deflate/lighttpd-1.4.13.mod_deflate.jz.patch?format=raw
mv lighttpd-1.4.13.mod_deflate.jz.patch?format=raw lighttpd-1.4.13.mod_deflate.jz.patch
cd lighttpd-1.4.13
patch -p1 < ../lighttpd-1.4.13.mod_deflate.jz.patch

If everything goes well, the output of the last command should look like this:

server1:/usr/src/lighttpd-1.4.13# patch -p1 < ../lighttpd-1.4.13.mod_deflate.jz.patch
patching file configure.in
patching file src/base.h
patching file src/chunk.c
patching file src/chunk.h
patching file src/connections.c
patching file src/http_chunk.c
patching file src/joblist.c
patching file src/Makefile.am
patching file src/Makefile.in
patching file src/mod_deflate.c
patching file src/plugin.c
patching file src/plugin.h
patching file src/request.c
patching file src/response.c
patching file src/server.c
Hunk #1 succeeded at 176 (offset 1 line).
Hunk #2 succeeded at 270 (offset 1 line).
Hunk #3 succeeded at 1043 (offset 1 line).
Hunk #4 succeeded at 1271 (offset 1 line).
Hunk #5 succeeded at 1324 (offset 1 line).
Hunk #6 succeeded at 1344 (offset 1 line).
server1:/usr/src/lighttpd-1.4.13#

After the patch has been applied without errors, we build our new lighttpd .deb package:

dpkg-buildpackage

dpkg-buildpackage will most likely complain about missing packages that it needs to build the new lighttpd .deb package:

server1:/usr/src/lighttpd-1.4.13# dpkg-buildpackage
dpkg-buildpackage: source package is lighttpd
dpkg-buildpackage: source version is 1.4.13-4etch1
dpkg-buildpackage: source changed by Steve Kemp <skx@debian.org>
dpkg-buildpackage: host architecture i386
dpkg-buildpackage: source version without epoch 1.4.13-4etch1
dpkg-checkbuilddeps: Unmet build dependencies: debhelper (>= 5.0.0) cdbs libssl-dev zlib1g-dev libbz2-dev libattr1-dev libpcre3-dev libmysqlclient15-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev libxml2-dev
dpkg-buildpackage: Build dependencies/conflicts unsatisfied; aborting.
dpkg-buildpackage: (Use -d flag to override.)
server1:/usr/src/lighttpd-1.4.13#

If you see an error like this, install the missing packages, e.g. like this:

apt-get install debhelper cdbs libssl-dev zlib1g-dev libbz2-dev libattr1-dev libpcre3-dev libmysqlclient15-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev libxml2-dev

Afterwards, run dpkg-buildpackage again:

dpkg-buildpackage

(On http://trac.lighttpd.net/trac/wiki/Mod_Deflate it says that bzip2 support must be enabled in lighttpd for mod_deflate to work. Fortunately, Debian's lighttpd package (and therefore also our new package) has bzip2 support by default, so there's nothing we must do to enable it.)

The dpkg-buildpackage command should now compile lighttpd again and create a new .deb package in the /usr/src directory. This can take some time, so please be patient. It's possible that you get some warnings about signatures at the end - you can ignore them.

Afterwards, we go to the /usr/src directory and see what we got:

cd ..
ls -l

The output could look like this:

server1:/usr/src# ls -l
total 1628
drwxr-xr-x 8 root root   4096 2007-08-08 19:09 lighttpd-1.4.13
-rw-r--r-- 1 root src   29377 2007-08-08 19:07 lighttpd_1.4.13-4etch1.diff.gz
-rw-r--r-- 1 root src     861 2007-08-08 19:07 lighttpd_1.4.13-4etch1.dsc
-rw-r--r-- 1 root src    2000 2007-08-08 19:12 lighttpd_1.4.13-4etch1_i386.changes
-rw-r--r-- 1 root src  287998 2007-08-08 19:12 lighttpd_1.4.13-4etch1_i386.deb
-rw-r--r-- 1 root src   69033 2006-12-13 17:22 lighttpd-1.4.13.mod_deflate.jz.patch
-rw-r--r-- 1 root src  793309 2007-06-01 20:15 lighttpd_1.4.13.orig.tar.gz
-rw-r--r-- 1 root src   99606 2007-08-08 19:11 lighttpd-doc_1.4.13-4etch1_all.deb
-rw-r--r-- 1 root src   63136 2007-08-08 19:12 lighttpd-mod-cml_1.4.13-4etch1_i386.deb
-rw-r--r-- 1 root src   62948 2007-08-08 19:12 lighttpd-mod-magnet_1.4.13-4etch1_i386.deb
-rw-r--r-- 1 root src   58546 2007-08-08 19:12 lighttpd-mod-mysql-vhost_1.4.13-4etch1_i386.deb
-rw-r--r-- 1 root src   60212 2007-08-08 19:12 lighttpd-mod-trigger-b4-dl_1.4.13-4etch1_i386.deb
-rw-r--r-- 1 root src   70268 2007-08-08 19:12 lighttpd-mod-webdav_1.4.13-4etch1_i386.deb
server1:/usr/src#

As you see, a new lighttpd_1.4.13-4etch1_i386.deb package has been created which we can now install like this (this will replace your existing lighttpd):

dpkg -i lighttpd_1.4.13-4etch1_i386.deb

(If you need these modules, you can install lighttpd-mod-cml_1.4.13-4etch1_i386.deb, lighttpd-mod-magnet_1.4.13-4etch1_i386.deb, lighttpd-mod-mysql-vhost_1.4.13-4etch1_i386.deb, lighttpd-mod-trigger-b4-dl_1.4.13-4etch1_i386.deb and lighttpd-mod-webdav_1.4.13-4etch1_i386.deb the same way.)

Now we must copy the file mod_deflate.so to the lighttpd modules directory:

cp /usr/src/lighttpd-1.4.13/debian/tmp/usr/lib/lighttpd/mod_deflate.so /usr/lib/lighttpd

If you like, you can now run

lighttpd -V

to check the build date to make sure that the new lighttpd is installed:

server1:/usr/src# lighttpd -V
lighttpd-1.4.13 (ssl) - a light and fast webserver
Build-Date: Aug  8 2007 19:10:28

Event Handlers:

        + select (generic)
        + poll (Unix)
        + rt-signals (Linux 2.4+)
        + epoll (Linux 2.6)
        - /dev/poll (Solaris)
        - kqueue (FreeBSD)

Network handler:

        + sendfile

Features:

        + IPv6 support
        + zlib support
        + bzip2 support
        + crypt support
        + SSL Support
        + PCRE support
        + mySQL support
        + LDAP support
        + memcached support
        - FAM support
        + LUA support
        + xml support
        + SQLite support
        + GDBM support

server1:/usr/src#

As you see, bzip2 support is built in, which is a prerequisite for mod_deflate. (Don't worry if you don't see mod_deflate in the output - that's normal.)

3 Configuring Lighttpd

The lighttpd configuration is in /etc/lighttpd/lighttpd.conf on Debian Etch. We open that file and add "mod_deflate", to the server.modules stanza (if mod_rewrite is enabled, it's important that mod_deflate gets listed behind mod_rewrite!):

vi /etc/lighttpd/lighttpd.conf

[...]
server.modules              = (
            "mod_access",
            "mod_alias",
            "mod_accesslog",
            "mod_fastcgi",
#           "mod_rewrite",
            "mod_deflate",
#           "mod_redirect",
#           "mod_status",
#           "mod_evhost",
#           "mod_compress",
#           "mod_usertrack",
#           "mod_rrdtool",
#           "mod_webdav",
#           "mod_expire",
#           "mod_flv_streaming",
#           "mod_evasive"
 )
[...]

Then, in the same file, we add the following mod_deflate configuration (you can put it at the end of /etc/lighttpd/lighttpd.conf):

[...]
deflate.enabled = "enable"
deflate.compression-level = 9
deflate.mem-level = 9
deflate.window-size = 15
# deflate.bzip2 only in patch for 1.4.x
deflate.bzip2 = "enable"
# deflate.allowed_encodings only in 1.5.x
#deflate.allowed_encodings = ( "bzip2", "gzip", "deflate" )
deflate.min-compress-size = 200
#deflate.sync-flush = "enable"
deflate.output-buffer-size = 4096
deflate.work-block-size = 512
deflate.mimetypes = ("text/html", "text/plain", "text/css", "text/javascript", "text/xml")
#deflate.debug = "enable"

Afterwards, we restart lighttpd:

/etc/init.d/lighttpd restart

It should restart without any errors.

Now try to access a few .html/.php/.txt/... files from your lighttpd server in a browser. It should work without problems. If you get blank pages, try to decrease the value of deflate.output-buffer-size in /etc/lighttpd/lighttpd.conf to 2048 or 1024 (don't forget to restart lighttpd after any changes to /etc/lighttpd/lighttpd.conf).

To verify that mod_deflate is really compressing your files, you can uncomment the line deflate.debug = "enable" in /etc/lighttpd/lighttpd.conf (restart lighttpd afterwards). Now take a look at /var/log/lighttpd/error.log, e.g. like this:

tail -f /var/log/lighttpd/error.log

(press CTRL-c if you want to leave the error log)

and try to access a few files from your lighttpd server. If a file gets compressed, you should find something similar to this in lighttpd's error log:

[...]
2007-08-08 18:40:33: (mod_deflate.c.919) in: 53875 out: 8446
2007-08-08 18:40:33: (mod_deflate.c.1020) finished uri: /info.php , query:
[...]

In this example it means that the file info.php which has a file size of 53875 bytes has been compressed to just 8446 bytes. That's less than 20% of the original size!

4 Links

• Lighttpd: http://www.lighttpd.net
• mod_deflate: http://trac.lighttpd.net/trac/wiki/Mod_Deflate
• Debian: http://www.debian.org