How To Tell Apache To Not Log Certain Requests In Its Access Log

On this page

  1. 1 Using SetEnvIf
  2. 2 Links

Normally Apache logs all requests in its access log. In certain cases this can distort your page view statistics (if you use a tool like Webalizer or AWStats that creates statistics based on Apache's access log), for example if you get lots of visits from search engine spiders or from a certain IP address (e.g. your own), or if each of your pages includes another page (e.g. in an iframe) from your website (that would instantly double your page views which is obviously not correct). This short guide shows how you use Apache's SetEnvIf directive to prevent Apache from logging such requests.

1 Using SetEnvIf

The SetEnvIf directive can be used in the following contexts in your Apache configuration: in the global Apache configuration (if the directive should be valid for the whole server), in vhost configurations (if the directive should be valid only for that specific vhost), between <Directory ...></Directory> (if the directive should be valid only for a certain directory and its subdirectories), and in .htaccess files (AllowOverride FileInfo must be set).

With SetEnvIf, you can prevent requests from getting logged based on the following criteria (among others - see https://httpd.apache.org/docs/current/mod/mod_setenvif.html for more details):

  • Host
  • User-Agent
  • Referer
  • Accept-Language
  • Remote_Host: the hostname (if available) of the client making the request.
  • Remote_Addr: the IP address of the client making the request.
  • Server_Addr: the IP address of the server on which the request was received (only with versions later than 2.0.43).
  • Request_Method: the name of the method being used (GET, POST, etc.).
  • Request_Protocol: the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.).
  • Request_URI: the resource requested on the HTTP request line - generally the portion of the URL following the scheme and host portion without the query string.

The SetEnvIf directive has the following form:

SetEnvIf attribute regex env-variable

where attribute is one of the criteria I've just mentioned, and regex is a Perl compatible regular expression.

Now let's assume that Monit is requesting the file /monit/token once a minute to check if Apache is still running. Obviously, we don't want to log these requests because they are not from a real user. Therefore we use the following SetEnvIf directive:

SetEnvIf Request_URI "^/monit/token$" dontlog

^ means that the Request_URI must begin with /monit/token, $ means that it must also end with /monit/token (so only /monit/token matches this regular expression). If we used "^/monit/token", any URL beginning with /monit/token would match the regular expression, e.g. /monit/token/example.html; "/monit/token$" would match any URL ending in /monit/token, e.g. /example/monit/token.

Now we have an iframe in /iframe/iframe.html that we don't want to log either. This is what we'd use:

SetEnvIf Request_URI "^/iframe/iframe.html$" dontlog

Now we must tell Apache that it must not log all requests labeled with dontlog. Find the CustomLog directive in your Apache configuration, e.g.

CustomLog /var/log/apache2/access.log combined

or

CustomLog "|/usr/bin/cronolog --symlink=/var/log/httpd/access.log /var/log/httpd/access.log_%Y_%m_%d" combined

and add env=!dontlog to the line:

CustomLog /var/log/apache2/access.log combined env=!dontlog

or

CustomLog "|/usr/bin/cronolog --symlink=/var/log/httpd/access.log /var/log/httpd/access.log_%Y_%m_%d" combined env=!dontlog

Restart Apache afterwards. Now it won't log any request anymore that is labelled with dontlog.

Here are some further examples that I've found on these pages:

To prevent all requests made with a certain browser, e.g. Internet Explorer, from getting logged, you could use:

SetEnvIf User_Agent "(MSIE)" dontlog

To not log requests from any client whose hostname ends in bla.example.com, use:

SetEnvIf Remote_Host "bla.example.com$" dontlog

To not log requests from any client whose hostname begins with example, use:

SetEnvIf Remote_Host "^example" dontlog

To not log requests from a certain IP address, use something like:

SetEnvIf Remote_Addr "192\.168\.0\.154" dontlog

If you don't want requests of your robots.txt to get logged, use:

SetEnvIf Request_URI "^/robots\.txt$" dontlog

Apart from SetEnvIf, which is case-sensitive, you can use SetEnvIfNoCase which is case-insensitive.

For example, in order not to log certain search engine spiders, you could use:

SetEnvIFNoCase User-Agent "Slurp/cat" dontlog
SetEnvIFNoCase User-Agent "Ask Jeeves/Teoma" dontlog
SetEnvIFNoCase User-Agent "Googlebot" dontlog
SetEnvIFNoCase Remote_Host "fastsearch.net$" dontlog

Or to not log certain file extensions, use something like this:

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)|(eot)$" dontlog

To not log certain referrals (e.g. from your own domain), use something like:

SetEnvIfNoCase Referer "www\.mydomain\.com" dontlog
Share this page:

Suggested articles

15 Comment(s)

Add comment

Comments

By: Frank

Hi and thanks,

 this saved me a lot of unneccessary log file space ( 5 gb a day ).

 keep up the good work!

 

By: Anonymous

great, thanks a lot. very elegant & shows deep understanding of those piles of heaps of options!

By: Egbert O'Foo

> This document comes without warranty of any kind!

Ah, good ole BSD licensed, eh?

No worries ... I didn't need a warranty, just these instructions, and my years of previous experience with sysadmin work.

Worked for me with BSD!

By: Biren Desai

Nice post. Keep sharing such this kinds of post here and aware us from it. Keep it up.

By: Anonymous

I tried following your instructions but it's not working. Can you help? I need to know which conf file to place these items in, and if I"m using the right criteria as well. I posted a question here, thanks.

 http://serverfault.com/questions/626741/how-to-tell-apache-to-not-log-certain-requests-in-its-access-log

By: Anonymous

Well let me post exactly what I did:

I am trying to set a SetEnvIf variable in apache2 and exclude it in CustomLog in order to stop my logs from filling up. I want to match a regex and also specific IPs.

I wasn't sure where to put the SetEnvIf statement, so I put them in apache2.conf and also /sites-available/default.

Here is an example of what I want to filter in access.log:

    x.x.x.x - - [06/Sep/2014:15:02:35 -0500] "HEAD /dir/dir2/file1/file2.gif HTTP/1.1" 200 224 "-" "-"

    127.0.0.1 - - [06/Sep/2014:15:02:30 -0500] "GET /foo1/foo2.php? HTTP/1.0" 200 28956 "-" "Wget/1.12 (linux-gnu)"

So in apache2.conf and /sites-available/default, i put this:

    SetEnvIf Request_URI "HEAD /dir/" dontlog
    SetEnvIf Request_Addr "127\.0\.0\.1" dontlog
    SetEnvIf Request_Addr "::1" dontlog

And added env=!dontlog to the existing /default/CustomLog entry:

    CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog

I then executed apachectl graceful .

Nothing is changing.

By: hbokh

Use SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog instead.

By: Lukas

User_Agent does not exist ;-) 

Please correct it to User-Agent

 

By: Masikh

Thanks, it works like a charm. 

By: timptimpe

 Thank you for telling me so many things. I think that I have to study more with so much information. Read SetEnvIf directive.Regards

Takahiko Nakazawa

By: timptimpe

I am using awstats. I found out that it is based on "access_log".The problem is that I write in .htaccess as follows.RewriteEngine ONRewriteCond% {HTTPS} offRewriteRule ^ (. *) $ Https: //% {HTTP_HOST}% {REQUEST_URI} [R, L]For this reason code 302 appears in large quantities. Ask the visitor to access the home page for a certain period of time at https: //, then delete the following RewriteEngine On and use / var / log / httpd / access_log.Thank you very much.

 

By: Spork Schivago

Is it possible to use more than one attribute?   Maybe something like:[code]

SetEnvIf Remote_Addr "127\.0\.0\.1" Request_URI "^/whm-server-status\/?$" dontlog_serverstatus

[/code]So any request to whm-server-status is logged, unless the remote address is 127.0.0.1, then if it's 127.0.0.1 that's trying to access whm-server-status, it's not logged?Thanks!

By: ksair

QUERY_STRING example:CustomLog /var/log/apache2/access.log combined env=!dontlogRewriteCond %{QUERY_STRING} "^.*(username|password)=.*$"RewriteRule .* -  [E=dontlog:true]

By: Abhijeet

Awesome article. This is one of the best article with all possible solutions with examples. Keep up the good work.

By: Anonymous

Hi, thank you very much.

I had to change User_Agent (from your post) to User-Agent (from Apache docs)

Is this a typo?

Thank you again