Apache log rotation not configured in ISPConfig 3?

[gkovacs]

I've just realized that my webserver logs were piling up in the tens of gigabytes. I have checked the logrotate.d config for apache, but it is not configured for the log files in /var/log/ispconfig/httpd/, it only handles the default apache2 log directory, which is empty:

root@ispconfig:/etc/logrotate.d# cat apache2
/var/log/apache2/*.log {
weekly
missingok
rotate 52
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
/etc/init.d/apache2 reload > /dev/null
fi
endscript
}

Furthermore, there is no trace of logrotate in the cron folders or crontab.

Is ISPConfig able to handle log rotation, or should I configure this myself, separately?

Logrotate.d seems to pose a problem, as I would have to add every site to it manually, as ISPConfig keeps the log files in separate folders.

[till]

ISPConfig handles the logfile rotation, so do not use logrotate as it will conflict with ISPConfig.

The logfiles are cromresssed after one day and removed after 30 days. See the script /usr/local/ispconfig/server/cron_daily.php for details.

[gkovacs]

Till, how is it possible that I had just over 5 months of logs, none of it deleted or compressed? We are running ISPConfig 3.0.1 currently.

The script you mentioned is not in crontab nor is it visible in /etc/cron.daily. Should I copy/link it over?

[till]

There was a bug ion older release regarding log rotation. You should update your system regularily. The cuurent version is 3.0.1.3.

[gkovacs]

I have updated my ISPConfig to the latest version, but there is still no log rotation script (or anything connected to ISPConfig whatsoever) in the cron.daily folder or in /etc/crontab.

Here is the install script output:

Operating System: Debian Lenny/Sid or compatible

This application will update ISPConfig 3 on your server.
MySQL root password []: xxxxxxxxxxxxxxxx

Reconfigure Services? (yes,no) [yes]: no

Updating ISPConfig
ISPConfig Port [8080]:

chown: invalid user: `getmail'
Reconfigure Crontab? (yes,no) [yes]: yes

Updating Crontab
Update finished.

Do I need to reconfigure all services to enable the log rotation?

[till]

I have updated my ISPConfig to the latest version, but there is still no log rotation script (or anything connected to ISPConfig whatsoever) in the cron.daily folder or in /etc/crontab.

Please red my post above. I did not talk about the cron daily folder and I posted you the exact location of the script which does the log rotation and all other nightly tasks.

[gkovacs]

Please red my post above. I did not talk about the cron daily folder and I posted you the exact location of the script which does the log rotation and all other nightly tasks.

Ok, but how does it run if not from cron? How can I set the time it runs, so it would not interfere with my other nightly tasks on the server?

By the way nothing happened last night. Log files are still there, none is compressed or deleted.

[till]

As root user run:

crontab -l

to get a list of the ispconfig cronjobs.

By the way nothing happened last night. Log files are still there, none is compressed or deleted.

The script compresses just the file which is one day old and deletes the file which is one month old. You will have to delete other files manually that are left on your server.

[gkovacs]

The script compresses just the file which is one day old and deletes the file which is one month old. You will have to delete other files manually that are left on your server.

Well, I let it run for 2 days but nothing happened. The cron job is there, the script runs according to syslog, but it just doesn't compress the day old file and doesn't delete the month old one. Checked at several different sites, with log folders both empty and full with hundreds of files.

/var/log/ispconfig/cron.log shows nothing relevant.

What may be wrong? Any ideas how to nail the problem down?

[Alex1]

why error.log don't rotate like access.log?

[till]

As far as I know there is no rotation of the error log yet. You can post it as feature request in the bugtracker.

[gkovacs]

Till, any idea why on my machine the log rotation would not work?
Maybe a thing or two to check?

[till]

Till, any idea why on my machine the log rotation would not work?
Maybe a thing or two to check?

unpack and copy the attached file to /usr/local/ispconfig/server/ and check if this one works for you. Its the file from the current svn.

[gkovacs]

unpack and copy the attached file to /usr/local/ispconfig/server/ and check if this one works for you. Its the file from the current svn.

I have installed the file you provided, but still nothing: logs don't get compressed or deleted. Any idea what else to check?

[Keeper]

Looks like I'm bumping an old topic, but forgive me. :)

I was curious about log rotating too after I saw error logs aren't rotated, but looks like future versions will have that too.

But I am concerned that cron jobs are in fact php files and thus depend on lenient PHP security settings. On my server for example, I have disabled exec function among others which is used in cron_daily.php.

Now of course following functions do not work.

exec("$webalizer -c $webalizer_conf -n $domain -s $domain -r $domain -q -T -o $statsdir $logfile");

exec("gzip -c $logfile > $logfile.gz");


Isn't this a rather severe oversight in ISPConfig 3?

[till]

Isn't this a rather severe oversight in ISPConfig 3?

Not at all. We are talking here about shell scripts and not web scripts. Nothing in ispconfig backend will work if you disable exec for shell scripts of course. You can disable exec in the php.ini for mod_php but not for cli. By the way, disabling exec for cli does not make any sense as all as cli means that the script is executed by someone on the shell anyway.

[Undo]

ISPConfig handles the logfile rotation, so do not use logrotate as it will conflict with ISPConfig.

The logfiles are cromresssed after one day and removed after 30 days. See the script /usr/local/ispconfig/server/cron_daily.php for details.

Hello Till,

I watched into the given file but it seems that it is not exactly it that handles the log rotation.

In this file we have the compression of the file once it's already been renamed by another something and the generation of the statistics etc. etc. ...

So, if you could tell me what is the script / program which is, every day, re-generating the symlink from access.log -> yesterday.log to access.log -> today.log, it would be really neat, because I have a misfunctionning of the log rotation for one website and I would like to see what's hapening and why it is not happening for a particular website.


Thanks in advance.

[Undo]

Hello guys,

I've been thinking about all this and finally came with this conclusion : Get rid of Vlogger for Nginx.

Well, after all, nginx is significantly faster than Apache and it is known to handle correctly disk access.

But remains the problem of the filenames for Webalizer and AWStats, so I made that :

Create the directory /var/log/ispconfig/nginx and make nginx log into domaine.tld/access.log as Apache do for the httpd/ folder :

(configuration Nginx for a given server domain.tld) :


access_log /var/log/ispconfig/nginx/domain.tld/access.log;


In /etc/logrotate.d/nginx, add at the end of file :

/var/log/ispconfig/nginx/*/access.log {
daily
missingok
nocompress
rotate 1
notifempty
sharedscripts
postrotate
[ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
/usr/bin/nohup /usr/local/ispconfig/server/logrotate_nginx.sh >> /var/log/ispconfig/cron.log
endscript
}

Now create the two following files :
/usr/local/ispconfig/server/logrotate_nginx.sh

#!/bin/sh

PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin

/usr/bin/php -q /usr/local/ispconfig/server/logrotate_nginx.php


/usr/local/ispconfig/server/logrotate_nginx.php
<?php

require('lib/config.inc.php');
require('lib/app.inc.php');

set_time_limit(0);
ini_set('error_reporting',E_ALL & ~E_NOTICE);

$yesterday=date('Ymd', time()-86399);

chdir($conf['ispconfig_log_dir']);

$dirs = glob('nginx/*/');
foreach($dirs as $dir) {
// Put the content of the access.log of today in the file supposed to be used for yesterday :
file_put_contents($dir.$yesterday.'-access.log', file_get_contents($dir.'access.log.1'));
// Empty the current access.log to have only statistics about today at next daily logrotation :
unlink($dir.'access.log.1');
}

?>


Now remains two choices :

Either you modify server/cron_daily.php in order to go through nginx directories too.
Either you modify my script and replace "$dir.$yesterday.'.log'" by something like "httpd/".str_replace('nginx/', '', $dir).$yesterday.'-access.log' and this will copy the file into the httpd directory which is parsed by the cron_daily.php