apache log issues

[abubin]

My website is only serving around 10 websites.

One of the website is very active thus generating lots of logs.

I just found out about this only when I got alerted for diskspace running out email.

I tried searching in ispconfig and there are no option to disable logging for certain website.

I am using ispconfig 3.0.1.6.

Can I know where I can disable http logging for specified websites?? I think this is a pretty important feature and I assume it is already implemented? Maybe I didn't find properly?

What about compressions on the logs? Is there any?

What about log rotate?

As of now, these 3 features seems like not available. Without these, the logs will build up like crazy, even for a normal website, the logs generated daily is around 1GB already. With compression, this could reduce to 100mb.

Please help....thanks.

[abubin]

Just manage to view back one of the thread about compression of log.

http://www.howtoforge.com/forums/sho...light=compress

I looked at my cron_daily.php file and there are lines already added for compression of logs and also removal of logs more than 30 days. But seems like this is not running properly. Don't know why...anyone can take a look into this?

[abubin]

When I rung the cron_daily.php manually,

I get error: Warning: Truncating oversized referrer field

This could be the thing causing the failed access_log files processing?

[till]

This could be the thing causing the failed access_log files processing?

No. The truncate error is normal and happens on log lines that are too long.

Please update your server to ispconfig 3.0.2.2 as it has an improved log handling.

[abubin]

i have upgrade to 3.0.2.2. There are no additional function for handling logs per website. I am assuming the improved log handling you meant was the compression and 30 days removal script is working now?

If not, please do point me to the right direction.

Thanks.

[abubin]

i have looked at another server that is using 3.0.2.1. The logs compression is there for most of the domains except for 2. One is as11 and as11.domain.com. FYI, as11 is the server's hostname. So it's like the compression doesn't work on local hostname.

Upgrade to 3.0.2.2?

[abubin]

I left the upgraded 3.0.2.2 server overnight. Checked the access logs in /var/log/ispconfig/httpd and no compressed access log file. The log file of the previous day is still uncompressed.

Please help....

[till]

Please post the output of:

ls -la /var/log/ispconfig/httpd/yourdomain.com

replace yourdomain.com with the domain name of a affected website. The websites that are affected had been created in ispconfig or have you created them manually?

[abubin]

server 1 (3.0.2.2):
- all access log also does not compress nor delete files older than 30 days:


ls -la /var/log/ispconfig/httpd/127.0.0.1

-rw-r--r--  1 root root   59973494 2010-07-01 23:59 20100701-access.log.gz
-rw-r--r--  1 root root   57519689 2010-07-02 23:59 20100702-access.log.gz
-rw-r--r--  1 root root   56580180 2010-07-03 23:59 20100703-access.log.gz
-rw-r--r--  1 root root   54573798 2010-07-04 23:59 20100704-access.log.gz
-rw-r--r--  1 root root   58566076 2010-07-05 23:59 20100705-access.log.gz
-rw-r--r--  1 root root   61830847 2010-07-06 23:59 20100706-access.log.gz
-rw-r--r--  1 root root   58912995 2010-07-07 23:59 20100707-access.log.gz
-rw-r--r--  1 root root   57487132 2010-07-08 23:59 20100708-access.log.gz
-rw-r--r--  1 root root   52348721 2010-07-09 23:59 20100709-access.log.gz
-rw-r--r--  1 root root   52555526 2010-07-10 23:59 20100710-access.log.gz
-rw-r--r--  1 root root   53555419 2010-07-11 23:59 20100711-access.log.gz
-rw-r--r--  1 root root   38464667 2010-07-12 23:59 20100712-access.log.gz
-rw-r--r--  1 root root   44748612 2010-07-13 23:59 20100713-access.log.gz
-rw-r--r--  1 root root   49687442 2010-07-14 23:59 20100714-access.log.gz
-rw-r--r--  1 root root   87718804 2010-07-15 23:59 20100715-access.log.gz
-rw-r--r--  1 root root   89333655 2010-07-16 23:59 20100716-access.log.gz
-rw-r--r--  1 root root   87059890 2010-07-17 23:59 20100717-access.log.gz
-rw-r--r--  1 root root   85310862 2010-07-18 23:59 20100718-access.log.gz
-rw-r--r--  1 root root  127998030 2010-07-19 23:59 20100719-access.log.gz
-rw-r--r--  1 root root 1830110297 2010-07-20 23:59 20100720-access.log
-rw-r--r--  1 root root 2011749233 2010-07-21 23:59 20100721-access.log
-rw-r--r--  1 root root 1764172052 2010-07-22 23:59 20100722-access.log
-rw-r--r--  1 root root 1754949565 2010-07-23 23:59 20100723-access.log
-rw-r--r--  1 root root 1677808377 2010-07-24 23:59 20100724-access.log
-rw-r--r--  1 root root 1670813799 2010-07-25 23:59 20100725-access.log
-rw-r--r--  1 root root 1729431133 2010-07-26 23:59 20100726-access.log
-rw-r--r--  1 root root 1687999078 2010-07-27 23:59 20100727-access.log
-rw-r--r--  1 root root 1700423428 2010-07-28 23:59 20100728-access.log
-rw-r--r--  1 root root 1429171939 2010-07-29 17:31 20100729-access.log
lrwxrwxrwx  1 root root         19 2010-07-29 12:48 access.log -> 20100729-access.log

ls -la /var/log/ispconfig/httpd/vn.domain.com/

-rw-r--r--  1 root root     206761957 2010-07-16 23:59 20100716-access.log.gz
-rw-r--r--  1 root root     174249432 2010-07-17 23:59 20100717-access.log.gz
-rw-r--r--  1 root root     149160599 2010-07-18 23:59 20100718-access.log.gz
-rw-r--r--  1 root root     179652719 2010-07-19 23:59 20100719-access.log.gz
-rw-r--r--  1 root root    1323127637 2010-07-20 23:59 20100720-access.log
-rw-r--r--  1 root root    1451718336 2010-07-21 23:59 20100721-access.log
-rw-r--r--  1 root root    1439908507 2010-07-22 23:59 20100722-access.log
-rw-r--r--  1 root root    1201661855 2010-07-23 23:59 20100723-access.log
-rw-r--r--  1 root root     898080288 2010-07-24 23:59 20100724-access.log
-rw-r--r--  1 root root     874027550 2010-07-25 23:59 20100725-access.log
-rw-r--r--  1 root root    1142218307 2010-07-26 23:59 20100726-access.log
-rw-r--r--  1 root root    1137946356 2010-07-27 23:59 20100727-access.log
-rw-r--r--  1 root root    1218598540 2010-07-28 23:59 20100728-access.log
-rw-r--r--  1 root root     799965574 2010-07-29 17:35 20100729-access.log
lrwxrwxrwx  1 root root            19 2010-07-29 12:48 access.log -> 20100729-access.log
-rw-r--r--  1 web2 client1    1183785 2010-07-29 17:31 error.log
-rw-r--r--  1 root root         25707 2010-07-29 00:30 webalizer.conf

This file I compressed them manually. And already removed files older than July 2010.

The domain of 127.0.0.1 is the default domain created manually (from ubuntu's default site) and vn.domain.com is created using ispconfig


server 2 (3.0.2.1):

ls -la /var/log/ispconfig/httpd/as69

drwxr-xr-x  2 root root  4096 2010-07-29 00:27 .
drwxr-xr-x 27 root root  4096 2010-07-23 16:51 ..
-rw-r--r--  1 root root 27627 2010-05-26 23:59 20100526-access.log
-rw-r--r--  1 root root 39069 2010-05-27 23:55 20100527-access.log
-rw-r--r--  1 root root 36137 2010-05-28 23:55 20100528-access.log
-rw-r--r--  1 root root 36178 2010-05-29 23:57 20100529-access.log
-rw-r--r--  1 root root 35886 2010-05-30 23:59 20100530-access.log
-rw-r--r--  1 root root 57538 2010-05-31 23:58 20100531-access.log
-rw-r--r--  1 root root 26358 2010-06-01 23:56 20100601-access.log
-rw-r--r--  1 root root 12807 2010-06-02 23:34 20100602-access.log
-rw-r--r--  1 root root 13814 2010-06-03 23:42 20100603-access.log
-rw-r--r--  1 root root 21864 2010-06-04 23:50 20100604-access.log
-rw-r--r--  1 root root  9855 2010-06-05 23:49 20100605-access.log
-rw-r--r--  1 root root 11110 2010-06-06 23:43 20100606-access.log
-rw-r--r--  1 root root  8399 2010-06-07 23:55 20100607-access.log
-rw-r--r--  1 root root  6039 2010-06-08 20:25 20100608-access.log
-rw-r--r--  1 root root   518 2010-06-09 07:38 20100609-access.log
-rw-r--r--  1 root root 22656 2010-06-10 17:43 20100610-access.log
-rw-r--r--  1 root root   495 2010-06-11 18:25 20100611-access.log
-rw-r--r--  1 root root  2202 2010-06-13 15:12 20100613-access.log
-rw-r--r--  1 root root  1425 2010-06-14 16:26 20100614-access.log
-rw-r--r--  1 root root 13509 2010-06-15 12:52 20100615-access.log
-rw-r--r--  1 root root  1148 2010-06-17 11:27 20100617-access.log
-rw-r--r--  1 root root   173 2010-06-18 21:55 20100618-access.log
-rw-r--r--  1 root root  1842 2010-06-20 06:33 20100620-access.log
-rw-r--r--  1 root root  1017 2010-06-21 22:19 20100621-access.log
-rw-r--r--  1 root root   258 2010-06-22 22:18 20100622-access.log
-rw-r--r--  1 root root 11764 2010-06-23 15:13 20100623-access.log
-rw-r--r--  1 root root   857 2010-06-24 23:04 20100624-access.log
-rw-r--r--  1 root root 15296 2010-06-25 21:48 20100625-access.log
-rw-r--r--  1 root root   512 2010-06-26 18:04 20100626-access.log
-rw-r--r--  1 root root  4870 2010-06-27 22:18 20100627-access.log
-rw-r--r--  1 root root  1762 2010-06-28 23:01 20100628-access.log
-rw-r--r--  1 root root   658 2010-06-29 19:58 20100629-access.log
-rw-r--r--  1 root root 13850 2010-06-30 15:26 20100630-access.log
-rw-r--r--  1 root root   335 2010-07-01 14:43 20100701-access.log
-rw-r--r--  1 root root   696 2010-07-02 23:39 20100702-access.log
-rw-r--r--  1 root root   206 2010-07-03 15:34 20100703-access.log
-rw-r--r--  1 root root  2575 2010-07-04 18:26 20100704-access.log
-rw-r--r--  1 root root 15167 2010-07-05 15:17 20100705-access.log
-rw-r--r--  1 root root   256 2010-07-06 14:00 20100706-access.log
-rw-r--r--  1 root root   528 2010-07-07 23:08 20100707-access.log
-rw-r--r--  1 root root   297 2010-07-08 23:32 20100708-access.log
-rw-r--r--  1 root root  1528 2010-07-09 23:32 20100709-access.log
-rw-r--r--  1 root root   239 2010-07-10 08:13 20100710-access.log
-rw-r--r--  1 root root  4666 2010-07-11 20:42 20100711-access.log
-rw-r--r--  1 root root   260 2010-07-12 16:47 20100712-access.log
-rw-r--r--  1 root root   333 2010-07-13 15:52 20100713-access.log
-rw-r--r--  1 root root   111 2010-07-14 23:23 20100714-access.log
-rw-r--r--  1 root root  1257 2010-07-15 22:57 20100715-access.log
-rw-r--r--  1 root root   444 2010-07-16 12:11 20100716-access.log
-rw-r--r--  1 root root   763 2010-07-17 16:33 20100717-access.log
-rw-r--r--  1 root root  2586 2010-07-18 23:34 20100718-access.log
-rw-r--r--  1 root root  2298 2010-07-19 19:39 20100719-access.log
-rw-r--r--  1 root root   128 2010-07-20 12:26 20100720-access.log
-rw-r--r--  1 root root 20287 2010-07-21 18:35 20100721-access.log
-rw-r--r--  1 root root  1942 2010-07-22 20:38 20100722-access.log
-rw-r--r--  1 root root 23792 2010-07-23 23:11 20100723-access.log
-rw-r--r--  1 root root  1925 2010-07-24 23:44 20100724-access.log
-rw-r--r--  1 root root  7010 2010-07-25 21:50 20100725-access.log
-rw-r--r--  1 root root 31775 2010-07-26 23:12 20100726-access.log
-rw-r--r--  1 root root  3219 2010-07-27 23:08 20100727-access.log
-rw-r--r--  1 root root  3348 2010-07-28 23:07 20100728-access.log
-rw-r--r--  1 root root 11323 2010-07-29 17:41 20100729-access.log
lrwxrwxrwx  1 root root    19 2010-07-29 00:27 access.log -> 20100729-access.log

ls -la /var/log/ispconfig/httpd/as69.domain.com

drwxr-xr-x  2 root root    4096 2010-05-26 11:31 .
drwxr-xr-x 27 root root    4096 2010-07-23 16:51 ..
-rw-r--r--  1 root root  257362 2010-03-22 18:57 20100322-access.log
-rw-r--r--  1 root root   24944 2010-03-23 18:00 20100323-access.log
-rw-r--r--  1 root root   41341 2010-03-24 18:19 20100324-access.log
-rw-r--r--  1 root root  187878 2010-03-25 23:53 20100325-access.log
-rw-r--r--  1 root root   46191 2010-03-26 18:24 20100326-access.log
-rw-r--r--  1 root root     256 2010-03-27 10:50 20100327-access.log
-rw-r--r--  1 root root    1386 2010-03-28 06:28 20100328-access.log
-rw-r--r--  1 root root    1592 2010-03-29 17:00 20100329-access.log
-rw-r--r--  1 root root   78299 2010-03-30 18:28 20100330-access.log
-rw-r--r--  1 root root    2930 2010-03-31 20:54 20100331-access.log
-rw-r--r--  1 root root     570 2010-04-01 16:10 20100401-access.log
-rw-r--r--  1 root root   48175 2010-04-02 18:32 20100402-access.log
-rw-r--r--  1 root root     130 2010-04-03 08:46 20100403-access.log
-rw-r--r--  1 root root    1893 2010-04-04 17:22 20100404-access.log
-rw-r--r--  1 root root   11802 2010-04-05 18:45 20100405-access.log
-rw-r--r--  1 root root     110 2010-04-07 21:34 20100407-access.log
-rw-r--r--  1 root root   14720 2010-04-08 21:38 20100408-access.log
-rw-r--r--  1 root root     129 2010-04-09 17:35 20100409-access.log
-rw-r--r--  1 root root     129 2010-04-10 21:44 20100410-access.log
-rw-r--r--  1 root root    1665 2010-04-11 20:05 20100411-access.log
-rw-r--r--  1 root root     130 2010-04-12 17:22 20100412-access.log
-rw-r--r--  1 root root 1436450 2010-04-13 23:59 20100413-access.log
-rw-r--r--  1 root root 2413651 2010-04-14 23:59 20100414-access.log
-rw-r--r--  1 root root 2378880 2010-04-15 23:59 20100415-access.log
-rw-r--r--  1 root root 2399040 2010-04-16 23:59 20100416-access.log
-rw-r--r--  1 root root 2062750 2010-04-17 23:59 20100417-access.log
-rw-r--r--  1 root root 2117572 2010-04-18 23:59 20100418-access.log
-rw-r--r--  1 root root 2481353 2010-04-19 23:59 20100419-access.log
-rw-r--r--  1 root root 2487938 2010-04-20 23:59 20100420-access.log
-rw-r--r--  1 root root 2542348 2010-04-21 23:59 20100421-access.log
-rw-r--r--  1 root root 2685128 2010-04-22 23:59 20100422-access.log
-rw-r--r--  1 root root 2393308 2010-04-23 23:59 20100423-access.log
-rw-r--r--  1 root root 2131292 2010-04-24 23:59 20100424-access.log
-rw-r--r--  1 root root 2097671 2010-04-25 23:59 20100425-access.log
-rw-r--r--  1 root root 2559121 2010-04-26 23:59 20100426-access.log
-rw-r--r--  1 root root 2536506 2010-04-27 23:59 20100427-access.log
-rw-r--r--  1 root root 2425253 2010-04-28 23:59 20100428-access.log
-rw-r--r--  1 root root 2429532 2010-04-29 23:59 20100429-access.log
-rw-r--r--  1 root root 2416206 2010-04-30 23:59 20100430-access.log
-rw-r--r--  1 root root 1832299 2010-05-01 23:59 20100501-access.log
-rw-r--r--  1 root root 1916858 2010-05-02 23:59 20100502-access.log
-rw-r--r--  1 root root 2274764 2010-05-03 23:59 20100503-access.log
-rw-r--r--  1 root root 2492260 2010-05-04 23:59 20100504-access.log
-rw-r--r--  1 root root 2217581 2010-05-05 23:59 20100505-access.log
-rw-r--r--  1 root root 2439680 2010-05-06 23:59 20100506-access.log
-rw-r--r--  1 root root 2346986 2010-05-07 23:59 20100507-access.log
-rw-r--r--  1 root root 2001514 2010-05-08 23:59 20100508-access.log
-rw-r--r--  1 root root 1994963 2010-05-09 23:59 20100509-access.log
-rw-r--r--  1 root root 2495807 2010-05-10 23:59 20100510-access.log
-rw-r--r--  1 root root 2385358 2010-05-11 23:59 20100511-access.log
-rw-r--r--  1 root root 2397078 2010-05-12 23:59 20100512-access.log
-rw-r--r--  1 root root 2428856 2010-05-13 23:59 20100513-access.log
-rw-r--r--  1 root root 2251266 2010-05-14 23:59 20100514-access.log
-rw-r--r--  1 root root 1963221 2010-05-15 23:59 20100515-access.log
-rw-r--r--  1 root root 2418050 2010-05-16 23:59 20100516-access.log
-rw-r--r--  1 root root 3335868 2010-05-17 23:59 20100517-access.log
-rw-r--r--  1 root root 3493862 2010-05-18 23:59 20100518-access.log
-rw-r--r--  1 root root 2840056 2010-05-19 23:59 20100519-access.log
-rw-r--r--  1 root root 3037030 2010-05-20 23:59 20100520-access.log
-rw-r--r--  1 root root 2642671 2010-05-21 23:59 20100521-access.log
-rw-r--r--  1 root root 2373177 2010-05-22 23:59 20100522-access.log
-rw-r--r--  1 root root 2192809 2010-05-23 23:59 20100523-access.log
-rw-r--r--  1 root root 2713228 2010-05-24 23:59 20100524-access.log
-rw-r--r--  1 root root 2667320 2010-05-25 23:59 20100525-access.log
-rw-r--r--  1 root root 1520034 2010-05-26 15:52 20100526-access.log
lrwxrwxrwx  1 root root      19 2010-05-26 11:31 access.log -> 20100526-access.log

Both as69 and as69.domain.com also is created manually. Again this is under "000-default" under apache configuration.

Other sites on server2 is no problem. The access log files are compressed and old file deleted.

Thanks.

[till]

The log compression are only for sites created in ispconfig. Th examples that you posted above are not from sites that had been created in ispconfig, so the logs dont get compressed.

ISPConfig can only compress logs for sites that it knows off. If you want to compress logs for non ispconfig websites, you might have to write your own scripts for that.