fc6/ISPConfig/cronolog problem

[zetnsh]

[Apologies - just realised I posted this in the wrong forum!!!]

Hi there,

Just built two new servers with ISPConfig, side by side, one works, one doesn't!

The build is Fedora Core 6 and the latest ISPConfig tarball from SourceForge.

I've followed the instructions here (http://www.howtoforge.org/perfect_server_fedora7) to set it up (with ref. to a previous post of mine detailing the differences for fc6 - see here (http://www.howtoforge.org/forums/showthread.php?t=14868)), and also set up suphp as per here (http://www.howtoforge.com/suphp_fedora7_centos5_with_ispconfig):

However, whilst one of the boxes works, the other one fails to start apache, with this error:

unable to start piped log program '/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d': Permission denied
Unable to open logs

I can't see the difference between the servers, and yes, selinux is definately disabled on both! I noticed in previous posts that this helped...

Anyone got any ideas?

Thanks!

Neil

[falko]

What's the output of ls -la /root/ispconfig and ls -la /var/log/httpd?

[zetnsh]

Thanks Falko,

[root@mail ~]# ls -la /root/ispconfig
total 144
drwxr-xr-x 9 root root 4096 Jan 17 16:54 .
drwxr-x--- 3 root root 4096 Jan 17 22:27 ..
-rwxr-xr-x 1 root root 33078 Jan 17 16:53 cronolog
-rwxr-xr-x 1 root root 9673 Jan 17 16:53 cronosplit
drwxr-xr-x 12 root root 4096 Jan 17 16:41 httpd
drwxr-xr-x 15 root root 4096 Jan 17 16:53 isp
drwxr-xr-x 6 root root 4096 Jan 17 16:36 openssl
drwxr-xr-x 6 root root 4096 Jan 17 16:46 php
drwxr-xr-x 4 root root 4096 Jan 17 16:53 scripts
drwxr-xr-x 4 root root 4096 Jan 17 16:53 standard_cgis
drwxr-xr-x 2 root root 4096 Jan 17 16:53 sv
-rwx------ 1 root root 9389 Jan 17 16:53 uninstall

[root@mail ~]# ls -la /var/log/httpd
total 28
drwx------ 2 root root 4096 Jan 17 17:15 .
drwxr-xr-x 10 root root 4096 Jan 17 23:59 ..
-rw-r--r-- 1 root root 0 Jan 17 16:21 access_log
-rw-r--r-- 1 root root 2893 Jan 17 18:05 error_log

You'll note access log is zero length because httpd won't start:

[root@mail ~]# tail /var/log/httpd/error_log
unable to start piped log program '/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d': Permission denied
Unable to open logs

Interestingly enough, I have compared this to a working ISPConfig install I did at the same time:

[root@historydirect ~]# ls -al /var/log/httpd/
total 5524
drwx------ 2 root root 4096 Jan 18 11:29 .
drwxr-xr-x 10 root root 4096 Jan 17 23:59 ..
-rw-r--r-- 1 root root 2203585 Jan 18 19:02 access_log
-rw-r--r-- 1 root root 506961 Jan 18 11:08 access_log.old
-rw-r--r-- 1 root root 8281 Jan 18 13:12 error_log
lrwxrwxrwx 1 root root 46 Jan 18 11:29 ispconfig_access_log -> /var/log/httpd/ispconfig_access_log_2008_01_18
-rw-r--r-- 1 root root 9083 Jan 17 23:59 ispconfig_access_log_2008_01_17
-rw-r--r-- 1 root root 2882896 Jan 18 19:02 ispconfig_access_log_2008_01_18
-rw-r--r-- 1 root root 0 Jan 18 08:55 ssl_access_log
-rw-r--r-- 1 root root 1332 Jan 18 11:29 ssl_error_log
-rw-r--r-- 1 root root 0 Jan 18 08:55 ssl_request_log
-rw------- 1 root apache 5292 Jan 18 16:25 suphp_log

The faulty server's directory is missing quite a few files and symlinks! And on that paricular install I had modified the logformat directive and renamed the old log file out of the way, hence the ".old" file. Ignore that though - that installation works well and is now happily hosting a large website :)

Thanks once again!

Neil

[falko]

Can you try a chmod 777 /var/log/httpd for testing purposes? Does it work then?

[zetnsh]

Here's a log of what I did:

[root@mail httpd]# cd /var/log/httpd
[root@mail httpd]# chmod 777 .
[root@mail httpd]# ls -al
total 28
drwxrwxrwx 2 root root 4096 Jan 17 17:15 .
drwxr-xr-x 10 root root 4096 Jan 18 23:59 ..
-rw-r--r-- 1 root root 0 Jan 17 16:21 access_log
-rw-r--r-- 1 root root 2893 Jan 17 18:05 error_log
[root@mail httpd]# /etc/init.d/httpd start
Starting httpd: [FAILED]
FYI, the error in the error log is the same:

unable to start piped log program '/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d': Permission denied
Unable to open logs

Interestingly enough, the file (empty!) ssl_error_log has now appeared in the httpd log directory.


I was wondering if it's simply that for some reason the apache user can't call the cronolog program, but if I understand it correctly, at that point in the httpd process, it should still be running as root. The webserver is set (in httpd.conf) to run as apache:apache

Thanks again,

Neil

[falko]

Maybe it's a problem with the cronolog program. Can you run
/root/ispconfig/cronolog --help and see if you get any errors?

[zetnsh]

For what it's worth:

[root@mail ~]# /root/ispconfig/cronolog --help
usage: /root/ispconfig/cronolog [OPTIONS] logfile-spec

-H NAME, --hardlink=NAME maintain a hard link from NAME to current log
-S NAME, --symlink=NAME maintain a symbolic link from NAME to current log
-P NAME, --prev-symlink=NAME maintain a symbolic link from NAME to previous log
-l NAME, --link=NAME same as -S/--symlink
-h, --help print this help, then exit
-p PERIOD, --period=PERIOD set the rotation period explicitly
-d DELAY, --delay=DELAY set the rotation period delay
-o, --once-only create single output log from template (not rotated)
-x FILE, --debug=FILE write debug messages to FILE
( or to standard error if FILE is "-")
-a, --american American date formats
-e, --european European date formats (default)
-s, --start-time=TIME starting time
-z TZ, --time-zone=TZ use TZ for timezone
-V, --version print version number, then exit

For what it's worth though, when I get chance (which has to be soon because I need to move onto this box ASAP!) I'll enable more detailed logging in Apache - that may well reveal the problem. I may well also go though the setup again - I just used Perfect Server Fedora 7. It went fine for the other machine I configured at the same time.

Hopefully one of those two steps will reveal the problem, and when it does (he says confidently!) I'll post here.

Thanks,

Neil

[falko]

When you disabled SELinux, did you restart the system?

[zetnsh]

I'm not sure I did actually, (surely you don't have to reboot with Linux!! ;-) but I've rebooted it now, and still the same result.

It's got to be something really simple.

I need to sort this today or tomorrow anyway, so I will get to the answer! But I'll probably just try deinstalling and reinstalling ISPConfig just in case something screwey happened!

Thanks,

N

[zetnsh]

OK - well to cut a long story short, I'm still struggling, and have no idea why!

I have uninstalled ISPConfig, yum remove'd httpd INCLUDING /var/log/httpd and /etc/httpd

I reinstalled httpd as per part of this post (http://www.howtoforge.com/perfect_server_fedora7_p5).

I re-downloaded the ISPConfig installer, tar zxvf'd it and ran the setup program again.

In short, same problem. I reckon the problem is with cronolog not being executable by apache, although it looks like it is executable by anyone.

What completely mystifies me is that with two identical fc6 and ISPConfig installs - done together in parallell, one works one doesn't!

Help!

I might add, extensive googling on the subject hasn't got me anywhere :-(

[zetnsh]

A bit more info on this: a partial solution is to comment out this line from httpd.conf like this:

#CustomLog "|/root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/httpd/ispconfig_access_log_%Y_%m_%d" combined_ispconfig

It has the obvious drawback that logging doesn't work via the normal route. HOWEVER, as I use a custom awstats installation with a separate log directive, it won't matter that much in this case. It would just be nice to solve it properly!

So what do we know?

1. The line quoted above is giving some sort of permission denied error - the error is being returned from executing the whole command (in quotation marks).
2. /root has the permissions 755 (I changed that as a test case).
/root/ispconfig has the permissions 755
/root/ispconfig/cronolog has the permissions 755 (so is executable by anyone)
3. Apache user/group according to httpd.conf is apache:apache

It looks like apache can't execute cronolog.

So - I created a very simple perl script called /root/ispconfig/testlog.pl which writes to a file called /tmp/ispdebug.log, just the user under which the script is running. I restarted apache. It failed with the usual error - so I know it's not a cronolog problem. I then gave the apache user a valid shell, su'd to it, and checked that I could manually run /root/ispconfig/testlog.pl. It ran fine and created the temporary log file.

Therefore it looks like something to do with apache itself. Version info:

[root@mail conf]# httpd -v
Server version: Apache/2.2.6 (Unix)
Server built: Sep 18 2007 11:26:13
[root@mail conf]# httpd -l
Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c

Does anyone have any thoughts on this? I'm running out of ideas! No, that's a lie. I ran out of ideas days ago :-o

[falko]

Can you try a chmod 755 /root?

[zetnsh]

Thanks, but I've already tried that. No difference! Very very wierd.

I really don't think this is a permissions problem, even though it looks like it. For some reason apache just won't run ANY piped log process I try, so maybe it's some wierd kernel issue. Can't think what though!

N

[falko]

You could copy /root/ispconfig/cronolog to some other directory, e.g. /home/admispconfig/ispconfig and also change the path in your Apache configuration. Maybe that works.

[zetnsh]

As documented on other posts, I have found the answer!

See this post (http://www.howtoforge.com/forums/showthread.php?p=106153#post106153).

Thanks everyone for their help. The problem was indeed to do with selinux!