Multiplexing webservers with ISPConfig statistics

[Serg]

Hello, everybody!
I'm using ISPConfig some weeks and can say, that is nice software, congradulation to developement team.
But there is a wider client wishes, and I must to solve some problems.
Preface.
There is standard ISPConfig installation.
Also there is port-based hosting due to plone/zope service on rank of ports 2XXX-2X10
Also there is another independend webserver with old feautures like MYSQL 4.0 and PHP4 (XAMPP powered) on port 85.
And I can(and make) multiplexing to all servers via squid proxy.
Main Apache2 now listen port 82.
Control ISPConfig Apache1.3 listen port 81.
Squid listen port 80 and work as transparent proxy according to simple acl rules.
All works fine, but statistics still not.
I reformat squid log outpot as ISPConfig log format.
There is one difference(example)
ISPConfig /var/log/httpd/ispconfig_access_2007_09_10 :
...GET /favicon.ico HTTP/...
Squid(I point squid log to same place and make needed symlink to it) :
...GET http://test.domain.net/favicon.ico HTTP/...
Then I point main apache2 log file to /dev/null
Squid log utilization is made by my script in cron at 00.01 am, it unlink ispconfig_access_log from old log, delete two-days-old log, creates new day log, put symlink to it and then reconfigure(force reload) squid.
But at morning I see two separate directory in /var/www with same name.(test.mydomain.com was created by ISPConfig panel)
One is test.mydomain.net - symlink, as known to /var/www/webuserID with old log, generating when apache2 stored log directly and belonging to virtual host test.mydomain.net
Second is real direcrory test.mydomain.net with new(not empty,right!) log, belonging to test site and generating from squid log.
First directory is owned by testclient:webuserID
Second is owned by root:root
Directory creation time is 00:30
Webalizer page operates with the old statistic, but all files in stat/ directory was rewritten in 04:00 am last day.
1.What happen? Why there are two separate dirs with same name(but one is symlink)?
And how I can tell script to write site log to correct dir/file(eq symlink)
2.Which process is starter of scripts/shell/logs.php ? In 00.30 as I understand?
It seems, it script run automatically by ... program
3. Can I run(for test purpose, of course, not on production host) logs.php script to see changes with logfiles?
testhost#php logs.php (or any parameters required?)
Big thnks for reply

[falko]

Originally Posted by Serg

2.Which process is starter of scripts/shell/logs.php ? In 00.30 as I understand?
It seems, it script run automatically by ... program

It is run by cron. See

crontab -l

Originally Posted by Serg

3. Can I run(for test purpose, of course, not on production host) logs.php script to see changes with logfiles?
testhost#php logs.php (or any parameters required?)
Big thnks for reply

The correct command is

/root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php

[Serg]

Thank you, Falko

Originally Posted by falko

The correct command is

/root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php

Yes! When I run this script with the analyze some variables (thanks to print_r() ) I catch an error - finishing SPACE in filenames. This was my (hidden ) bug in Logformat string. Shame to me!
And now it fixed and all OK.
Is squid-with-ISPconfig integration guide necessary for anyone?

Great! We have now nice statistics of port-based hosting and (even!) of co-location traffic to another hosts with their web(ftp, etc)-servers. All clients can to operate statistics with ISPConfig client's web face and we can make new hosting plans and invoices.
In this case there is a idea - to prepare a "tiny" client's interface or make a possibility to construct client ifaces by admin, depend of client's needs (eq with/without WEB-FTP, with/without database/co-domain/mail/user management, less menus items and so on)
Any suggestions?

[the_spy]

Originally Posted by Serg

Is squid-with-ISPconfig integration guide necessary for anyone?

can you please explain how is working your installation of squid ? I don't understand how it works in squid and which configuration is needed to distribute requests to one port service or an other, and it could interest me if you can put a small tutorial here why not with ispconfig statistics also... Thanks

[Serg]

Originally Posted by the_spy

can you please explain how is working your installation of squid ? I don't understand how it works in squid and which configuration is needed to distribute requests to one port service or an other, and it could interest me if you can put a small tutorial here why not with ispconfig statistics also.

Well. This is not tutorial, but common issue, because I don't have such time to write full guide now(and in future week too)

We assume, that our apache will listen on port 82, as one of any port-based services and we use Debian
There are steps to prepare hosting service to work in complex:
1. Stop main Apache.
2. Install squid and squidguard by apt-get. (We don't need to use base squid from Debian distro, because it not responces all logformat parameters, discribed in squid tutorial , but we need squidguard)
3. Install squid from source to /usr/local. You can install squidguard from source too and skip step 2 at all.
4. Configure squid. Next code not whole, but touch only engine with ISPConfig

http_port IP.IP.IP.IP:80 transparent
tcp_outgoing_address IP.IP.IP.IP
emulate_httpd_log	off
logformat squid %{Host}>h||||%st||||%>a %un %ul [%tl] "%rm %ru HTTP/%rv" %Hs %st "%{Referer}>h" "%{User-Agent}>h"
access_log /var/log/httpd/ispconfig_access_log squid

acl all src 0.0.0.0/0.0.0.0
acl to_localhost dst 127.0.0.0/8
acl our_host dst IP.IP.IP.IP

http_access deny to_localhost
http_access allow our_host
http_access deny all

follow_x_forwarded_for allow all
log_access allow our_host
log_access deny all
logfile_rotate 0
url_rewrite_program /usr/bin/squidGuard -c /etc/squid/squidGuard.conf
url_rewrite_children 5
url_rewrite_host_header on
# via on
via off #this inject warning in cache log, but works
cache_effective_user proxy
cache_effective_group proxy
digest_generation off
icp_port 0
htcp_port 0
forwarded_for on

You can play with some options such as http_port (transparent, accel, vhost, vport) And we use external redirect program but it is possible use cache_peer directive and point to our apache server as parent cache on port 82 with some successful options, it works too.
5. Configure squidguard

rew dmz {
	s@://control.youdomain.com@://youdomain.com:81@i
	s@://IP.IP.IP.IP@://sharedip.youdomain.com:82@i
	s@://youdomain.com@://youdomain.com:82@i
	s@://www.youdomain.com@://youdomain.com:82@i
#and something else including all virtual hosts, webmail and redirects to another services
	#Last string. If host exist in DNS but has no service on port. This prevent loop
	s@://.*\.youdomain.com@://no-in-serve.youdomain.com:82@i
}
dest local {
}
acl {
	default {
		pass	 local any
		rewrite	 dmz
	}
}

6. chown -R /var/log/httpd to rootroxy
chmod /var/log/httpd to 775
make a file /var/log/httpd/ispconfig_access_log_2007_M_D
where M and D - current month and day
make a symlink to it with ispconfig_access_log
7. start and testing squid.
8. Patch /root(orwhereisyouinstallISP)/ispconfig/scripts/lib/config.lib.php
Near string No 1264 write port :82 instead :80
9. Patch apache's conf
ports.conf
Listen:82

apache2.conf
Comment (#)LogFormat and CustomLog
Put new directives:
SetEnv dontlog 1
LogFormat "%v %b" combined
CustomLog /var/log/something../access.log combined env=!dontlog
10. Start apache. Go to URL youdomain.com:81 (to ISPConfig) and create new hosts with DNS records: control.youdomain.com shareip.youdomain.com and no-in-serve.youdomain.com, put to web directory on last two hosts nice pages with text.
All you hosts in Vhost.conf will updated to port 82 automatilally.
11. Now you can point you browser to control.youdomain.com and see login page of ISPConfig.

Thats not at all. You must take care to rotate logs and to remake correct symlink and permissions in log/httpd, after logs rotation you must reload squid. This is may be done by our frend Cron Oliveira.
Also we must update check_services.php about 82 port.

Now all works, but there are some manual operation when you create or delete site - you must put(or delete) record in squidguard.conf
Work to make this automatically is in progress.
Also squid make different records in logfile about host which has aliases
Another words, requests to domain.com and www.domain.com has separate records. This problem is half-solved(alpha version ) and I not describe jet.