ISPConfig and BandWidth control (mod_cband)

[ecorona]

I wrote a good tutorial how to set up ISPConfig for bandwidth control using diferent settings per vhosts or users.

ISPConfig and BandWidth control (mod_cband) (http://www.e-corona.org/blog,245).

[falko]

Thanks for this one! :)

[ecorona]

I think one i can understand how ispconfig forms work i can automate the process and create a new section in the site creation to manage it.

If some one does it please trhow me a line.

[danf.1979]

That would be great!

[juampi]

Great Job!
I will test it!

JP

[nenad]

I wrote a good tutorial how to set up ISPConfig for bandwidth control using diferent settings per vhosts or users.

ISPConfig and BandWidth control (mod_cband) (http://www.e-corona.org/blog,245).

Please note that the pkg mentioned in how-to is not valid anymore.

Current release is: libapache2-mod-cband_0.9.7.4-0bpo1_i386.deb
You can get it here:
wget http://www.backports.org/debian/pool/main/m/mod-cband/libapache2-mod-cband_0.9.7.4-0bpo1_i386.deb
-------------------------------------------------------------------------------------------------------
Please note that:
/root/ispconfig/isp/conf/host.conf.master

mentioned in your manual is:

/root/ispconfig/isp/conf/vhost.conf.master
-----------------------------------------------------
Do you know why this is not working:

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

CBandClassRemoteSpeed googlebot_class 20kb/s 2 3

I receive error #NOT SUPPORTED#
--------------------------------------------------------
BTW what is exact purpose of "scoreboard" ? (file?)
Is it logfile? In which format? How can I read it?

[falko]

I receive error #NOT SUPPORTED#

It probably means that mod_cband is not enabled in your Apache configuration.

[nenad]

Actually it is:

take a look here:

http://www.cvecara-neven.com/cband-status?refresh=5

besides it accept these directives:

CBandPeriod 4W
CBandScoreboard /var/run/apache2/cvecara-neven.com.scoreboard

--------------------------------------------------------
BTW what is exact purpose of "scoreboard" ? (file?)
Is it logfile? In which format? How can I read it?

[nenad]

Is that error #NOT SUPPORTED# Apache or ISPConfig or mod_cband specific error? Who generates info about that error?

[falko]

ISPConfig checks your directives with the httpd -t command, and if that doesn't say "Syntax OK", ISPConfig comments out the directives. So it's Apache that's complaining.

Do you have a ServerName directive before your mod_cband directive? That's important, otherwise it doesn't work.

[nenad]

I guess that I have:

----------------------------------------------
File /root/ispconfig/isp/conf/vhost.conf.master

{FP_RESOURCE_CONFIG}
{FP_ACCESS_CONFIG}
###################################
#
# ISPConfig vHost Configuration File
# Version 1.0
#
###################################
#
<!-- BEGIN DYNAMIC BLOCK: namevhost -->
{SERVERIP}
<!-- END DYNAMIC BLOCK: namevhost -->
#
#
<!-- BEGIN DYNAMIC BLOCK: vhost -->

######################################
# Vhost: {SERVERNAME}
######################################
#
#
<VirtualHost {IP}>
{SUEXEC}
ServerName {SERVERNAME}
{HTTPD_INCLUDE}
ServerAdmin {SERVERADMIN}
DocumentRoot {DOCUMENTROOT}
{SERVERALIAS}
{DIRECTORYINDEX}
{CGI}
ErrorLog {WEB_ERROR_LOG}
{PHP}
{SSI}
{WAP}
{ERRORALIAS}
{ERROR}
AliasMatch ^/~([^/]+)(/(.*))? {HTTPD_ROOT}/{WEB}/user/$1/web/$3
AliasMatch ^/users/([^/]+)(/(.*))? {HTTPD_ROOT}/{WEB}/user/$1/web/$3
{REWRITE_RULE}
{FRONTPAGE}
</VirtualHost>
#
{SSL}
#
#
<!-- END DYNAMIC BLOCK: vhost -->
--------------------------------------------------------
Othervise this wouldn't work

http://www.cvecara-neven.com/cband-status-me?refresh=5

or it would?

-----------------------------------------------------
Instructions taken from http://www.e-corona.org/blog,245

------------------------------------------------------------
If you get no errors at this point we are ready to change ISPConfig to be able to use mod_cband Apache directives.

mod_cband directives must be placed after we set the vhost name ServerName and ISPConfig places our Apache Directives before it so it wont work.

To solve this we have to edit root/ispconfig/isp/conf/vhost.conf.master which is the template to generate vhosts in /etc/apache2/vhost/Vhosts_ispconfig.conf

Move the first line {HTTPD_INCLUDE} below ServerName {SERVERNAME} so Apache Directives from ISPConfig will be placed after ServerName and mod_cband will work.

We are done here so let's take a look in to a few mod_cband Apache directives.

[falko]

What's in your Vhosts_ispconfig.conf (the real one, not the template)?

[nenad]

###################################
#
# ISPConfig vHost Configuration File
# Version 1.0
#
###################################
#
NameVirtualHost 192.168.123.201:80
<VirtualHost 192.168.123.201:80>
ServerName localhost
ServerAdmin root@localhost
DocumentRoot /var/www/sharedip
</VirtualHost>
#
#
######################################
# Vhost: www.web-hosting.co.yu:80
######################################
#
#
<VirtualHost 192.168.123.201:80>
SuexecUserGroup web1_nenad web1
ServerName www.web-hosting.co.yu:80
ServerAdmin webmaster@web-hosting.co.yu
DocumentRoot /var/www/web1/web
ServerAlias web-hosting.co.yu
DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 index.shtml index.cgi index.pl index.jsp Default.htm default.htm
ScriptAlias /cgi-bin/ /var/www/web1/cgi-bin/
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
ErrorLog /var/www/web1/log/error.log
AddType application/x-httpd-php .php .php3 .php4 .php5
php_admin_flag safe_mode Off
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
AddType application/vnd.wap.wmlscriptc .wmlsc .wsc
AddType text/vnd.wap.wml .wml
AddType text/vnd.wap.wmlscript .ws .wmlscript
AddType image/vnd.wap.wbmp .wbmp
Alias /error/ "/var/www/web1/web/error/"
ErrorDocument 400 /error/invalidSyntax.html
ErrorDocument 401 /error/authorizationRequired.html
ErrorDocument 403 /error/forbidden.html
ErrorDocument 404 /error/fileNotFound.html
ErrorDocument 405 /error/methodNotAllowed.html
ErrorDocument 500 /error/internalServerError.html
ErrorDocument 503 /error/overloaded.html
AliasMatch ^/~([^/]+)(/(.*))? /var/www/web1/user/$1/web/$3
AliasMatch ^/users/([^/]+)(/(.*))? /var/www/web1/user/$1/web/$3
</VirtualHost>
#
#
#
######################################
# Vhost: www.cvecara-neven.com:80
######################################
#
#
<VirtualHost 192.168.123.201:80>
SuexecUserGroup web4_nenad web4
ServerName www.cvecara-neven.com:80
CBandPeriod 4W
CBandScoreboard /var/run/apache2/cvecara-neven.com.scoreboard
ServerAdmin webmaster@cvecara-neven.com
DocumentRoot /var/www/web4/web
ServerAlias cvecara-neven.com
DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 index.shtml index.cgi index.pl index.jsp Default.htm default.htm
ScriptAlias /cgi-bin/ /var/www/web4/cgi-bin/
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
ErrorLog /var/www/web4/log/error.log
AddType application/x-httpd-php .php .php3 .php4 .php5
php_admin_flag safe_mode Off
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
AddType application/vnd.wap.wmlscriptc .wmlsc .wsc
AddType text/vnd.wap.wml .wml
AddType text/vnd.wap.wmlscript .ws .wmlscript
AddType image/vnd.wap.wbmp .wbmp
Alias /error/ "/var/www/web4/web/error/"
ErrorDocument 400 /error/invalidSyntax.html
ErrorDocument 401 /error/authorizationRequired.html
ErrorDocument 403 /error/forbidden.html
ErrorDocument 404 /error/fileNotFound.html
ErrorDocument 405 /error/methodNotAllowed.html
ErrorDocument 500 /error/internalServerError.html
ErrorDocument 503 /error/overloaded.html
AliasMatch ^/~([^/]+)(/(.*))? /var/www/web4/user/$1/web/$3
AliasMatch ^/users/([^/]+)(/(.*))? /var/www/web4/user/$1/web/$3
</VirtualHost>
#
#
#
######################################
# Vhost: www.web-hosting-solutions.biz:80
######################################
#
#
<VirtualHost 192.168.123.201:80>
SuexecUserGroup web8_nenad web8
ServerName www.web-hosting-solutions.biz:80
ServerAdmin webmaster@web-hosting-solutions.biz
DocumentRoot /var/www/web8/web
ServerAlias web-hosting-solutions.biz
DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 index.shtml index.cgi index.pl index.jsp Default.htm default.htm
ScriptAlias /cgi-bin/ /var/www/web8/cgi-bin/
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
ErrorLog /var/www/web8/log/error.log
AddType application/x-httpd-php .php .php3 .php4 .php5
php_admin_flag safe_mode Off
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
AddType application/vnd.wap.wmlscriptc .wmlsc .wsc
AddType text/vnd.wap.wml .wml
AddType text/vnd.wap.wmlscript .ws .wmlscript
AddType image/vnd.wap.wbmp .wbmp
Alias /error/ "/var/www/web8/web/error/"
ErrorDocument 400 /error/invalidSyntax.html
ErrorDocument 401 /error/authorizationRequired.html
ErrorDocument 403 /error/forbidden.html
ErrorDocument 404 /error/fileNotFound.html
ErrorDocument 405 /error/methodNotAllowed.html
ErrorDocument 500 /error/internalServerError.html
ErrorDocument 503 /error/overloaded.html
AliasMatch ^/~([^/]+)(/(.*))? /var/www/web8/user/$1/web/$3
AliasMatch ^/users/([^/]+)(/(.*))? /var/www/web8/user/$1/web/$3
</VirtualHost>
#
<IfModule mod_ssl.c>
<VirtualHost 192.168.123.201:443>
SuexecUserGroup web8_nenad web8
ServerName www.web-hosting-solutions.biz:443
ServerAdmin webmaster@web-hosting-solutions.biz
DocumentRoot /var/www/web8/web
ServerAlias web-hosting-solutions.biz
DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 index.shtml index.cgi index.pl index.jsp Default.htm default.htm
ScriptAlias /cgi-bin/ /var/www/web8/cgi-bin/
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
ErrorLog /var/www/web8/log/error.log
AddType application/x-httpd-php .php .php3 .php4 .php5
php_admin_flag safe_mode Off
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
AddType application/vnd.wap.wmlscriptc .wmlsc .wsc
AddType text/vnd.wap.wml .wml
AddType text/vnd.wap.wmlscript .ws .wmlscript
AddType image/vnd.wap.wbmp .wbmp
SSLEngine on
SSLCertificateFile /var/www/web8/ssl/www.web-hosting-solutions.biz.crt
SSLCertificateKeyFile /var/www/web8/ssl/www.web-hosting-solutions.biz.key
Alias /error/ "/var/www/web8/web/error/"
ErrorDocument 400 /error/invalidSyntax.html
ErrorDocument 401 /error/authorizationRequired.html
ErrorDocument 403 /error/forbidden.html
ErrorDocument 404 /error/fileNotFound.html
ErrorDocument 405 /error/methodNotAllowed.html
ErrorDocument 500 /error/internalServerError.html
ErrorDocument 503 /error/overloaded.html
AliasMatch ^/~([^/]+)(/(.*))? /var/www/web8/user/$1/web/$3
AliasMatch ^/users/([^/]+)(/(.*))? /var/www/web8/user/$1/web/$3
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
</VirtualHost>
</IfModule>
#
#

[falko]

That's ok, so mod_cband should work... :confused:

[nenad]

Should I use your install instructions
http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling
(build from source code)

Or should I use:
http://www.e-corona.org/blog,245
(debian precompiled pkg)

Right now I installed mod_cban with both tutorials.
Mybie should I un-install both, and then try only one?

How to deinstall this version:
http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling

And how to deinstall this version:
http://www.e-corona.org/blog,245
-------------------------------------------------------------------------------------
Anyway this is accepted by apache:

CBandClassRemoteSpeed googlebot_class 20kb/s 2 3

But this is not accepted:

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

[nenad]

Debbuging :)

This:

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

Is not accepted, so if I break that into only this

CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24

Than it is accepted.

Therefore my conclusion is that

<CBandClass googlebot_class>
causes all of of the troubles.
I do believe that mod_cband should interpret this in proper manner.
I am wondering at this moment is the support for this command included/properly compiled into

/usr/lib/apache2/modules/mod_cband.so ?
------------------------------------------------------------------
With all RESPECT to the ecorona and his manual this example from his tutorial:

<ifmodule>
# Max 1024kbps speed for this virtualhost
# Max 10 requests per second for this virtualhost
# Max 30 open connections for this virtualhost
CBandSpeed 1024 10 30

# Max 10kB/s speed, 3 requests/s and 2 open connections for any remote client
CBandRemoteSpeed 10kb/s 3 2

# Maximal 20kB/s speed, 2 requests/s and 3 open connections for remote
# clients from class googlebot_class <img src="http://www.e-corona.org/plugins/Emoticons/images/tongue.png" border="0" alt="tongue.png " />
CBandClassRemoteSpeed googlebot_class 20kb/s 2 3
</ifmodule>

Do nothing (the bold /red line) if CBandClass is NOT defined... :(

Correct me if I am wrong.

[nenad]

After I deinstalled every single trace of mod_cband, I installed it once agin according to Falko's tutorial, and guess what?

CBandClass command is not working again thus I do belive that there is a bug in mod_cband.
I will try to write to author of mod_cband.

However it might be posible that I made somewhere some mistake...?

Any toughts about this?

Is there anyone who can make this work, or with working example for
mod_cband and CBandClass command ?

[nenad]

BTW I noticed this:

<VirtualHost 192.168.123.201:80>
SuexecUserGroup web1_nenad web1
ServerName www.web-hosting.co.yu:80
ServerAdmin webmaster@web-hosting.co.yu
DocumentRoot /var/www/web1/web
ServerAlias web-hosting.co.yu

What if I don't want this? "webmaster"@web-hosting.co.yu

I think that there should be an option in ISPConfig to define email of server admin, actually domain admin..?

[nenad]

So, maybie I found what is the problem.
First I stopped ISPConfig, then I put manually

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

And then I checked apache syntax to see what is really happening and here is result:

server201:~# /etc/init.d/ispconfig_server stop
Shutting down ISPConfig system...
/root/ispconfig/httpd/bin/apachectl stop: httpd stopped
ISPConfig system stopped!

server201:~# apache2 -t
[Sat Sep 23 03:05:05 2006] [warn] module cache_module is already loaded, skipping
Syntax error on line 71 of /etc/apache2/vhosts/Vhosts_ispconfig.conf:
<CBandClass> cannot occur within <VirtualHost> section

[nenad]

And BINGO!!!

I added CBandClass manually:

######################################
# Vhost: www.cvecara-neven.com:80
######################################
#
#

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
CBandClassDst 66.249.66.107/24
</CBandClass>

<VirtualHost 192.168.123.201:80>


as you can see just before <VirtualHost 192.168.123.201:80>

and here is result:

http://www.web-hosting.co.yu/images/001.jpg

Nasty GoogleBot who is visiting my small website (600+ pictures & 200+ text pages) every single day all day long, and generates over 300 MB of traffic is finally limited :)

Don't take me wrong, I love Google and I need his spiders crawl my website but not so fast and preferably not to enerate so much traffic.

So, it seems that "ISPConfig & mod_cband for Dummies (as I am)" is therfore finished...

For now :) :cool:

[pontifex]

Hello,

you wrote that u add the command manual above the vhost directives. but what happens when u change a setting in the ispconfig backend that causes the httpd.conf to be rewritten? (please test it once..)

you changes will be gone?! is that correct?

any help appreciated ;-)

Ciao
PM

[nenad]

Actually, when I change any setting in ISPConfig

/etc/apache2/vhosts/Vhosts_ispconfig.conf

is NOT rewritten. Even if I want to. :eek: HELP !

This

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

can be places in
/etc/apache2/apache2.conf

just before this line:

Include /etc/apache2/vhosts/Vhosts_ispconfig.conf

[falko]

Actually, when I change any setting in ISPConfig

/etc/apache2/vhosts/Vhosts_ispconfig.conf

is NOT rewritten. Even if I want to. :eek: HELP !

What's the output of ls -la /root/ispconfig?
What happens when you run
/root/ispconfig/php/php /root/ispconfig/scripts/writeconf.php?

[nenad]

server201:~# ls -la /root/ispconfig
total 124
drwxr-xr-x 9 root root 4096 2006-09-23 11:54 .
drwxr-xr-x 9 root root 4096 2006-09-22 23:03 ..
-rwxr-xr-x 1 root root 54016 2006-05-28 16:51 cronolog
-rwxr-xr-x 1 root root 9673 2006-05-28 16:51 cronosplit
drwxr-xr-x 12 root root 4096 2006-05-28 16:16 httpd
drwxr-xr-x 12 root root 4096 2006-05-28 16:51 isp
-rw-r--r-- 1 root root 8 2006-09-23 11:54 .old_path_httpd_root
drwxr-xr-x 6 root root 4096 2006-05-28 16:02 openssl
drwxr-xr-x 6 root root 4096 2006-05-28 16:26 php
drwxr-xr-x 4 root root 4096 2006-06-29 22:47 scripts
drwxr-xr-x 4 root root 4096 2006-05-28 16:51 standard_cgis
drwxr-xr-x 2 root root 4096 2006-05-28 16:51 sv
-rwx------ 1 root root 9389 2006-05-28 16:51 uninstall
server201:~#


server201:~# /root/ispconfig/php/php /root/ispconfig/scripts/writeconf.php
start
ende
server201:~#


And /etc/apache2/vhosts/Vhosts_ispconfig.conf
is NOT rewritten?

[falko]

The outputs are ok. Are you sure you changed anything in ISPConfig, anything that should be changed in Vhosts_ispconfig.conf?

[nenad]

Yes, I deleted mod_cband directives from Apache directives option in ISPConfig, and turned on anonymus FTP, then restarted Apache...

[falko]

What's the output of ps aux|grep wconf?

[nenad]

server201:~# ps aux|grep wconf
root 2267 0.0 0.4 2468 1080 ? S Sep23 0:00 /bin/bash /root/ispconfig/sv/ispconfig_wconf
root 25570 0.0 0.2 1820 568 pts/0 R+ 20:13 0:00 grep wconf
server201:~#

[falko]

That's also ok.
Maybe your disk is full? What's the output of df -h? Maybe you also need to reboot your system... :confused:

[nenad]

System rebooted several times.

server201:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 18G 4.9G 13G 29% /
tmpfs 125M 0 125M 0% /dev/shm
server201:~#


Is there any chance that saving file as DOS instead of UNIX type, or ASCII or UTF8 conversion caused this problem ?

[falko]

Is there any chance that saving file as DOS instead of UNIX type, or ASCII or UTF8 conversion caused this problem ?
Yes, whenever you edit a file on your Windows system and you want to transfer it back to your Linux system, make sure you save it with Unix linebreaks (if you use FTP, the conversion is done automatically).

[nenad]

That was the problem.

Some files I was editing directly with JOE, and some other with UltraEdit through FTP interface. Probably there was problem with codepage or alike, since all files were in UNIX format (linebreaks).

Anyway, to avoid to much experimetation, I deleted:

/etc/apache2/vhosts/Vhosts_ispconfig.conf

and

I deleted and then created fresh file

/root/ispconfig/isp/conf/vhost.conf.master

and now everything works fine.

These apache directives:

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>

I put into:

/etc/apache2/apache2.conf

CBandScoreFlushPeriod 1
CBandRandomPulse On

<CBandClass googlebot_class>
CBandClassDst 66.249.64/24
CBandClassDst 66.249.65/24
CBandClassDst 66.249.79/24
</CBandClass>


Include /etc/apache2/vhosts/Vhosts_ispconfig.conf

[nenad]

BTW I noticed this:



What if I don't want this? "webmaster"@web-hosting.co.yu

I think that there should be an option in ISPConfig to define email of server admin, actually domain admin..?

Any answer to this?

[falko]

Any answer to this?
You'd have to change the ISPConfig source code (function make_vhost() in /root/ispconfig/scripts/lib/config.lib.php) to change it.