Logsize issue

Discussion in 'General' started by cryptic, Nov 6, 2007.

  1. wpwood3

    wpwood3 New Member

    Results of debug script

    Till
    Thanks for the debug script. I ran the script on my test server and my live server and got different results. It seemed to run correctly on my test server and displayed the logsizes in each web as well as the settings in ISPConfig for logsize.

    The results from my live server are posted below. As you can see, it does not display the debug information. At first I thought maybe my XCache was using the old webalizer.php but then I remembered that ISPConfig has it's own version of php so that should not be an issue. Regardless, it appears to me that the entire webalizer.php script did not execute. Do you think the script timed out?

    Code:
    [root@centos5 shell]# /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    5006121 records (5006121 ignored) in 905.50 seconds, 5528/sec
    [root@centos5 shell]# 
     
    Last edited: Jan 9, 2008
  2. till

    till Super Moderator

    It executed as you got the output. Please make sure that debugging is enabled in the script. There is a variable to enable and disable it with description.
     
  3. wpwood3

    wpwood3 New Member

    It was set as follows:
    Code:
     
    //** 1 = Debugging enabled, 0 = Debugging disabled -----------------
    $webalizer_debug = 1;
    // -----------------------------------------------------------------
    
    I used exactly the same script with the same settings on both servers.
     
  4. falko

    falko Super Moderator

    I've jut checked this, but can't reproduce the problem. What exactly did you enter in the logsize field? 5 or 5MB? It must be 5 (without MB).
     
  5. wpwood3

    wpwood3 New Member

    Logsize setting is ignored in ISPConfig

    Hi falko-
    I entered 5 and NOT 5MB. I have also tried 20 but it makes no difference. At the end of January my web2 logfile was 5,617,626,707 bytes. This is on my CentOS 5.1 64bit server. See attached picture.

    My 32bit CentOS 5.1 test server appears to have the same problem.

    The logsize setting has never worked on either of my 2 systems.
    Webalizer runs nightly and produces no errors. There are no cron errors either. I have not made any modifications to webalizer.

    Any ideas?
     

    Attached Files:

  6. falko

    falko Super Moderator

    And
    Code:
    /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    shows no errors?
     
  7. wpwood3

    wpwood3 New Member

    No errors

    Here's the output:
    Code:
     
    [root@centos ~]# /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    1230076 records (1230076 ignored) in 201.00 seconds, 6119/sec
    [root@centos ~]#
    It took a long time to run but there are no errors.
     
  8. falko

    falko Super Moderator

    Can you post the contents of your /root/ispconfig/scripts/shell/webalizer.php?
     
  9. wpwood3

    wpwood3 New Member

    webalizer.php

    Here it is:
    Code:
    <?
    /*
    Copyright (c) 2005, projektfarm Gmbh, Till Brehm, Falko Timme
    All rights reserved.
    Redistribution and use in source and binary forms, with or without modification,
    are permitted provided that the following conditions are met:
        * Redistributions of source code must retain the above copyright notice,
          this list of conditions and the following disclaimer.
        * Redistributions in binary form must reproduce the above copyright notice,
          this list of conditions and the following disclaimer in the documentation
          and/or other materials provided with the distribution.
        * Neither the name of ISPConfig nor the names of its contributors
          may be used to endorse or promote products derived from this software without
          specific prior written permission.
    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
    WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
    IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
    INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
    BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
    OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
    EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    */
    /**************************
    *
    * ISPConfig Webalizer Script
    * Version 1.0
    *
    ***************************/
    set_time_limit(0);
    include("/root/ispconfig/scripts/lib/config.inc.php");
    include("/root/ispconfig/scripts/lib/server.inc.php");
    $server_id = $mod->system->server_id;
    $server = $mod->system->server_conf;
    $path_httpd_root = stripslashes($server["server_path_httpd_root"]);
    exec("which webalizer", $webalizer_location, $verify);
    unset($webalizer_location);
    if($verify == 0){
    $web_home = $path_httpd_root;
    $message = "";
    $handle = @opendir($web_home);
    while ($dir = @readdir ($handle)) {
        if ($dir != "." && $dir != "..") {
            if(@!is_link("$web_home/$dir") && substr($dir,0,3) == "web" && is_numeric(substr($dir,3))) {
            // ist kein symbolischer Link
                $webname = $dir;
                $web_doc_id = str_replace("web", "", $webname);
                if($web_data = $mod->db->queryOneRecord("SELECT * FROM isp_isp_web WHERE doc_id = '$web_doc_id' AND webalizer_stats = '1'")){
                  $web_path = $web_home . "/$webname/web";
                  $stats_path = $web_path . "/stats";
                  $logfile = $web_home . "/$webname/log/web.log";
                  $web_user = fileowner($web_path);
                  $web_group = filegroup($web_path);
                  // erstelle Stats Verzeichnis, wenn nicht vorhanden
                  if(!@is_dir($stats_path)) {
                      mkdir($stats_path,0775);
                      chown($stats_path,$web_user);
                      chgrp($stats_path,$web_group);
                      $message .= "Erstelle Statistik Verzeichnis: $stats_path\n";
                  }
     
                  // Experimentell: erstelle .htaccess Dateien mit Zugangsberechtigung für Gruppe des Webs
                  if(!@is_dir($stats_path."/.htaccess")) {
                      $ht_file = "AuthType Basic
    AuthName \"Members Only\"
    AuthUserFile $web_home/$webname/.htpasswd
    <limit GET PUT POST>
    require valid-user
    </limit>";
                      $fp = fopen ($stats_path."/.htaccess", "w");
                      fwrite($fp,$ht_file);
                      fclose($fp);
                      chmod($stats_path."/.htaccess",0664);
                  }
                  if(!@is_dir($web_home."/".$webname."/.htpasswd")) {
                      exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname."/ |cut -f1 -d:", $users);
                      exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname.": |cut -f1 -d:", $users);
                      $ht_file = "";
                      if(!empty($users)){
                         foreach($users as $user){
                           $user_password = exec("cat ".$mod->system->server_conf["shadow_datei"]." | grep '$user:' | grep -w $user |cut -f2 -d:");
                           $ht_file .= "$user:$user_password\n";
                         }
                      } else {
                        $ht_file .= "admin:\$1\$TAVCXZlv\$NAjnpdNgAfPMNT4/A61Z.0\n";
                      }
                      unset($users);
                      $fp = fopen ($web_home."/".$webname."/.htpasswd", "w");
                      fwrite($fp,$ht_file);
                      fclose($fp);
                      chmod($web_home."/".$webname."/.htpasswd",0664);
                      exec("chown :".$webname." ".$web_home."/".$webname."/.htpasswd");
                  }
     
                  // Starte Webalizer
                  if(@is_file($logfile)) {
                      if(!empty($web_data["web_host"])){
                        $web_real_name = $web_data["web_host"].".".$web_data["web_domain"];
                      } else {
                        $web_real_name = $web_data["web_domain"];
                      }
                      $message .= exec("webalizer -n $web_real_name -s $web_real_name -r $web_real_name -q -T -o $stats_path -c /root/ispconfig/scripts/shell/webalizer.conf $logfile")."\n";
                  }
                  exec("chown -R $web_user:$web_group $stats_path &> /dev/null");
                }
            }
        }
    }
    echo $message;
    }
    ////////////// LOGSIZE //////////////////
    function dir_size($dir) {
      $totalsize=0;
      if ($dirstream = @opendir($dir)) {
        while (false !== ($filename = readdir($dirstream))) {
          if ($filename!="." && $filename!=".."){
            if (is_file($dir."/".$filename) && !is_link($dir."/".$filename)){
              $totalsize+=filesize($dir."/".$filename);
            }
            if (is_dir($dir."/".$filename)) $totalsize+=dir_size($dir."/".$filename);
          }
        }
      }
      closedir($dirstream);
      clearstatcache();
      return $totalsize;
    }
    function dir_array($dir){
      $directory_array = array();
      if ($dirstream = @opendir($dir)) {
        while (false !== ($filename = readdir($dirstream))) {
          if ($filename!="." && $filename!=".." && $filename!=".no_delete"){
            if (is_file($dir."/".$filename) && !is_link($dir."/".$filename)){
              $directory_array[$dir."/".$filename] = filemtime($dir."/".$filename);
            }
            if (is_dir($dir."/".$filename)) $directory_array = array_merge($directory_array, dir_array($dir."/".$filename));
          }
        }
      }
      closedir($dirstream);
      clearstatcache();
      return $directory_array;
    }
    $webs = $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    if(!empty($webs)){
      foreach($webs as $web){
        $log_dir = $path_httpd_root."/web".$web["doc_id"]."/log";
        if(is_dir($log_dir)){
          $max_directory_size = str_replace(",", ".", trim($web["optionen_logsize"]));
          if(strstr($max_directory_size, '%')){
            if($web["web_speicher"] == -1){
              $log_check = false;
            } else {
              $parts = explode('%', $max_directory_size);
              if(is_numeric(trim($parts[0])) && trim($parts[0]) >= 0){
                $max_directory_size = str_replace(",", ".", $web["web_speicher"]) * 1048576 * floatval($max_directory_size) / 100;
                $log_check = true;
              } else {
                $log_check = false;
              }
              $parts = NULL;
            }
          } else {
            if(is_numeric($max_directory_size) && $max_directory_size >= 0){
              $max_directory_size = $max_directory_size * 1048576;
              $log_check = true;
            } else {
              $log_check = false;
            }
          }
          $directory_size = dir_size($log_dir);
          if($log_check){
            while($directory_size >= $max_directory_size){
              $files = dir_array($log_dir);
              if(!empty($files)){
                asort($files);
                $files = array_slice ($files, 0, 1);
                foreach($files as $key => $val){
                  if(is_file($key)) unlink($key);
                }
              } else {
                break;
              }
              unset($files);
              $directory_size = dir_size($log_dir);
            }
          }
        }
      }
    }
    //////////////// LOGSIZE ENDE ////////////////
    ?>
     
  10. till

    till Super Moderator

    What is the value of the "website Quota" setting of this site?
     
  11. wpwood3

    wpwood3 New Member

    7500mb

    It is set to 7500 in ISPConfig. The Admin user for this site is also set to 7500.

    Code:
    [root@centos ~]# quota -g web2
    Disk quotas for group web2 (gid 10002): 
         Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
          /dev/sda4 6423692  7680000 7681024            7923       0       0  
     

    Attached Files:

  12. falko

    falko Super Moderator

    Please put this into webalizer.php and run the script again:

    PHP:
    <?
    /*
    Copyright (c) 2005, projektfarm Gmbh, Till Brehm, Falko Timme
    All rights reserved.

    Redistribution and use in source and binary forms, with or without modification,
    are permitted provided that the following conditions are met:

        * Redistributions of source code must retain the above copyright notice,
          this list of conditions and the following disclaimer.
        * Redistributions in binary form must reproduce the above copyright notice,
          this list of conditions and the following disclaimer in the documentation
          and/or other materials provided with the distribution.
        * Neither the name of ISPConfig nor the names of its contributors
          may be used to endorse or promote products derived from this software without
          specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
    WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
    IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
    INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
    BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
    OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
    EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    */

    /**************************
    *
    * ISPConfig Webalizer Script
    * Version 1.0
    *
    ***************************/
    set_time_limit(0);

    //** 1 = Debugging enabled, 0 = Debugging disabled -----------------
    $webalizer_debug 1;
    // -----------------------------------------------------------------

    include("/root/ispconfig/scripts/lib/config.inc.php");
    include(
    "/root/ispconfig/scripts/lib/server.inc.php");
    $server_id $mod->system->server_id;
    $server $mod->system->server_conf;
    $path_httpd_root stripslashes($server["server_path_httpd_root"]);
    $global_stats_user trim($server['global_stats_user']);
    $global_stats_password trim($server['global_stats_password']);
    if(
    $global_stats_password != ''){
            
    //calculate 2/8 random chars as salt for the crypt // by bjmg
            
    if($go_info["server"]["password_hash"] == 'crypt') {
                
    $salt="";
                for (
    $n=0;$n<2;$n++) {
                    
    $salt.=chr(mt_rand(64,126));
                }
            } else {
                
    $salt="$1$";
                for (
    $n=0;$n<8;$n++) {
                    
    $salt.=chr(mt_rand(64,126));
                }
                
    $salt.="$";
            }

            
    $global_stats_password crypt($global_stats_password$salt);
    }

    exec("which webalizer"$webalizer_location$verify);

    unset(
    $webalizer_location);

    if(
    $verify == 0){

    $web_home $path_httpd_root;

    $message "";

    $handle = @opendir($web_home);
    while (
    $dir = @readdir ($handle)) {
        if (
    $dir != "." && $dir != "..") {
            if(@!
    is_link("$web_home/$dir") && substr($dir,0,3) == "web" && is_numeric(substr($dir,3))) {
            
    // ist kein symbolischer Link

                
    $webname $dir;
                
    $web_doc_id str_replace("web"""$webname);
                if(
    $web_data $mod->db->queryOneRecord("SELECT * FROM isp_isp_web WHERE doc_id = '$web_doc_id' AND webalizer_stats = '1'")){
                  
    $web_path $web_home "/$webname/web";
                  
    $stats_path $web_path "/stats";
                  
    $logfile $web_home "/$webname/log/web.log";
                  
    $web_user fileowner($web_path);
                  
    $web_group filegroup($web_path);

                  
    // erstelle Stats Verzeichnis, wenn nicht vorhanden
                  
    if(!@is_dir($stats_path)) {
                      
    mkdir($stats_path,0775);
                      
    chown($stats_path,$web_user);
                      
    chgrp($stats_path,$web_group);
                      
    $message .= "Erstelle Statistik Verzeichnis: $stats_path\n";
                  }


                  
    // Experimentell: erstelle .htaccess Dateien mit Zugangsberechtigung für Gruppe des Webs
                  
    if(!@is_dir($stats_path."/.htaccess")) {

                      
    $ht_file "AuthType Basic
    AuthName \"Members Only\"
    AuthUserFile 
    $web_home/$webname/.htpasswd
    <limit GET PUT POST>
    require valid-user
    </limit>"
    ;
                      
    $fp fopen ($stats_path."/.htaccess""w");
                      
    fwrite($fp,$ht_file);
                      
    fclose($fp);
                      
    chmod($stats_path."/.htaccess",0664);
                  }

                  if(!@
    is_dir($web_home."/".$webname."/.htpasswd")) {

                      
    exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname."/ |cut -f1 -d:"$users);
                      
    exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname.": |cut -f1 -d:"$users);
                      
    $ht_file "";
                      if(!empty(
    $users)){
                         foreach(
    $users as $user){
                           
    $user_password exec("cat ".$mod->system->server_conf["shadow_datei"]." | grep '$user:' | grep -w $user |cut -f2 -d:");
                           
    $ht_file .= "$user:$user_password\n";
                         }
                      } else {
                        
    //$ht_file .= "admin:\$1\$TAVCXZlv\$NAjnpdNgAfPMNT4/A61Z.0\n";
                      
    }
                      if(
    $global_stats_user != '' && $global_stats_password != ''$ht_file .= $global_stats_user.":".$global_stats_password."\n";

                      unset(
    $users);
                      
    $fp fopen ($web_home."/".$webname."/.htpasswd""w");
                      
    fwrite($fp,$ht_file);
                      
    fclose($fp);
                      
    chmod($web_home."/".$webname."/.htpasswd",0664);
                      
    exec("chown :".$webname." ".$web_home."/".$webname."/.htpasswd");
                  }


                  
    // Starte Webalizer
                  
    if(@is_file($logfile)) {
                      if(!empty(
    $web_data["web_host"])){
                        
    $web_real_name $web_data["web_host"].".".$web_data["web_domain"];
                      } else {
                        
    $web_real_name $web_data["web_domain"];
                      }
                      
    $message .= exec("webalizer -n $web_real_name -s $web_real_name -r $web_real_name -q -T -o $stats_path -c /root/ispconfig/scripts/shell/webalizer.conf $logfile")."\n";
                  }

                  
    exec("chown -R $web_user:$web_group $stats_path &> /dev/null");
                }

            }
        }
    }

    echo 
    $message;

    }

    ////////////// LOGSIZE //////////////////
    function dir_size($dir) {
      
    $totalsize=0;
      if (
    $dirstream = @opendir($dir)) {
        while (
    false !== ($filename readdir($dirstream))) {
          if (
    $filename!="." && $filename!=".."){
            if (
    is_file($dir."/".$filename) && !is_link($dir."/".$filename)){
              
    $totalsize+=filesize($dir."/".$filename);
            }
            if (
    is_dir($dir."/".$filename)) $totalsize+=dir_size($dir."/".$filename);
          }
        }
      }
      
    closedir($dirstream);
      
    clearstatcache();
      return 
    $totalsize;
    }

    function 
    dir_array($dir){
      
    $directory_array = array();
      if (
    $dirstream = @opendir($dir)) {
        while (
    false !== ($filename readdir($dirstream))) {
          if (
    $filename!="." && $filename!=".." && $filename!=".no_delete"){
            if (
    is_file($dir."/".$filename) && !is_link($dir."/".$filename)){
              
    $directory_array[$dir."/".$filename] = filemtime($dir."/".$filename);
            }
            if (
    is_dir($dir."/".$filename)) $directory_array array_merge($directory_arraydir_array($dir."/".$filename));
          }
        }
      }
      
    closedir($dirstream);
      
    clearstatcache();
      return 
    $directory_array;
    }

    $webs $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    if(!empty(
    $webs)){
      foreach(
    $webs as $web){
        if(
    $webalizer_debug == 1) echo "Domain: $web[web_domain]\n";
        
    $log_dir $path_httpd_root."/web".$web["doc_id"]."/log";
        if(
    is_dir($log_dir)){
              if(
    $webalizer_debug == 1) echo "Log Dir: $log_dir\n";
          
    $max_directory_size str_replace(",""."trim($web["optionen_logsize"]));
          if(
    strstr($max_directory_size'%')){
            if(
    $web["web_speicher"] == -1){
              
    $log_check false;
                      if(
    $webalizer_debug == 1) echo "We do not check the logsize as Quota is set to unlimited.\n";
            } else {
              
    $parts explode('%'$max_directory_size);
              if(
    is_numeric(trim($parts[0])) && trim($parts[0]) >= 0){
                
    $max_directory_size str_replace(",""."$web["web_speicher"]) * 1048576 floatval($max_directory_size) / 100;
                
    $log_check true;
                            if(
    $webalizer_debug == 1) echo "Max Log size: $max_directory_size\n";
              } else {
                
    $log_check false;
              }
              
    $parts NULL;
            }
          } else {
            if(
    is_numeric($max_directory_size) && $max_directory_size >= 0){
              
    $max_directory_size $max_directory_size 1048576;
              
    $log_check true;
                      if(
    $webalizer_debug == 1) echo "Max Log size: $max_directory_size\n";
            } else {
              
    $log_check false;
            }
          }
          
    $directory_size dir_size($log_dir);
              if(
    $webalizer_debug == 1) echo "Current Log size: $directory_size\n";

          if(
    $log_check){
            while(
    $directory_size >= $max_directory_size){
              
    $files dir_array($log_dir);
              if(!empty(
    $files)){
                
    asort($files);
                
    $files array_slice ($files01);
                foreach(
    $files as $key => $val){
                  if(
    is_file($key)) unlink($key);
                              if(
    $webalizer_debug == 1) echo "Deleting logfile $key\n";
                }
              } else {
                break;
              }
              unset(
    $files);
              
    $directory_size dir_size($log_dir);
            }
          }
        }
      }
    }
    //////////////// LOGSIZE ENDE ////////////////
    ?>
    What's the output?
     
  13. wpwood3

    wpwood3 New Member

    Here it is

    Code:
    [root@centos shell]# /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Error: Skipping oversized log record
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    2630005 records (2629881 ignored, 1 bad) in 443.50 seconds, 5930/sec
    [root@centos shell]# 
    
     
  14. falko

    falko Super Moderator

    It seems as if the script can't read out the web sites from your ISPConfig database. Can you change this section:
    Code:
    $webs = $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    if(!empty($webs)){
      foreach($webs as $web){
        if($webalizer_debug == 1) echo "Domain: $web[web_domain]\n";
    so that it looks like this:
    Code:
    $webs = $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    [B][COLOR="Red"]print_r($webs);[/COLOR][/B]
    if(!empty($webs)){
      foreach($webs as $web){
        if($webalizer_debug == 1) echo "Domain: $web[web_domain]\n";
    and run the script again? What's the output then?
     
  15. wpwood3

    wpwood3 New Member

    No difference

    I made the changes you requested to the same webalizer.php script you gave me above. It did not seem to make any difference.

    Here are the results:
    Code:
     
    [root@centos shell]# /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Error: Skipping oversized log record
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    3414569 records (3414568 ignored, 1 bad) in 555.50 seconds, 6146/sec
     
    
     
  16. falko

    falko Super Moderator

    That's really strange. Even if the MySQL query was emtpy, it should print an empty array.
    Please try this:
    Code:
    $webs = $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    [B][COLOR="Red"]echo "hello\n";[/COLOR][/B]
    if(!empty($webs)){
      foreach($webs as $web){
        if($webalizer_debug == 1) echo "Domain: $web[web_domain]\n";
    and run the script again. If it doesn't print "hello", then you should check if your changes were saved.
     
  17. wpwood3

    wpwood3 New Member

    This is the script I am running:

    Ok, I am running the script again now. I have double checked that the changes you requested are included in the script. Just to be sure, here is the script I am running:
    Code:
     
    <? 
    /* 
    Copyright (c) 2005, projektfarm Gmbh, Till Brehm, Falko Timme 
    All rights reserved. 
    Redistribution and use in source and binary forms, with or without modification, 
    are permitted provided that the following conditions are met: 
        * Redistributions of source code must retain the above copyright notice, 
          this list of conditions and the following disclaimer. 
        * Redistributions in binary form must reproduce the above copyright notice, 
          this list of conditions and the following disclaimer in the documentation 
          and/or other materials provided with the distribution. 
        * Neither the name of ISPConfig nor the names of its contributors 
          may be used to endorse or promote products derived from this software without 
          specific prior written permission. 
    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 
    WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
    IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
    INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 
    BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY 
    OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, 
    EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
    */ 
    /************************** 
    * 
    * ISPConfig Webalizer Script 
    * Version 1.0 
    * 
    ***************************/ 
    set_time_limit(0); 
    //** 1 = Debugging enabled, 0 = Debugging disabled ----------------- 
    $webalizer_debug = 1; 
    // ----------------------------------------------------------------- 
    include("/root/ispconfig/scripts/lib/config.inc.php"); 
    include("/root/ispconfig/scripts/lib/server.inc.php"); 
    $server_id = $mod->system->server_id; 
    $server = $mod->system->server_conf; 
    $path_httpd_root = stripslashes($server["server_path_httpd_root"]); 
    $global_stats_user = trim($server['global_stats_user']); 
    $global_stats_password = trim($server['global_stats_password']); 
    if($global_stats_password != ''){ 
            //calculate 2/8 random chars as salt for the crypt // by bjmg 
            if($go_info["server"]["password_hash"] == 'crypt') { 
                $salt=""; 
                for ($n=0;$n<2;$n++) { 
                    $salt.=chr(mt_rand(64,126)); 
                } 
            } else { 
                $salt="$1$"; 
                for ($n=0;$n<8;$n++) { 
                    $salt.=chr(mt_rand(64,126)); 
                } 
                $salt.="$"; 
            } 
            $global_stats_password = crypt($global_stats_password, $salt); 
    } 
    exec("which webalizer", $webalizer_location, $verify); 
    unset($webalizer_location); 
    if($verify == 0){ 
    $web_home = $path_httpd_root; 
    $message = ""; 
    $handle = @opendir($web_home); 
    while ($dir = @readdir ($handle)) { 
        if ($dir != "." && $dir != "..") { 
            if(@!is_link("$web_home/$dir") && substr($dir,0,3) == "web" && is_numeric(substr($dir,3))) { 
            // ist kein symbolischer Link 
                $webname = $dir; 
                $web_doc_id = str_replace("web", "", $webname); 
                if($web_data = $mod->db->queryOneRecord("SELECT * FROM isp_isp_web WHERE doc_id = '$web_doc_id' AND webalizer_stats = '1'")){ 
                  $web_path = $web_home . "/$webname/web"; 
                  $stats_path = $web_path . "/stats"; 
                  $logfile = $web_home . "/$webname/log/web.log"; 
                  $web_user = fileowner($web_path); 
                  $web_group = filegroup($web_path); 
                  // erstelle Stats Verzeichnis, wenn nicht vorhanden 
                  if(!@is_dir($stats_path)) { 
                      mkdir($stats_path,0775); 
                      chown($stats_path,$web_user); 
                      chgrp($stats_path,$web_group); 
                      $message .= "Erstelle Statistik Verzeichnis: $stats_path\n"; 
                  } 
     
                  // Experimentell: erstelle .htaccess Dateien mit Zugangsberechtigung für Gruppe des Webs 
                  if(!@is_dir($stats_path."/.htaccess")) { 
                      $ht_file = "AuthType Basic 
    AuthName \"Members Only\" 
    AuthUserFile $web_home/$webname/.htpasswd 
    <limit GET PUT POST> 
    require valid-user 
    </limit>"; 
                      $fp = fopen ($stats_path."/.htaccess", "w"); 
                      fwrite($fp,$ht_file); 
                      fclose($fp); 
                      chmod($stats_path."/.htaccess",0664); 
                  } 
                  if(!@is_dir($web_home."/".$webname."/.htpasswd")) { 
                      exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname."/ |cut -f1 -d:", $users); 
                      exec("cat ".$mod->system->server_conf["passwd_datei"]." | grep ".$web_home."/".$webname.": |cut -f1 -d:", $users); 
                      $ht_file = ""; 
                      if(!empty($users)){ 
                         foreach($users as $user){ 
                           $user_password = exec("cat ".$mod->system->server_conf["shadow_datei"]." | grep '$user:' | grep -w $user |cut -f2 -d:"); 
                           $ht_file .= "$user:$user_password\n"; 
                         } 
                      } else { 
                        //$ht_file .= "admin:\$1\$TAVCXZlv\$NAjnpdNgAfPMNT4/A61Z.0\n"; 
                      } 
                      if($global_stats_user != '' && $global_stats_password != '') $ht_file .= $global_stats_user.":".$global_stats_password."\n"; 
                      unset($users); 
                      $fp = fopen ($web_home."/".$webname."/.htpasswd", "w"); 
                      fwrite($fp,$ht_file); 
                      fclose($fp); 
                      chmod($web_home."/".$webname."/.htpasswd",0664); 
                      exec("chown :".$webname." ".$web_home."/".$webname."/.htpasswd"); 
                  } 
     
                  // Starte Webalizer 
                  if(@is_file($logfile)) { 
                      if(!empty($web_data["web_host"])){ 
                        $web_real_name = $web_data["web_host"].".".$web_data["web_domain"]; 
                      } else { 
                        $web_real_name = $web_data["web_domain"]; 
                      } 
                      $message .= exec("webalizer -n $web_real_name -s $web_real_name -r $web_real_name -q -T -o $stats_path -c /root/ispconfig/scripts/shell/webalizer.conf $logfile")."\n"; 
                  } 
                  exec("chown -R $web_user:$web_group $stats_path &> /dev/null"); 
                } 
            } 
        } 
    } 
    echo $message; 
    } 
    ////////////// LOGSIZE ////////////////// 
    function dir_size($dir) { 
      $totalsize=0; 
      if ($dirstream = @opendir($dir)) { 
        while (false !== ($filename = readdir($dirstream))) { 
          if ($filename!="." && $filename!=".."){ 
            if (is_file($dir."/".$filename) && !is_link($dir."/".$filename)){ 
              $totalsize+=filesize($dir."/".$filename); 
            } 
            if (is_dir($dir."/".$filename)) $totalsize+=dir_size($dir."/".$filename); 
          } 
        } 
      } 
      closedir($dirstream); 
      clearstatcache(); 
      return $totalsize; 
    } 
    function dir_array($dir){ 
      $directory_array = array(); 
      if ($dirstream = @opendir($dir)) { 
        while (false !== ($filename = readdir($dirstream))) { 
          if ($filename!="." && $filename!=".." && $filename!=".no_delete"){ 
            if (is_file($dir."/".$filename) && !is_link($dir."/".$filename)){ 
              $directory_array[$dir."/".$filename] = filemtime($dir."/".$filename); 
            } 
            if (is_dir($dir."/".$filename)) $directory_array = array_merge($directory_array, dir_array($dir."/".$filename)); 
          } 
        } 
      } 
      closedir($dirstream); 
      clearstatcache(); 
      return $directory_array; 
    } 
    $webs = $mod->db->queryAllRecords("SELECT * FROM isp_isp_web");
    echo "hello\n";
    if(!empty($webs)){
      foreach($webs as $web){
        if($webalizer_debug == 1) echo "Domain: $web[web_domain]\n"; 
        $log_dir = $path_httpd_root."/web".$web["doc_id"]."/log"; 
        if(is_dir($log_dir)){ 
              if($webalizer_debug == 1) echo "Log Dir: $log_dir\n"; 
          $max_directory_size = str_replace(",", ".", trim($web["optionen_logsize"])); 
          if(strstr($max_directory_size, '%')){ 
            if($web["web_speicher"] == -1){ 
              $log_check = false; 
                      if($webalizer_debug == 1) echo "We do not check the logsize as Quota is set to unlimited.\n"; 
            } else { 
              $parts = explode('%', $max_directory_size); 
              if(is_numeric(trim($parts[0])) && trim($parts[0]) >= 0){ 
                $max_directory_size = str_replace(",", ".", $web["web_speicher"]) * 1048576 * floatval($max_directory_size) / 100; 
                $log_check = true; 
                            if($webalizer_debug == 1) echo "Max Log size: $max_directory_size\n"; 
              } else { 
                $log_check = false; 
              } 
              $parts = NULL; 
            } 
          } else { 
            if(is_numeric($max_directory_size) && $max_directory_size >= 0){ 
              $max_directory_size = $max_directory_size * 1048576; 
              $log_check = true; 
                      if($webalizer_debug == 1) echo "Max Log size: $max_directory_size\n"; 
            } else { 
              $log_check = false; 
            } 
          } 
          $directory_size = dir_size($log_dir); 
              if($webalizer_debug == 1) echo "Current Log size: $directory_size\n"; 
          if($log_check){ 
            while($directory_size >= $max_directory_size){ 
              $files = dir_array($log_dir); 
              if(!empty($files)){ 
                asort($files); 
                $files = array_slice ($files, 0, 1); 
                foreach($files as $key => $val){ 
                  if(is_file($key)) unlink($key); 
                              if($webalizer_debug == 1) echo "Deleting logfile $key\n"; 
                } 
              } else { 
                break; 
              } 
              unset($files); 
              $directory_size = dir_size($log_dir); 
            } 
          } 
        } 
      } 
    } 
    //////////////// LOGSIZE ENDE //////////////// 
    ?>
    
     
  18. wpwood3

    wpwood3 New Member

    Here is the output of the script:

    It printed your "hello" at the end
    Code:
    [root@centos shell]# /root/ispconfig/php/php -q /root/ispconfig/scripts/shell/webalizer.php
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Error: Skipping oversized log record
    Warning: Truncating oversized request field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized request field
    Warning: Truncating oversized referrer field
    4122293 records (4122292 ignored, 1 bad) in 705.00 seconds, 5847/sec
    hello
    One more piece of information...
    I just tried running your webalizer.php script on my 32bit test server and it ran fine and produced the correct debug output. The issue appears to be limited to my 64bit production server. These two servers have identical software installed.
     
    Last edited: Feb 7, 2008
  19. falko

    falko Super Moderator

    That's strange. I think I have to test this on a 64bit system...
     
  20. wpwood3

    wpwood3 New Member

    Falco-
    Did you test this on a 64-bit system? It's still an issue for me.
    Why isn't this problem listed in the bugtracker?

    If you want me to try running any more test scripts on my server, let me know. I'll be glad to help.
     

Share This Page