ISPconfig is not processing apache logs: the web Traffic overview and stats are gone

Discussion in 'General' started by KingBeto, Nov 23, 2009.

  1. KingBeto

    KingBeto New Member

    Hi everyone!
    I have a very big problem.. My ISPconfig is not processing the Apache Logs.
    Therefore the system does not have any web statistics and the Traffic overview is wrong since it is only processing ftp and email traffic.

    Here I have an example. Note that the web traffic is not processed, but I have the same amount of visits in Google Analytics each month:

    Month, Web, FTP, Mail, Overall
    11/2009:, 0.00, 9.65, 29.74, 39.38
    10/2009:, 0.00, 26.41, 33.20, 59.61
    09/2009:, 9756.34, 21.90, 40.51, 9818.74
    08/2009:, 428163.42, 1999.26, 69.78, 430232.46


    I checked the apache httpd.conf and everything is ok... And I checked the web.log file that the apache generates and it is OK too.

    Since the ISPconfig is not processing the web.log file, it does not delete old logs after the rotation, so old logs are accumulating and using all the disc space..

    Does anyone had the same problem? Any Idea of what should I do?? I'm lost and it is a very serious problem for me! Thank you!!
     
    Last edited: Nov 23, 2009
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Please post the output of:

    crontab -l
     
  3. KingBeto

    KingBeto New Member

    Hello Till! This is my root crontab -l output

    # crontab -l
    1 1 * * * /scripts/snapshot-adverit.sh
    1 2 * * * /scripts/snapshot-mysql.sh
    0 4 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/awstats.php &> /dev/null
    30 00 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php &> /dev/null
    59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/ftp_logs.php &> /dev/null
    59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/mail_logs.php &> /dev/null
    59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/cleanup.php &> /dev/null
    0 4 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php &> /dev/null
    0,30 * * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php &> /dev/null
    15 3,15 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/quota_msg.php &> /dev/null
    40 00 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/traffic.php &> /dev/null
    05 02 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/backup.php &> /dev/null
    41 5 * * * /etc/webmin/cron/tempdelete.pl

    Thank you!
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Please run this command as root:

    /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php

    and when it is finished, run:

    /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php

    Do you get any errors?
     
  5. KingBeto

    KingBeto New Member

    Here it is. Please Note that for some reason, the ispconfig creates junk folders in the /var/www folder.
    Like the folder "/var/www/foPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3;fficeLivePatch.0.0)" without quotes.



    root@azul:~# /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php

    Warning: readlink(): Invalid argument in /root/ispconfig/scripts/shell/logs.php on line 125

    Warning: readlink(): Invalid argument in /root/ispconfig/scripts/shell/logs.php on line 125
    sh: -c: line 0: syntax error near unexpected token `)'
    sh: -c: line 0: `chown -R 0:web0 /var/www/foPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"'



    root@azul:~# /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    [...]

    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    Warning: Truncating oversized referrer field
    15194 records (15194 ignored) in 3.00 seconds, 5064/sec
    359872 records (359872 ignored) in 76.50 seconds, 4704/sec
    2182 records (2182 ignored) in 0.00 seconds
    57468789 records (57468789 ignored) in 94860.50 seconds, 605/sec
    root@azul:~#
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    There is strange directory in /var/www/ which causes the script to stop. Please take a look in /var/www/, there must be a directory with the name "foPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)". Delete that directory and run the scripts again.
     
  7. KingBeto

    KingBeto New Member

    Till:
    I deleted junk folders. I ran the same command once again as root (root@azul:~# /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php) and I discovered that the logs.php script was generating those junk folders.
    This time it generated a 'ET CLR 3.0.4506.2152; .NET CLR 3.5.30' folder and a localhost folder. I deleted them and ran the same command once again. And the script created AGAIN other junk folders. What do you think It is going on?? Very strange!

    Do you think that maybe the Apache is generating a uncompatible with ISPConfig log file?

    Thank you in advance!!!
     
    Last edited: Dec 3, 2009
  8. falko

    falko Super Moderator ISPConfig Developer

    Can you post an excerpt of your Apache access log?
     
  9. KingBeto

    KingBeto New Member

    Hi!
    This is an example of the log generated. I attached my apache2.conf file.
    Something strange is that it has several log config lines. Weird..

    This is the output of the "tail -f web.log" command:

    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/compartir/linkedin.gif HTTP/1.1" 200 746 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/compartir/linkedin.gif HTTP/1.1" 200 746 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/perfil/ico_enviar_pm.png HTTP/1.1" 200 1686 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/perfil/ico_enviar_pm.png HTTP/1.1" 200 1686 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/common/bt_ingresar.gif HTTP/1.1" 200 543 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.30.35.199 - - [03/Dec/2009:23:59:59 -0300] "GET /images/common/bt_ingresar.gif HTTP/1.1" 200 543 "http://www.cpca.org.ar/contenido/a420_Flan-de-manzana" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 1.1.4322; .NET CLR 3.0.30729)"
    190.51.26.250 - - [03/Dec/2009:23:59:59 -0300] "GET /js/zoom.js HTTP/1.1" 200 244 "http://www.cpca.org.ar/contenido/a835_Sorteo%3A-Navidad-en-Comunidad" "Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.0.15) Gecko/2009101601 Firefox/3.0.15"
    190.51.26.250 - - [03/Dec/2009:23:59:59 -0300] "GET /js/zoom.js HTTP/1.1" 200 244 "http://www.cpca.org.ar/contenido/a835_Sorteo%3A-Navidad-en-Comunidad" "Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.0.15) Gecko/2009101601 Firefox/3.0.15"
    201.231.55.139 - - [03/Dec/2009:23:59:59 -0300] "GET /images/contenidos/blt_articulos_mas_populares.gif HTTP/1.1" 200 995 "http://www.cpca.org.ar/contenido/s149_Concursos" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
    201.231.55.139 - - [03/Dec/2009:23:59:59 -0300] "GET /images/contenidos/blt_articulos_mas_populares.gif HTTP/1.1" 200 995 "http://www.cpca.org.ar/contenido/s149_Concursos" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"


    Thank you in advance!
     

    Attached Files:

  10. KingBeto

    KingBeto New Member

    Please somebody help!!! :(
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Thats because you added some code in the ispconfig configuration part which resultet in dupliacte lines as ispconfig tried to repair the configuration. You can remove the duplicate parts and do not add any code in the ispconfig section for your default vhost.

    But this can not cause your problem. I guess that one of the web log files must be correupted, it contains the data for a single request splitted to more then one line or a similar problem.
     
  12. cubells

    cubells Member HowtoForge Supporter

  13. KingBeto

    KingBeto New Member

    • I reconfigured the httpd.conf following the recomendations and this faq http://www.webalizer.org/faq.html
    • I run sudo /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php and everything is fine
    • Then i run /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php and i continue tu have this kind of output
      Warning: Truncating oversized referrer field
      Warning: Truncating oversized referrer field
      Warning: Truncating oversized referrer field
      Warning: Truncating oversized referrer field
      Warning: Truncating oversized referrer field
      Warning: Truncating oversized referrer field

    Any clue?
     

Share This Page