Need to restore server logs to webalizer

Discussion in 'ISPConfig 3 Priority Support' started by lobius, Apr 14, 2014.

  1. lobius

    lobius Member

    I have them all.

    I tried to delete current and history files and run webalizer.

    No effect.

    I wrote a Python script to merge the files in chronological order and run mergelog | webalizer.

    It's telling me it cannot read March 1. Might be an old program reading a different format?

    Anybody here have experience restoring server logs?

    Please enlighten me.
     
  2. lobius

    lobius Member

    Looks like /var/log/apache2 has a log for each week.

    These files will not work as far as I can tell.

    So I have two months of access logs generated by webalizer.

    I wrote a little Python script to run webalizer against each log file in my directory.

    The output follows…
    Code:
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140317-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    3146 records (3146 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140318-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5817 records (5817 ignored) in 1.00 seconds, 5817/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140319-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5247 records (5247 ignored) in 1.00 seconds, 5247/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140320-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    10326 records (10326 ignored) in 1.50 seconds, 6884/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140321-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    7890 records (7890 ignored) in 1.50 seconds, 5260/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140322-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    2735 records (2735 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140323-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    3198 records (3198 ignored) in 0.00 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140324-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5235 records (5235 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140325-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4491 records (4491 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140326-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5275 records (5275 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140327-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    6767 records (6767 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140328-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4445 records (4445 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140329-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4031 records (4031 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140330-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    6903 records (6903 ignored) in 1.00 seconds, 6903/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140331-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4958 records (4958 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140401-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4020 records (4020 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140402-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    6934 records (6934 ignored) in 1.00 seconds, 6934/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140403-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    Warning: Truncating oversized hostname
    Skipping bad record (1)
    7421 records (7420 ignored, 1 bad) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140404-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5571 records (5571 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140405-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    Warning: Truncating oversized hostname
    Skipping bad record (1)
    2948 records (2947 ignored, 1 bad) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140406-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    2848 records (2848 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140407-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4434 records (4434 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140408-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    5291 records (5291 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140409-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    4572 records (4572 ignored) in 0.00 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140410-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    8973 records (8973 ignored) in 1.00 seconds, 8973/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140411-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    3764 records (3764 ignored) in 0.50 seconds
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140413-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    Saving current run data... [04/13/2014 23:48:15]
    Generating report for April 2014
    Generating summary report
    Saving history information...
    2118 records (133 ignored) in 2.50 seconds, 847/sec
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile 20140414-access.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'sunpin'
    Reading history file... webalizer.hist
    Reading previous run data.. webalizer.current
    Saving current run data... [04/14/2014 22:00:01]
    Generating report for April 2014
    Generating summary report
    Saving history information...
    2759 records in 2.00 seconds, 1379/sec
    
    Going to bed now. I will see what it did be in the morning.
     
  3. lobius

    lobius Member

    Except that you know that's not true. I got all my months back. Somehow.

    I'm not sure my script did anything because April 1-5 is missing and six through 13 has garbage data.

    Any ideas?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    The files in that directory do not contain the log data of your websites, so they can not be used to restore log data for sites. The log data of the websites is in the log directory of the sites which is a bind mount to /var/log/ispconfig/httpd/....
     
  5. lobius

    lobius Member

    I have the log data of my websites as well.

    Right now, I'm up to April be in the remaining month to correct.
     
  6. lobius

    lobius Member

    Merged my April logs to create a log file called restore.log.

    Deleted webalizer.current from my stats directory

    Ran webalizer -n restore.log.

    No effect.
    Code:
    [email protected]:/usr/src/pyscripts# python mergeLogs.py
    [email protected]:/usr/src/pyscripts# cd /var/www/clients/client5/web11/log
    [email protected]:/var/www/clients/client5/web11/log# webalizer -n restore.log
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile /var/log/apache2/access.log.1 (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'restore.log'
    Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist
    Reading previous run data.. /var/www/clients/client5/web11/web/stats/webalizer.current
    3268 records (3268 ignored) in 0.00 seconds
    [email protected]:/var/www/clients/client5/web11/log# webalizer -n restore.log
    Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile /var/log/apache2/access.log.1 (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'restore.log'
    Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist
    Previous run data not found...
    Saving current run data... [04/13/2014 06:45:01]
    Generating report for April 2014
    Generating summary report
    Saving history information...
    3268 records in 1.00 seconds, 3268/sec
    [email protected]:/var/www/clients/client5/web11/log#
    
    Getting the rest of the history back has saved my job for a moment. Need to get April squared away.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    You might want to take a look at the ispconfig script /usr/local/ispconfig/server/cron_daily.php to see which parameters are used by ispconfig to generate the statistics. ispconfig geerates the statistics day by day, so you might use a script that loops trough the log files aftre you uncompressed them.
     
  8. lobius

    lobius Member

    Deleted current. Got a clean response on the following:
    Code:
    [email protected]:/var/www/clients/client5/web11/log# webalizer restore.log          Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8
    Using logfile restore.log (clf)
    Using default GeoIP database
    Creating output in /var/www/clients/client5/web11/web/stats
    Hostname for reports is 'ellatinodigital.com'
    Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist
    Previous run data not found...
    Generating report for March 2014
    Saving current run data... [04/15/2014 11:17:45]
    Generating report for April 2014
    Generating summary report
    Saving history information...
    64391 records in 22.00 seconds, 2926/sec
    
    What did you want me to see in the mail log daily?

    This is my Python code:

    Code:
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    #Copyright 2014 Christopher James Uzal. 
    
    #This will merge the log file to restore the statistics.
    
    import sys, os
    
    path = r'/var/www/clients/client5/web11/log/'
    logs = []
     
    os.chdir(path)
    
    #Get the filenames into a list
    logs += [each for each in os.listdir(path) if each.endswith('-access.log')]
    
    logs.sort()
    
    os.system('mergelog ' + ' '.join(logs) + ' > restore.log ' )
    
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    the code and webalizer that ispcpnfig uses to create the statistics for all sites:

    Code:
    $sql = "SELECT domain_id, domain, document_root, web_folder, type, parent_domain_id FROM web_domain WHERE (type = 'vhost' or type = 'vhostsubdomain') and stats_type = 'webalizer' AND server_id = ".$conf['server_id'];
    $records = $app->db->queryAllRecords($sql);
    
    foreach($records as $rec) {
    	//$yesterday = date('Ymd',time() - 86400);
    	$yesterday = date('Ymd', strtotime("-1 day", time()));
    
    	$log_folder = 'log';
    	if($rec['type'] == 'vhostsubdomain') {
    		$tmp = $app->db->queryOneRecord('SELECT `domain` FROM web_domain WHERE domain_id = '.intval($rec['parent_domain_id']));
    		$subdomain_host = preg_replace('/^(.*)\.' . preg_quote($tmp['domain'], '/') . '$/', '$1', $rec['domain']);
    		if($subdomain_host == '') $subdomain_host = 'web'.$rec['domain_id'];
    		$log_folder .= '/' . $subdomain_host;
    		unset($tmp);
    	}
    	$logfile = escapeshellcmd($rec['document_root'].'/' . $log_folder . '/'.$yesterday.'-access.log');
    	if([email protected]_file($logfile)) {
    		$logfile = escapeshellcmd($rec['document_root'].'/' . $log_folder . '/'.$yesterday.'-access.log.gz');
    		if([email protected]_file($logfile)) {
    			continue;
    		}
    	}
    
    	$domain = escapeshellcmd($rec['domain']);
    	$statsdir = escapeshellcmd($rec['document_root'].'/'.($rec['type'] == 'vhostsubdomain' ? $rec['web_folder'] : 'web').'/stats');
    	$webalizer = '/usr/bin/webalizer';
    	$webalizer_conf_main = '/etc/webalizer/webalizer.conf';
    	$webalizer_conf = escapeshellcmd($rec['document_root'].'/log/webalizer.conf');
    
    	if(is_file($statsdir.'/index.php')) unlink($statsdir.'/index.php');
    
    	if([email protected]_file($webalizer_conf)) {
    		copy($webalizer_conf_main, $webalizer_conf);
    	}
    
    	if(@is_file($webalizer_conf)) {
    		setConfigVar($webalizer_conf, 'Incremental', 'yes');
    		setConfigVar($webalizer_conf, 'IncrementalName', $statsdir.'/webalizer.current');
    		setConfigVar($webalizer_conf, 'HistoryName', $statsdir.'/webalizer.hist');
    	}
    
    
    	if([email protected]_dir($statsdir)) mkdir($statsdir);
    	exec("$webalizer -c $webalizer_conf -n $domain -s $domain -r $domain -q -T -p -o $statsdir $logfile");
    }
     
  10. lobius

    lobius Member

    Solved.

    Definitely need to merge the log files before restoring. Going one file at a time doesn't work from my little adventure but I could be wrong.

    Also, you need to delete the webalizer.current file and pop out the current day from your list (or array) and then, in spite of warnings in the documentation, you can pull this off in the middle of the month.

    w00t. Thanks for reading.
     

Share This Page