jobs not processing

Discussion in 'General' started by Toucan, Oct 12, 2012.

  1. Toucan

    Toucan New Member

    This evening one of my servers, which happens to be a slave filled it's hard disk up by mistake causing the server to grind to halt.

    In an attempt to free up the disk space I set the master to delete a number of test sites. However, because the slave was in such a bad way it didn't respond and now a number jobs are sat in the job queue waiting to process.

    After freeing up disk space the slave has now returned to normal operation, however there are still the same jobs stuck in the queue.

    I can now SSH into the slave without problem. Is there a way I can prompt the pair to start the jobs again?
     
  2. till

    till Super Moderator Howtoforge Staff HowtoForge Supporter ISPConfig Developer

  3. Toucan

    Toucan New Member

    Sorry, yes should have given you a bit more information.

    Debian Lenny
    ISP Config3.0.4.6


    ok, set the master to debug, commented out the line and ran /usr/local/ispconfig/server/server.sh

    and got this back at the command line:
    Code:
    badbuntu:~# /usr/local/ispconfig/server/server.sh
    13.10.2012-22:05 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    13.10.2012-22:05 - DEBUG - No Updated records found, starting only the core.
    /usr/bin/fail2ban-client
    /sbin/iptables
    /sbin/ip6tables
    13.10.2012-22:05 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
    
    I did the same on the slave and this time now it updated all the records and cleared the jobs. So that's great!

    The only problem I have now is the monitor is still not updating.

    The slaver server is still shown as critical and in red showing mysql offline, although it mysql is in fact online.


    The most recent entries in the log is looking like this:
    Code:
    2012-10-13 22:19 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:19 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
    2012-10-13 22:19 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	No Updated records found, starting only the core. 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	No Updated records found, starting only the core. 	
    2012-10-13 22:19 	badbison.badbison.com 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:18 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:18 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
    2012-10-13 22:18 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:16 	badbison.co.uk 	Debug 	Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 	
    2012-10-13 22:16 	badbison.co.uk 	Debug 	No Updated records found, starting only the core. 	
    2012-10-13 22:16 	badbison.co.uk 	Debug 	Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    So the only real problem left is to get the monitor to update itself?

    Thanks
     
    Last edited: Oct 14, 2012
  4. Toucan

    Toucan New Member

    [solved]

    In due course the monitor also updated, although monitor on my set is often a little off.
     
  5. till

    till Super Moderator Howtoforge Staff HowtoForge Supporter ISPConfig Developer

    Then the cron daemon was not running on the slave.

    The monitor updating its records at specific times, so you can not update it by running the script manually if you dont do this at the exact minute were its processing a monitor job. When you start your cron daemon again, then it will update the monitor automatically after some time.
     
  6. Toucan

    Toucan New Member

    A similar problem seems to have come back where jobs are not processing on the slave...

    running the script manually, the slave complains it cannot connect to the master:
    Code:
    25.10.2012-00:45 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    25.10.2012-00:45 - WARNING - Unable to connect to master server.
    25.10.2012-00:45 - DEBUG - No Updated records found, starting only the core.
    /usr/bin/fail2ban-client
    /sbin/iptables
    /sbin/ip6tables
    25.10.2012-00:45 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished.
     
  7. till

    till Super Moderator Howtoforge Staff HowtoForge Supporter ISPConfig Developer

    This means that it is not possible to connect frm slave to master mysql database with the login details for the master database from config.inc.php. Reasons can be:

    Firewall
    Changed hostname
    Changed mysql login details
    Mysql not listening on the external network interface on the master
    Changed ip address
     
  8. Toucan

    Toucan New Member

    Thanks Till

    I restarted the slave and then ran the script again but it didn't seem to make any difference.

    I restarted the master and then suddenly the jobs seem to start processing.

    I'll keep an eye on this.
     

Share This Page